Best-Worst Experiments

What are they?

A powerful econometric method used to determine the preference a human being places on each item within a set. Best-Worst experiments are a replacement for ratings and ranking scales and are also knowns as Maximum Difference.

Measurement of importance of multiple items, such as brands, product features, employee benefits or advertising claims is a common research task. In the past, approaches such as rating, ranking or points allocation have been used.

Ratings and importance scales tend to report that everything in the list is 'quite appealing'. Other techniques such as points allocation are so difficult that they can interfere with the capture of true preference. Best-Worst is superior to each one of these methods.

Best-Worst gets around these problems by presenting the task in a simple comparison, however enforcing a trade-off at each stage in the data collection.

What is the theory?

Best-Worst is a statistical method invented by Jordan Louviere in 1987 at the University of Alberta, Canada.

A major insight with Best-Worst is that humans are much better at judging items at extremes than in discriminating among items of middling importance.

Consequently in Best-Worst, survey respondents are shown a subset of the possible items and are asked to indicate only the best and worst items.

Best-Worst requires that respondents evaluate all possible pairs of items within the full set. This is achieved through an experimental design.

What are the steps involved in a Best-Worst experiment?

1. inputs

The inputs to a Best-Worst experiment are simply the items that need to be ranked. These may be a set of competing advertising tag lines, a set of features you are considering implementing or even a range of visual treatments.

2. experimental design

An appropriate experimental design should be chosen and statements constructed from it, either online or on paper.

3. collect the data

Data should be input or downloaded, then cleaned of non-completes and missing data.

4. analyse the data

MNL modelling approach is a robust method that we use, particularly as it estimates the error bounds.

However simpler rough and ready methods also may be employed such as ratios of the best and worst frequencies.

For a detailed step by step guide to executing a Best-Worst study with examples click here.

How can I use the results?

The results will be a list of each statement with a preference statistic alongside each.

The most preferred items will have higher utilities (or preference statistic) than the less preferred ones.