Step-by-step guide to runNING a Choice Experiment

This guide outlines the steps involved in running an online Choice Experiment.

Experiment Objective

The objective of a Choice Experiment is to produce a useful model. With that in mind, we should ask what choice are we trying to model about what product or service. Usually it will be the probability that someone will choose something.

Common dependant variables, and their questions are

• choice to buy - would you buy this product?
• preference - which do you prefer?
• time - how long would you stay in this job?

If you are new at this process, you might want to consider contacting a Choice Modelling expert to assist you.

Determine the Attributes and Levels of the Product or Service

Here we attempt to deconstruct our product or service into it's component attributes, so that we can systematically reconstruct hypothetical new ones to test.

These attributes should be independent from each other and generally something we can change.

A soft drink for example has the following attributes

• brand
• flavour
• price
• volume
• container

Note that price, volume and container are not strictly independent in the real world ... but they could be.

Now we break each attribute into it's component levels - for the moment we will limit ourselves to 4 levels maximum - this will become important later when we try to find an experimental design.

• brand :coke, pepsi, dr pepper, mr smedley's
• flavour: cola, cherry, lemonade, soda
• price: $1.00, $1.50, $2.00 $2.50
• volume: 375ml, 500ml, 750ml, 1 Litre
• container: can, glass bottle, plastic bottle, carton

Now review the combinations to get a feel for the hypothetical products by picking one level from each.

one might be

"375ml can of coke (cola flavoured) for $1.50" - which looks familiar

another might be

"1 Litre of dr pepper lemonade in a plastic bottle" - which looks a little odd but is plausible.

It is important not to get worried about hypothetical products that common sense tells you could never possibly exist. As long as they are conceivable - it benefits the model to include them in the data collection as they still add information.

Choose an Experimental Design

Unless you are an expert in this field this is a fairly complex undertaking and a science in itself. However, even if a Choice Modelling expert is assisting you - there is a valuable input you, as a product expert, need to make.

The question is how important it is to capture all the subtle interrelations between each factor and how much you are willing to pay to find that out.

There are effectively 3 paths you can choose from here.

 

 

 

 

 

 

 

 

 

 

 

Practically, Main Effects plans are the most commonly used.

An alternative is to use random designs with which it ia generally not possible to say in advance what effects will be able to be modelled.

Set the Number of Scenarios

Decide on how many treatments you want your respondents to see. Generally 8 or 16 is a common number. The actual number will depend on the size of your experimental design and should divide (or block) your design into equal sizes.

 

Determine Sample Size

As a rule you need more than 20 observations per treatment, per segment.

Assuming your software has a balanced allocation scheme, your total sample size can be calculated from

Sample size = NumSegments x 20 x NumBlocks

 

Build your Experiment

Refer to your software guide for assembling choice experiments if you are performing this step yourself. Your software or choice consultant should at a minimum provide for:

• row and column randomisation of experimental designs
• random without replacement block allocation
• block allocation re-balancing of non-completes
• segment based independent treatment allocation decks
• in block sequence randomisation

Omission of any of these will result in biased responses or require costly over sampling.

Monitor your Experiment

While the experiment is running - you may need to check the allocation with respect to incompletes. Incomplete responses should be recycled in your block allocation before reaching approximately 80% of sample.

Extract and Stack Data

Data should be extracted - incompletes removed and stacked one row per respondent per scenario. Elementary frequency analysis should reveal whether your block allocations have been even and whether further sample is required.

Model the Data

Depending on the model objective - a Multinomial Logit or Probit model can effectively model the data. This step should be performed by an expert analyst.

Present Model in a Meaningful Format

'Christmas tree' diagrams such as the one on the right are a simple and intuitive way of understanding the output.

More complex models, typically present the output as a Decision Support System where non-expert users can input hypothetical products attributes and view the predicted changes in choice.