Step-by-step guide to Running a survey

This guide outlines the steps involved is running a professional market research study with an emphasis on online methods.

Establishing your Objective

Before you start entering your survey you should define your objectives for the study. Clear measurable objectives are preferable to general exploration. Objectives may be:

• How do I sell more?
• How do I increase market share?
• What features of my product do people like or not?
• Am I getting value for money from my TVC campaign - are people remembering it?

If you are new at this process, you might want to consider contacting a market research firm to assist you.

Choosing a Methodology

What market research methodology will achieve your objectives. Some broad methodologies are

• Qualitative - free form conversational responses
• Quantitative - fixed responses to direct questions
• Experiment/Modelling - generation of hypothetical scenarios with a view to producing demand models

Determine Sample Frame

A sample frame is a specification of the number and type of respondents you want to participate in your survey.

• Who Do you need to provide answers to your research?
• Do you want to analyse the data by segments?
• What are the minimum numbers of each segment you require?
• How representative does your sample need to be?

Recruiting Respondents

Depending on your sample frame, you have a number of practical ways you can recruit people to your survey. You can also mix the various techniques.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

In general a wholly online method has major advantages over other methods, namely:

• lower cost
• faster turnaround
• better quality data

If you are using the online medium, there are a number of methods you can use. For fast turnaround commercial projects, using a Panel Provider is generally the most efficient.

Below are some online panel providers we can recommend:

• Pureprofile
• SSI
• ORU

Setting a Budget

You should have enough information to build a budget for the survey. Below is an exhaustive list of all the line items that should be considered when running a research program - not all will be relevant to every project.

Cost estimates are based on our experience with current market rates for 2008 and are displayed in Australian Dollars.

ITEM	Notes	Cost range
Management and Initiation Costs
Expert Consulting	If you need an expert to help design a complex experiment or are using a technology that you are unfamiliar with - e.g. streaming video	$200 to $1,000 per hour
Experiment design	Include this if you are running a structured experiment such as a Choice Model or Best-Worst	$2,000 to $5,000 per design
Project Management	Do you need to cost in your time as part of the project	$150 to $400 per hour
Survey Creation Costs
Video Processing	If you need to convert video for online use or need to de-brand a commercial for a tracking/awareness study	$100 to $300 per 30 sec video
Graphic Design	Are you happy with a standard survey look or are their parts that need to be designed - e.g. shelf mockups for consumer goods, or visuals to help explain a product or concept.	typically $100 to $200 per hour
Questionnaire Data Entry	Your paper based survey will need to be converted to an online format - usually this means data entry into a survey creation tool.	$50 to $100 per hour
Survey Proofing		$50 to $100 per hour
Survey Instrument Testing	If your survey is complex, has lots of branches or conditional statement you should consider testing it with an automated tool to generate simulated data	POA
Fieldwork Costs
Panel setup	Most online panel providers charge an administration fee for setting up your account	$200 to $1,000
Online respondent costs	All costs for recruiting, remuneration and software usage for online study	$5 to $15 per respondent depending on survey length and sample frame
Face to Face respondent costs	All costs for recruiting, remuneration, management and data entry	$50 to $200 per respondent depending on survey length and sample frame
Data Analysis Costs
Data Cleaning and reprocessing	Removing non-completes, duplicates or invalid responses. Assigning respondents to segments. Preparation of analysis variables.	$200 per hour
Standard Analysis	Producing frequency analysis, cross tabs and estimating variances	Depends on Agency
Data Modelling	Detailed modelling of data as per project objectives. Production of Decision Support Systems	from $2,000 to $50,000
Reporting	Integration of results, interpretation, discussion and strategic recommendations	Depends on Agency

 

Preparing the Content

Good preparation of all the assets required will make the final assembly of the survey more efficient, whether online or offline.

The Survey

Creating a hard copy of the questionnaire in a generic format that allows viewing, collaboration and editing. Microsoft Word is probably the best general purpose editor for this task.

At a minimum your survey should contain the following...

• An introduction explaining what they will be required to do and how long it should take them. If the study requires approval by an Ethics committee - the approval number and contact details should be provided
• A set of screen out questions at the beginning - if you are not confident that your sampling method is 100% targeted to your required sample frame
• The main body of the questionnaire or experiment
• Branching/Conditional Instructions - if certain questions only to be shown to respondents who answer in a certain way.
• Demographics questions at the end - in order to calculate the representability of the sample, recalculate the results or for segmentation purposes during analysis
• A thank you page

Some additional suggestions...

• For complex tasks, examples of how to respond.
• A one page free entry for feedback on the survey.
• recording of respondents contact details - if your recruitment method allows it. Note this may be prohibited by panel providers or by ethics constraints.

Images and Assets

Whether you are managing the images yourself or providing them to a print or online agency the following principles apply.

All images to be used in your survey should be produced in the highest quality format as source and then optimised and resized later - It is not possible to start with a low quality image and improve it later.

Images should be provided in either high quality JPG, PNG, TGA or TIFF format as individual files. It is generally not viable to provide them as PowerPoint, word documents or as email attachments.

Building the Survey

If you are using a bureau such as SurveyEngine to host your survey, generally you only need to provide your hardcopy and any assets such as images or videos. It is a good idea to get the content as finalised as possible. Our bureau will implement the survey, after which you will be given an opportunity to review and request final changes.

If you have your own survey software or are using the SurveyEngine survey tool - please refer to the relevant documentation on building a survey.

Testing the Survey

Before going live, you should make sure that the structure of the study and the content is correct. Manually testing and reviewing the data output for simple studies should provide you with the confidence to go ahead.

For complex studies where there may be many different branches - Conditional Logic or a Choice Experiment - manually checking every possible condition is unfeasible. In this case you should consider using a robot to simulate respondents. The SurveyEngine simulator can generate thousands of random responses and a simulated data set. Review of this simulated set can show up any subtle mistakes in survey implementation.

Running a Pilot

If the study you are running is new, complicated or has unknown factors - you should consider running a pilot on on a smaller sample, with options for the respondents to comment at the end. If everything is ok you can of course re-use the data from the pilot set.

Pre-live Checklist

You've set up your online survey and everything looks ok and are about to get respondents to complete it.
Before you invite people, you should make a final check.

A comprehensive checklist can be found here.

Once you have completed the checklist - all that remains is to activate your survey - either by posting a link, emailing to your list or contacting your panel provider to begin invitations.

Monitoring your Survey

While people are filling out your survey you should periodically check what the completion rates are and verify that data is being collected.

Extracting Data

Depending on the software you are using, data extracted may come in a variety of formats.

The most general purpose format is delimited text - also known as CSV or tab-delimited text. This format can be read with any text editor on any computer, has no limits to the number of columns or rows and is generally the smallest file size. Most statistical packages (if not all) have an option to import delimited text.

Cleaning the Data

The extracted data should be reviewed and a scheme for removing certain cases.

Non-completes - generally remove these unless you are desperately short of data. If you are using a panel you are generally not charged for incomplete responses.

Respondents who have taken too short a time or too long a time - a heuristic filter of less 50% or more than 200% of the median time to complete. A good panel provider will not charge you for respondents who are clearly not engaged.

Recoding segment identifiers - if your segments are derived in a complex way, creation of a binary column indicating inclusion in each segment is a flexible way to prepare the data for analysis.

Analysis and Reporting

Analysis and Reporting obviously depend on the objectives of the study. In general remember to

• Include error estimates on all statistics at 95% confidence to give meaning to your data.
• Use clear simple language and diagrams.
• When making interpretations - use multiple sources of evidence to back up those interpretations.