Pre-Live Checklist
You've set up your online survey, everything looks ok and you are about to get respondents to complete its.
Before you invite people you should check the following..
Panel Entry and Exit codes
! If you don't
It is unlikely that the panel provider will allow your study to proceed as they usually check these. Otherwise you will have great difficulty in reconciling payment with your panel provider for actual completes.
If you are using a third party panel to invite respondents, you will likely have been given a series of exit codes and possibly invitation codes.
These codes allow the panel to track which of their panel have participated in the survey and how they exited - screen out, successful completion or over quota.
Some panels also require that you upload a set of authorised id's into your system to ensure only those respondents on the list are allowed to participate.
Each panel provider is slightly difference so you will need to check their scheme.
Quotas
! If you don't
You may get more respondents than you need. If you are paying for their time, this can be a costly oversight.
Are all segment quotas and an overall quota to the survey set?
What will happen if the survey is closed or over quota, what will the respondent see?
Data Simulation
! If you don't
You are taking a chance that manual testing is good enough. In the worst case the entire sample will have to be re-invited because of data collection errors. For certain time-critical and hard to get one-off samples - such as exit polls this will result in the total failure of the project.
Before collecting data from real respondents, you can check that your survey is collecting data properly using a simulation bot or spider.
This is an automated webbot, much like what google uses to trawl web pages and links. The webbot 'reads' each page of the survey and randomly provides responses. This allows the generation of potentially thousands of simulated responses.
By examining the data produced by the webbots one can spot errors in the survey coding, especially in surveys with a lot of branching and conditional logic.
Is a respondent timeout set
! If you don't
You will not be able to estimate likely completions and the data may contain respondent data you don't want but are paying for.
A timeout is a period of inactivity that you decide means the respondent is no longer engaged. Most online survey tools should allow this to be set, typically at 30 minutes. Respondents who do nothing for this time will be prevented from completing the study.
A timeout is useful for several reasons. Firstly, it allows you to reasonably judge whether someone is active and is likely to complete, or whether they have closed their browser and will never complete. This statistic is critical for deciding whether to boost a sample.
Secondly, it allows you to exclude from the final data respondents who are not fully engaged with the survey material.
Depending on how complex and long the survey is, and how motivated the respondents are, about 10% to 20% of respondents give up.
Response validation
! If you don't
Your data may have missing fields or incorrect data
Have all the responses validation rules such as mandatory response and free text input checking e.g. numbers, email addresses etc. been set.
Progress indicators
! If you don't
Your drop out rate may be higher
Respondent's like to know how much of the survey they have done and how much more is left to go. Inclusion of a progress bar or page number increases the chance that they will complete the survey.
Server Load
! If you don't
you risk burnt sample, unhappy clients and burnt sample.
Will inviting your sample crash your server? You should know how many survey pages a minute your server can handle before noticeably slowing down. This is dependent on both the server hardware and the software you use to run surveys.
A second factor to consider is that responses are not constant over time. Below is an example of hourly response rates for a survey server running multiple concurrent surveys.
In addition, there is a 'burst factor'. A very large proportion of the entire sample will respond within the first hour of invitation.
Consider the worst case, that is 50% of the total sample responds at the same time and ask your service provider or technical department whether this will overload your system.
Clear Test Data
! If you don't
your data will be polluted with test cases that will have to be removed - if they can be identified later. It will also affect your ability to manage quota completions
Final Proof Read
It's obvious but its entirely likely that comments such as 'to be removed in live' don't get removed. Make yourself a cup of coffee, take a deep breath and even though you've been through the survey a hundred times, imagine you are a respondent and you are doing it for the first time.
Have you...
- checked that the introduction makes sense?
- told the respondents how long the survey will be?
- put a contact number for support or enquiries?
- if they are rewarded with points or dollars - are these still correct?
- included at least one free text entry question where respondents can provide feedback?
- put a final thank you page in?