Pilot Testing Questionnaires

It doesn’t matter whether you’ve previously built lots of questionnaires, or you’re working on your very first one. You’ll want to maximise its engagement with respondents and the quality of insight you can obtain from it. But you’re very unlikely to be able to achieve that with your first attempt, especially if you’re new to surveys.

The key reason for this is that even if you’re an expert on the topic of your questions, without prior input from a sample of your target audience, it’s unlikely you will know exactly how they will respond to your questions.

This is where the value of questionnaire pretesting and piloting at the survey planning stage, can really pay dividends.

Pretesting vs pilot testing

When it comes to survey testing, some people have the misconception that it takes too long to do. Meaning they end up running their survey without any testing at all. This is a huge mistake. Even testing it with a couple of people could throw up issues, that if not spotted and solved could cause real problems with your data later on. So, any testing is better than none at all.

Pretesting

Having completed the first draft of your survey questionnaire, there are a number of steps you can take to improve it.

Question critique

Re-examine the wording of your questions to see if they contain anything potentially problematic for the reader. This could include double-barrelled questions (essentially two questions in one), negative questions (those containing negative words) and overly long questions.

You also want to assess the flow of your questionnaire. Check that your question order and any transition statements make sense.

Informal testing

Try interviewing yourself, where you put yourself in the position of the respondent and try to answer each question. You might also consider doing a mock interview of the questions with a close colleague. Both methods can enable you to identify potential difficulties in your question-answer process.

Enlist further expertise support

While you may be reasonably knowledgeable on the subject area of your written questions, there could be others in your business with more detailed expertise. They could also have expertise in specific areas such as questions design and testing, field issues and the cultural perspectives of the survey.

Whatever beneficial knowledge they have, try getting a small panel of them together to review your questionnaire. This can provide some really useful initial feedback for you regarding any potential problems with your survey questionnaire.

Systemic review of the questionnaire

If your organisation has been creating questionnaires for some time already, you may have a checklist in place to assess whether you’re on the right track. This can be very useful in quickly identifying any flaws that might exist with your questions.

These pre-test steps offer an inexpensive way of quickly uncovering a range of potential issues you need to fix. However, while they are dependent on the capabilities and availability of experts involved, there are some limitations. Most notably due to the lack of feedback from the audience you’re looking to target. That’s where primary research methods such as pilot testing can add real value to your planning and testing.

Pilot testing

Pilot testing enables you to assess your whole questionnaire under survey conditions. The primary benefit of pilot testing is to identify problems before implementing the full survey. Pilot testing looks to examine the validity of each question. It’s concerned with whether the question is capturing the information it’s intended to measure. Not only to meet the goals and objectives of the study, but analysing various aspects of the whole questionnaire, such as:

• Analysing the question flow. Essentially, assessing the arrangement of questions, and whether they support the flow of the piece.
• Assessing how well it meets expectations, in terms of the response rate at first contact.
• The average time it takes respondents to complete each question.
• The length of the questionnaire and how quickly respondents are completing it against the time you originally estimated for it.
• Checking for any instances of confusion. Respondent trouble with understanding questions could result in many incorrect answers, respondents taking too long reply or skipping questions altogether.
• Checking the question types used and whether you need to tweak them, in order to improve clarity and response time.

The pilot test of your questionnaire which should be carried out under survey conditions, should be tested on a sample size that is representative of the target population for your research. Besides completing your survey, your respondents should offer up their feedback about their experiences of filling it out.

Ideally you want to be able to get all your respondents together to observe their behaviour while they take your survey. Making notes throughout, look for places where they hesitate or make mistakes. You can also record the overall time it takes each participant to complete your survey. Much of what you find could indicate that your survey questions and layout are not clear enough and need improving.

For anyone struggling to find enough respondents to carry out a test from their existing contacts, a consumer panels service is a good alternative. With their ability to provide instant access to millions of respondents worldwide, consumer panels are great for reaching the exact demographic or niche audience you require.

The advantages of pilot testing questionnaires

While we have already touched on how pilot testing can help with a number of aspects concerning your questionnaire. Here are some more key advantages to pilot testing your questionnaire.

Important considerations for your pilot test

Finally, there are different pilot approaches from an organisational and respondent point of view that you need to consider. It’s important to be aware of these differences before you proceed with your own pilot study, as each offers their own distinct advantages in different situations.

Internal or external pilot study?

Looking at pilot studies from the organisational standpoint first, there are two main types of pilot study, internal and external.

With the internal pilot study, respondents in the pilot are also considered as the first participants in the main survey. In contrast, with an external pilot survey the questionnaire will be administered to a small group of target participants who are not included in the main survey.

Internal pilot studies are generally more efficient, if you’re only looking to change feasibility aspects such as participant recruitment and not the trial design. While external pilot studies offer greater flexibility to change the trial design for the main study once the pilot is complete.

Undeclared or participatory?

It’s also important to consider the approach you want to take when it comes to respondent participation.

There are two main types of respondent approaches to consider – undeclared and participatory.

With an undeclared pilot, the participants are not aware that they are part of a pilot group. And you issue this survey as if it were real. This type of pilot is useful for reviewing the results you’ve received and ensuring there’s no issues with survey completion.

In contrast, with a participatory pilot participants are fully aware of their participation and clearly informed about what they need to do. This pilot type is useful when you require feedback from the target audience about your survey content, processes and messages.