Back in the days when telephone surveys ruled the earth there was a lot of research on the order in which questions were presented to survey-takers. Since control of question order was completely in the hands of the experimenter, it was important to get the order right.
A common type of question in telephone surveys (and in online surveys too) is called a part-whole question, where you ask a general question (How satisfied are you with your life right now?) as well as some more specific questions on the same topic (How satisfied are you with your job? How satisfied are you with your romantic relationships?).
Previous research showed that the order in which you ask these types of questions on the phone probably affects the answers. This makes sense since you don’t know what questions are coming next, and you can’t change your answers to previous questions.
But, does question order in part-whole questions matter in an online survey? Our hypothesis was that paging would have a greater effect than question order for online surveys. In an online survey with all of the questions on the same page, the question order doesn’t matter because we scroll up and down, reading all of the questions before answering any of them. We also have the option of changing answers as we please.
However, if you place questions on separate pages and don’t allow respondents to go back to previous pages the answering experience is more similar to phone, and question order matters again.
To test this, we set up an experiment with four conditions using SurveyMonkey Audience. Conditions 1 and 2 tested whether question order alone mattered. Conditions 3 and 4 tested whether paging mattered.
What did the results show? If all questions are on the same page, the order of questions doesn’t change how people respond. There were no significant differences between Condition 1 and Condition 2 on the general question (life satisfaction). However, when the general question was on a separate page from the specific questions and respondents were not able to go back to previous pages, there was a significant difference in life satisfaction (Conditions 3 and 4).
This necessarily mean however that one order makes people more or less satisfied with their lives. Instead, think of it like this—showing questions in a different order makes people think about and answer the question differently.
Why does this matter to you anyway? Here’s why:
- Don’t worry about question order if everything is on the same page. You should still think about survey flow (try not to jump around between topics) but respondents will probably scan the whole survey before they start answering questions.
- Think about paging. If there is a question you want to know the answer to before you introduce new information (because you think the new information will make them change their mind), make sure to put the new information on a new page.
- Q1. How much do you like Brand A?
- Q2. How much do you like Brand B?
- New Page
- Did you know that both Brand A and Brand B are both owned by Company C?
In a nutshell, think short.
If you keep the majority of your survey questions on one page, then the order of the questions matter less. A new page means new opportunity. Respondents seem to pay more attention when new information is presented to them on a separate page. Maybe even the act of clicking Next gives our brain a heads-up that new stuff is coming.
We’ll save that for another experiment on another day however.
Questions for Jillesa? Let her know in the Comments section below!