Online Surveys = Ongoing Insights

Survey Scientists Share Their Latest Research

 → 
 → 
Survey Scientists Share Their Latest Research
SurveyMonkey AAPOR

Mmm, data and tacos

Just in case you didn’t notice, we gave 10 presentations at AAPOR—THE place to go if you’re a survey research professional—and our team was thrilled to not only attend again this year but excited to share our research with colleagues and friends.

And don’t fret, in the next few weeks, you’ll read even more blogs from us on things we talked about at AAPOR, so get ready to be in the survey science loop!

Here’s the first study and a breakdown for you here.

Take a look at these four questions

  • In general, do you think now is a good time to buy a house or do you think now is a bad time to buy a house?
  • In general, do you think now is a good time to buy a house, or not?
  • In general, do you or do you not think now is a good time to buy a house?
  • In general, do you think now is a good time to buy a house?

They all ask about exactly the same topic and use the same response options: Yes or No. However, it’s clear that the question wordings aren’t the same. The first question asks both sides of the viewpoint, which we call a fully balanced scale.

The second and third questions both present only one side of the viewpoint and abbreviate the other end of the viewpoint with “or not” or “or do you not.” We call both of those a minimally balanced scale. The last question only presents one side of the viewpoint and omits entirely the other side. We call this type of question an unbalanced scale. Got all those straight?

Now the question we’re most interested is whether these different ways of presenting a scale change how people answer the question. To find out the answer, we conducted an experiment where we randomly assigned participants into one of the four question wordings and then we examined their answers.

As you can see from the plot below, the answers provided by the respondents were very similar across the four conditions. A statistical test (chi-square test) showed that the differences aren’t statistically significant.

aapor_chart

We actually tested 16 questions in two survey experiments and all of them showed no significant difference across the four ways of asking questions. For more details about this study, see here.

3 reasons these results should matter to you

#1: Always test and see if you can shorten your questions. If you don’t see a difference, go with the shorter/unbalanced one.

#2: Consider the fast increasing number of mobile respondents in web surveys, the limited screen size that exists and always make your questions short—Laura will scold you otherwise.

#3: Because you should be using unbalanced scales in your surveys. We’ve got proof from two different studies to share. Study 1 and Study 2.

Remember, we’ll be back for even more survey science knowledge so be sure to check back in with us here on the blog.

Questions for Mingnan or the Survey Research team? Leave them down below!

Tags:

Inspired? Create your own survey.

Inspired? Create your own survey.

PRO Sign Up Sign Up FREE or Sign in
Write Surveys Like a Pro

Ever wonder what SurveyMonkey’s really made of?

Ever wonder what SurveyMonkey's really made of?

Read our engineering blog »