Order Matters: Don’t Let it Get In Your Way

surveyscience

Pop quiz!

Read this list of words quickly: pen, airplane, shell, butter, duct, blueberry, wrapper, zebra

Now cover the list with your hand.

Now list all of the words you remember.

If you’re like most people, you’ve remembered the first one (pen) and the last one (zebra), but not so many in between. How did we know that? Well, the order of a list can be just as important as the actual content. The order itself changes how you think and respond.

The same thing happens in surveys when you give respondents a list of answer choices. The first and the last answer choice stick out, and the middle tends to get ignored. Take this question for example:

Best Young Adult Literary Series?

More respondents will pick Harry Potter and Percy Jackson than would have otherwise simply because they are the first and last choices. In survey language, that’s called a response choice order effect.

Why might people pick the first response choices?

  1. Satisficing. Respondents start with the first option, thinking about all the reasons the Harry Potter series might or might not be the best series of the last ten years. They then do the same thing for the rest of the options on the list. As respondents go down the list, they start getting mentally tired. They then begin to satisfice–they stop paying close attention and skim over the last options in the list, neglecting to consider them carefully. This can leave the top response choices in the list with a much stronger case than the bottom ones.
  2. Priming. We talked before about how this works with questions. Ask someone their income and that’ll sway how they respond to a question asking if they believe in the American Dream. Believe it or not, the same thing happens with response options too. For example, if Harry Potter is first on the list, respondents might first think about how the series made billions of dollars. They’ll then use this same standard–which is pretty unique to Harry Potter–to judge the other options in the list. None of the other options made billions of dollars, so Harry Potter is left as the easy choice.

Why might people pick the last response choices?

  1. Satisficing. Yes, satisficing rears its ugly head here too. Again, respondents do the hard critical thinking work for the first few options, but get mentally tired and begin to satisfice. When they stop thinking critically about the last few choices, this can actually make them seem more appealing, not less. It becomes more of a gut instinct selection than a rationally thought-out opinion.
  2. Recency. Sometimes it’s just the last thing you hear that sticks in your mind. It’s not because you’re being sloppy or because you’re not thinking clearly but just because sometimes the last thing that you’re hearing seems the most compelling. Think of it as years of conditioning that you always “save the best for last.”

Does response order REALLY matter that much?

YES! For example, studies of voting have shown that the order of candidates’ names on a ballot affects vote totals. For example, in the 2000 presidential election, George W. Bush got 9 percent more votes when he was listed first on the ballot than when he was listed last–that’s a huge difference! This is one of the reasons that ballots randomize the order of candidates’ names.

Okay, then how do I fix it?

  1. You can randomize the options, so that each person sees the list in a different order.
  2. Sometimes, if you have an order list or a rating scale, randomizing the options can feel awkward. In this case, you can flip the options, so some of your respondents see a list that begins with Harry Potter and ends with Percy Jackson, and others see a list that begins with Percy Jackson and ends with Harry Potter.

If you randomize or flip the options, any order effects will balance themselves out and your data quality will be protected! And guess what? Both of these solutions are just a click away with SurveyMonkey.

Be sure to let us know how it goes in the Comments section below!

Tags: , ,

Inspired? Create your own survey.

Inspired? Create your own survey.

  • Polymars

    This is a good article. When there are 4 choices, as in your film choice question, would the respondent be more inclined to a balanced response if the choices were split into 2 sets of 2 and then and then a 3rd eliminator question?

    • kaytek

      Hi there,

      A creative suggestion but we recommend against complicating it too much. The best way to go is always to randomize the order of the responses, rather than try to guess how the respondent will be biased by the order. That way, biases wash out across your whole sample.

      Thanks for the question!

  • http://www.wordnerd.co.uk Sally Mayor

    Thanks for that, great blog and useful advice. I remember learning about the recency and latency effect at Uni (many years ago). Thanks again.

    • kaytek

      You’re very welcome!

      • Ali Ahmed

        Hi kaytek, i need some serious and continous help from you, as i am working on a small survey related project for my university ….

        if u can just leave an email from your email account, we can have our discussion on emails, so that this blog doest gets a personal help forum…

        My email sshjssh@gmail.com

        Awaiting ur response

        • kaytek

          Hi Ali,

          We’d be happy to help. If you could kindly send a message with the email address connected to your account to support@surveymonkey.com, our team will respond asap. Usually within 2 hours, often less.

          Thanks!

  • http://www.octarine.co.za/ Ann Druce

    Practical, usable advice. And it upsells too -for me, this adds value to the paid version.)

    • kaytek

      Awesome, glad to hear it. :)

  • Folkestone Winter Shelter

    great article, very informative thanks

  • kaytek

    Hi there, with answer choices like these, we recommend not randomizing these to keep them methodologically sound. Start with Strongly Agree, Agree, No Opinion, Disagree and Strongly Disagree.

    Check out this quick video tutorial on rating scale best practices here: https://www.surveymonkey.com/blog/en/blog/2013/06/10/jumpstart-video-series/

    And from our Help Center: http://help.surveymonkey.com/articles/en_US/kb/What-is-the-Rating-Average-and-how-is-it-calculated

    Hope this helps!