Online Surveys = Ongoing Insights

Survey Priming & Timing: Small Changes Can Have a Big Impact on Bias

Survey Priming & Timing: Small Changes Can Have a Big Impact on Bias

survey_primingIt’s possible you’ve heard this one before: Go hungry to a grocery store and you’re likely to come out with a shopping cart full of everything in the store except the refrigeration cases. Go on a full stomach, and you’re more apt to stick to your grocery list.

Why are we talking about shopping? Well, just as a person’s physical state can influence how they shop, a person’s mental state can influence the way they respond to your survey. Their answers—and your results—might differ dramatically based on priming (psychological effects of question order) and timing (when you send your survey).

But with some planning, you can minimize the effects of priming and timing so you get results that accurately reflect your respondents’ attitudes and behaviors. Here’s how.

When is the best time to send a survey?

timingsurveysIf you’ve got a decision to make, you know that collecting accurate survey responses will help you make the most informed choice. And if you want to collect honest data, you need to really think about the effects of when you send your survey.

For example, say you work at an electronics manufacturer and your boss asks you to conduct a survey to measure your company’s Net Promoter® Score (NPS), a standard measurement that companies around the world use for benchmarking customer satisfaction and loyalty. The timing of when you send out your survey could yield different results for the question, “How likely is it that you would recommend this company to a friend or colleague?”

If you send out the survey out to customers immediately after they purchase your product, your customer may not have had enough experience with the item to effectively answer whether they’d recommend your company to a friend or colleague. They also might still be in “shiny new toy” mode and give you a higher rating than they would if you sent out the survey a few weeks after their purchase. But, if you wait too long, they may have forgotten all about their experience with the product also making it hard for them to effectively answer that question.

Keep your survey goals in mind

So when should you send your survey? The ideal time to send out your survey is determined by your survey’s goals. Want to assess how their experience was with the purchase process? Send it out immediately after they leave the store. Want to assess their satisfaction with the product? Send your survey a few weeks after their purchase so they’ve had some time to get their hands dirty with the product.

Another issue with timing is that often things happen that you couldn’t plan for in advance, which may impact your results. For example, if there is a recall for a product your company manufactures, the NPS of your company could change dramatically.

In this situation, you can take steps in the analysis process to still have meaningful data. Examine the responses before the recall announcement to get a sense of sentiment before the recall. Take a look at the responses after the recall announcement to determine if the recall had an impact on your customer loyalty. If it did have a negative impact, you still have your previous NPS benchmark to now use as a goal for your company to aspire to.

How to approach survey priming

So that takes care of timing. But what is priming?

Good looking adult woman working at the office. Probably waiting for lunch...Priming definition: In terms of psychology, priming occurs when a person is exposed to a stimulus, and it influences how they respond to another stimulus. For example, if you were to see the word “yellow,” you’d be slightly faster than other people (who hadn’t seen that word) to recognize the word “banana.” That’s because people closely associate words like “yellow” and “banana” in their memory.

Let’s go back to the grocery store example—we mentioned that the state at which you enter into the store might have an effect on your purchases, but the layout of the grocery store may also influence what you buy. If the store displays fresh figs right in the entrance, you’ll be more likely to purchase figs on your trip than you would be if they were only in the produce section.

Similarly, you may be more likely to buy those chips that are right near you in the checkout line than if they were just in the aisle. Surveys are the same–depending on how you lay out your survey or order your questions, your results could differ.

If you specifically mention the device that the individual purchased in the email that invites them to the survey or in the survey introduction, they’ll be “primed” to be thinking about their experience with that specific item. With no mention of the item, they may be thinking about their experience with all the electronics they’ve purchased from your company, which may yield different results.

Be smart about survey question order

Think back to your survey goal. Are you trying to get a read on the overall sentiment about your company? Then don’t mention the item purchased. Trying to get a read on how the individual product impacted their feelings about your company? Go ahead and mention it!

We’ve already discussed how images can influence the results, but the order of the questions can also make a difference:

Ask the NPS question first and you get a top-of-the-mind reaction to your company. This helps you get a measure of what someone might say to a friend about you without carefully considering the many different factors that contribute to their overall feeling towards your company.

But what if you ask the NPS question after you’ve asked more specific questions about the service at the store, the check-out experience, the materials the product was made of, and how useful the product is? Well, that means that the respondent is primed to think about all of those factors when responding to the NPS question, giving a more holistic opinion of your company, which may be different from their top-of-the mind view.

Just like the timing issue, the right way to order your survey will depend on your goals and whether you want to get “snap judgments” about your company or whether you’d prefer to get an “informed review.” There’s a case for both.

Get the Feedback eGuide

Tags: , , ,

  • yousef

    hi i want to collect data from international tourists (not domestics) who stay in 4 and 5-star hotels in kuala lumpur how shold i get help from survey monkey?

    • KTsurveymonkey

      Hi there, Yousef!

      Try reaching out to our Audience sales team. I’m not entirely sure we have panelists in Kuala Lumpur, but it’s worth checking! You can contact them here:

      Good luck, Yousef!

  • dandassow

    This is clearly written article on the how to most effectively time a survey to satisfy the objects of the survey. As with any objective study, understanding and defining the objectives of the study are critical in the design and analysis of the survey. It is also evident that the survey designer and analyst need to understand the environment in which the study is conducted.

    Ideally, when benchmarking satisfaction and loyalty, it is important to conduct the survey over time in order to determine the impact of promotions and negative events. It is incumbent upon the analyst to collect relevant information on the survey environment in addition to demographic information.

    It occurred to me while reading the section on survey order, it may be appropriate to have two or more versions of the survey if the sample population is sufficiently large. With multiple surveys, the order of the question can be changed and the impact of question order can be determined. Sarah C and KTsurveymonkey, are you aware of any specific studies that use multiple surveys to measure impact of survey priming and timing, and question order? Can you provide any guidance on survey test design when using multiple versions of a survey?

    • KTsurveymonkey

      Hi there, Dan!

      Great question. We reached out to our blog writer, Sarah Cho, to answer this for you:

      Not sure too sure if you want to either get rid of order bias OR test if order bias exists- so I’ll address each option for you! To get rid of order bias, I’d recommend randomizing the order in which the questions appear or use page randomization, which will help minimize the effect of order bias.

      If you are looking to test if order bias exists, instead of having two versions of the survey (which is totally an option), you could consider using A/B testing (sometimes called half sampling) to test to see if order bias exists. That way you can see if having Q2 appear before Q1 actually makes a difference. I’d also advise against having more than two versions of a survey though since that would make it super complicated in the analysis phase and it may be harder to tease out order effects.

      We hope that helps, Dan! Feel free to follow up if you need more guidance.

      • dandassow

        Sarah and KT,

        Thank you for responding so quickly and answering the intent of my question. What you proposed is close to how I would have approached the problem. I proposed the question as a Socratic way for you to expand upon your excellent article.

        Please extend my congratulations to Sarah Cho for being named President of PAPOR (Pacific chapter of the American Association of Public Opinion Research (AAPOR)).

Inspired? Create your own survey.

Inspired? Create your own survey.

PRO Sign Up Sign Up FREE or Sign in
Write Surveys Like a Pro

Ever wonder what SurveyMonkey’s really made of?

Ever wonder what SurveyMonkey's really made of?

Read our engineering blog »