Online Surveys = Ongoing Insights

Stop Satisficers From Wreaking Havoc On Your Survey Results

 → 
 → 
Stop Satisficers From Wreaking Havoc On Your Survey Results

Often, all researchers see when evaluating the outcome of a study are survey results. Underneath any mean, however, there are genuine responses, as well as those that were made haphazardly, without careful thought. We explain the havoc these satisficers can wreak on your survey results in a previous blog post. The danger, ultimately, is that satisficers in your sample will pull your means in all sorts of directions, leaving you with unreliable data.

The image below shows this process in theory. The red line is the satisficing group and the blue line is the “true” group. The purple dotted line shows the average of the two groups. (Illustrative only, not real data.)

online survey results are more accurate when respondents don't satisfice

As you can see, satisficers answer very differently from genuine respondents. Leaving their survey responses in your sample may lead to misleading results, which would lead to the wrong conclusions. Therefore, it’s essential to be able to separate the satisficers from genuine respondents. Previous attempts to do this haven’t always been successful. At SurveyMonkey, however, we have developed a procedure to find the satisficers on any survey.

Our satisficing detection procedure depends on a statistical theory called Bayesian inference, with a bit of “machine learning.” Essentially, it will examine a data file to recognize a “normal” pattern of results. The machine-learning portion can either learn from the first portion of observations entered into the database if they are added in real time, or it can analyze an entire set if all data have been collected.

One easy way to understand this is if a person checks the ‘male’ box for gender, but the ‘yes’ box for “have you been pregnant in the past three years,” that would not be an expected response from a respondent carefully filling out the survey. Multiple mismatches in responses across a questionnaire with, say, five questions or more would suggest (though not guarantee) satisficing. The more outlandish the combinations of answers, the more likely the person isn’t who we need in our data-set—especially if we are doing research to understand what most people think, on average.

We’ve tested this procedure to see if it actually can find satisficers, and we’ve found that it does very well. As we are in the finishing stages, we will be able to share more details about the tool and what it will look like later on. For now, let us know what you generally think of this new tool in the comment section.

Tags: , , , , , ,

  • Michael C.

    We experience this quite often, though the numbers appear to be relativly small – less than 1% of all respondants. We often find a conflict between their answers – all negative/disatisfied – and their comments – positive/satisfied. I leave them as-is, since it’s not a (bio)logical impossibility like the example in the article.

  • Hi! Do you know if they make any plugins to help with SEO?

    I’m trying to get my blog to rank for some targeted keywords but I’m not
    seeing very good results. If you know of any please share.
    Thank you!

  • Doug Penhallegon

    Interesting! When will this be available, and under what plan? Will it work automatically behind the scenes, or will we have the option to activate it?

    • KTsurveymonkey

      Hi there, Doug-

      Thanks for checking out this post! Apologies, but could you provide more information on the feature you’re referencing here? This post reflects a theory and how that theory (Bayesian inference) is applied to data. so, this is not a feature per say, but a notation that can be applied to your data. Hope that helps clear things up!

Inspired? Create your own survey.

Inspired? Create your own survey.

PRO Sign Up Sign Up FREE or Sign in
Write Surveys Like a Pro

Ever wonder what SurveyMonkey’s really made of?

Ever wonder what SurveyMonkey's really made of?

Read our engineering blog »