Online Surveys = Ongoing Insights

The Online Research Data Quality Problem: Is it the Respondent or the Survey?

The Online Research Data Quality Problem: Is it the Respondent or the Survey?
Survey Graph

Hate the player or hate the game?  Depending on where their loyalties lie, people may fall into either camp when they consider the issue of how to maintain the data quality of online research – either the respondent is the problem or the survey is the problem.

Intuitively, one would think that the answer is that it is a little bit of both.  This is what we on the TrueSample team have been saying as well, and we have been approaching the problem from both directions.  To that end, we made a pair of bookend presentations at CASRO – at the CASRO Panel Conference in New Orleans in February 2010 and the CASRO Online Research Conference in Las Vegas back in March 2011.  You can download the whitepaper “How to Influence Good Behavior through Survey Design” by clicking here.  And, you can watch the recap video of our presentation  “How to Identify and Exclude “Bad” or Unengaged Respondents to Preserve Data Quality” here.

But what has been unavailable is real evidence on whether there actually are two types of respondents – the “bad” ones and the ones “driven bad” by the survey.  Until now.  In our recent research on engagement, we looked at the data in a slightly different way that points to this dichotomy in respondent types.  While this is not a smoking gun situation, I would say that it falls under a scenario where the barrel is warm and someone has a guilty look on their face.

In this research, we analyzed over 1600 surveys that spanned several months and product categories, and tracked several hundred thousand respondents over time.  About 20,000 of these respondents had been unengaged at least once (unengaged in the TrueSample definition refers to a respondent that either sped relative to the norm on over 40% of the pages in a survey or straight-lined while speeding on over 25% of the pages).  We looked to see at what point in the survey they started speeding and plotted that against the survey length.  In the analysis, we compared the unengaged respondents against those that sped on at least one page in the survey, but not enough to be marked as unengaged.

If there is only one class of unengaged respondent, i.e. those that are driven bad through survey design, I would expect to see a trend of the first time of speeding relative to the length of the survey – i.e. as the survey gets longer, unengaged respondents on average tend to speed later and later.  But the finding was quite remarkable in displaying the distinction between the two types of respondents.

As can be seen in the figure below, the respondents that were unengaged in the surveys started to speed right off the bat, and obviously, the length or design of the surveys had no bearing on when the speeding started.  However, the respondents that sped on at least one page but were still engaged overall tended to speed much later in the survey, with an average first time of speeding that varies with survey length.  Interestingly, the first incidence of speeding flattens out at the 8 minute mark – this appears to be the level where impatience starts to set in.  This graph clearly shows the two types of respondents – the “bad” ones that behave badly regardless of the survey, and the ones that are probably driven to do so by the design of the survey.

We will likely encounter both types of respondents in most surveys.  It is apparent, therefore, that we need to attack the problem from both sides – design better surveys through tools such as TrueSample’s SurveyScore to keep respondents engaged, and exclude bad respondents through tools such as TrueSample’s Engagement metrics to preserve data quality.

Hate the player or hate the game?  Why not just find a way to increase the love?


Tags: , , , , ,

  • Hi there, just wondering, I have 39 people who have it in their power to do my survey: but only 6 have completed it yet. It’s such a shame! I keep sending out emails reminding them to complete it; but hardly no one has yet. I made it on Friday, this is Tuesday, surely most people have got it by now????? What shall I do????

    • Bennett P

      Joy – So sorry to hear your response rates are low. Is your survey too long?

      How about adding a reward at the end? Maybe that would help? You could then tell people that they have a chance to win if they fill out your survey. Here’s more info:

  • We will likely encounter both types of respondents in most surveys. It is apparent, therefore, that we need to attack the problem from both sides

  • Miss.R

    Thanks for such a great article here. I was searching for something like this for quite a long time and at last I’ve found it on your blog. It was definitely interesting for me to read about web applications and their market situation nowadays. thanks one

  • Miss.R

    Thanks for sharing this information, i truly love your weblog. Keep this good work & enlighten us with your new post. Thanks.

    • Sheila G

      Thanks Miss R. We appreciate the love!

Inspired? Create your own survey.

Inspired? Create your own survey.

PRO Sign Up Sign Up FREE or Sign in

Write Surveys Like a Pro

Write Surveys Like a Pro

Ever wonder what SurveyMonkey’s really made of?

Ever wonder what SurveyMonkey's really made of?

Read our engineering blog »