Here’s how our research team ensures your surveys receive thoughtful, consistent responses.
From December 2015 through February 2016, we sent the same monthly survey to respondents in our SurveyMonkey Audience panel. Each survey ran for 1 week, included 25 questions, and received more than 1,000 responses.
We compared the results for each wave, weighting for age and gender, to see if we got meaningful data back from our respondents. And we built our survey in order to test for 3 respondent behaviors that may negatively impact data quality:
In addition to monitoring data quality, we wanted to make sure results were consistent from one wave to the next. Why? Even if members cycle in and out of Audience, we always want to be sure we maintain a panel that’s reliably representative of the general population.
Some of the questions we asked were included specifically because their responses should not change much over time, such as questions around:
We also checked to make sure the amount of time respondents took to complete the survey did not change significantly from wave to wave.
As we mentioned before, in order to determine whether or not our panel yields quality data, we measured 3 key metrics: straight lining, open-ended responses, and a trap (picture-identification) question. Here’s what we found:
As you can see, 97% of the respondents in the U.S. passed our straight lining test. This means almost all respondents carefully differentiate the questions in a matrix (grid) question type when providing answers. Similarly, about only 2–3% of the open-ended responses are invalid or gibberish, while the majority of respondents provided a valid and meaningful answer.
We also included a trap question in the survey to test whether respondents paid close attention to the question—and found 94% of respondents passed this quality check as well. The question was asked more than halfway through the survey, suggesting that most respondents still paid enough attention to instructions and picked the correct answers when they got close to the end of the survey.
Next we examined whether the 3 surveys over the course of 3 months provide consistent findings. We tested for this by asking questions that should yield stable answers in terms of proportionality. For example, a sample should always include roughly the same percentage of people who say they have green eyes.
In the table below, we’ve included responses to these questions. For one question, respondents could “select all that apply,” so we show each of those response options separately. A check mark indicates no significant change over the 3 waves.
Most of the questions show a consistent pattern over time, suggesting the data collected from Audience are reliable and trustworthy. From 1 wave to another, of the 23 indicators we tracked, 20 did not change significantly.
|I have green eyes||3|
|I have naturally red hair||3|
|My birthday is in February||3|
|I have a twin||3|
|I am right-handed||3|
|I am less than 6 feet tall||3|
|I wear corrective lenses (glasses or contacts)||3|
|None of the above||å|
|Own a credit card||3|
|Own a debit card||3|
|Used a credit card in the past year||3|
|Used a debit card in the past year||3|
|Moral acceptability of alcohol use||å|
|Moral acceptability of homosexuality||3|
|Picture verification question||3|
|Motivation for taking the survey||3|
|Rating of survey quality||3|
|Time to complete||å|
|Device completed on||3|
Quality: The 3 data quality indicators we examined all indicate SurveyMonkey Audience respondents provide thoughtful, high-quality responses. Only very small percentages of respondents failed our data quality checks.
Reliability: Over the 3 survey waves we ran, our results did not change substantially, indicating SurveyMonkey Audience provides consistent survey results representative of your target demographic regardless of when you’re conducting your survey.
This means you can survey away with confidence knowing you’ll get results you can trust! And remember: Follow Audience’s survey design guidelines to increase the likelihood of receiving thoughtful answers.