For obvious reasons, we’re pretty crazy about data collection over here. But even more exciting than collecting data (be honest, you know you get a little thrill when you see survey results start pouring in) is what you do with that great big bundle of information at your disposal. When we heard about a survey that was recently conducted by Smartsheet, a popular online project management and collaboration tool, we just had to share. By testing three different subject headlines, the Smartsheet team was able to determine which messaging appealed most to their customers and helped maximize their response rates.
Smartsheet’s online application provides a wide range of project and task tracking needs and as a result, they’ve acquired 10k+ customers who represent a wide range of industries, are located in 50 different countries, and range in size from small businesses to Fortune 500 companies.
Since their customers use the application for a wide variety of project and task tracking needs, the team at Smartsheet struggled with how to prioritize their proposed product enhancements and found themselves repeatedly asking themselves following questions:
- How will specific features improve customer adoption?
- Which features will produce revenue the soonest?
The team needed a clearer picture of which feature improvements their customer base was most interested in seeing. They identified several factors that would influence the importance of each user’s ranking, such as: industry, company size, and how Smartsheet was being used within the organization.
To help the team refine and confirm its product priorities and roadmap, Smartsheet decided to create a survey to send out to their customers.
Selecting the Best Questions
To provide the best possible user experience for their customers, the team chose to keep the list of questions to a maximum of 10, primarily multiple-choice, questions. The goal was for the survey to take less than 2 minutes to complete which was important to maximize response rates and also respect their customers’ busy schedules.
Using Question Bank as a guide, the team had crafted and finalized the survey within a matter of hours. “Everything about the process was straightforward — from choosing the template design to setting up the question structure,” said Brent Frei, Smartsheet.
Testing for the Best Subject Line
The Smartsheet team has a deep commitment to testing. They test product UI graphics, customer engagement campaigns, videos, advertising copy and so forth. “While we think we might know the right direction, there’s nothing better than running an A/B test to prove it out,” said Frei. “I can’t tell you how many times the test results have completely changed our course.”
So when it came to the survey, the team wanted to run a test to see which subject line would generate the highest response rate. They took a small, random sample of customers and tested 3 different approaches.
- Payback Reward: “Give feedback – Win $100”
- Pleading: “Smartsheet Needs Your Input”
- Benefit Yourself: “Help Smartsheet Get Smarter – 2 Min!”
There were slight variances in response rates based on size of company and how frequently they used Smartsheet in their companies. So, when it came to send out the full survey, the team opted to use a combination of the “Give feedback – Win $100” (which pulled slightly better with the customers whose usage patterns were lighter) and the “Help Smartsheet Get Smarter – 2 Min!” (which generated the best response from the heaviest users) approaches. Ultimately, the team decided to include the Rewards Instant Win Game in their survey, which was described to customers in the survey introduction. As for the subject line, they featured the “Help Smartsheet Get Smarter – 2 Min!” wording because of the test results and the customer segment that version appealed to.The team designed an email with an embedded survey link, queued it to send at 5am EST, and sat back and watched as the results flowed in. A day and a half later, the response rate topped an impressive 13 percent.
What Happened Next?
Interested in the SurveyMonkey best practices that Smartsheet used? Click the following links to see how the number of questions added and the length of time a survey takes to complete can affect the response rates of your surveys.
To learn more about Smartsheet, visit their website at www.smartsheet.com.
Have you done any interesting testing with your data? Let us know in the comments section below.