Ask more, know more, do more

A Technical Look: How We Created Question Builder Suggested Questions

A Technical Look: How We Created Question Builder Suggested Questions

As an engineer here at SurveyMonkey, I have a cool job.  I get to build features such as the suggested questions autocomplete, and I thought I’d share how we built this new feature directly into our Question Builder to make it even easier to use Question Bank.

One of the main goals of autocomplete is suggesting highly relevant questions to our customers as they are designing their surveys, which is a lot harder than it may seem!  To aid us in this, we have our great Question Bank from which we can draw questions.  So how/where do we start?

Step 1:  Stemming your Question

As a user is typing, we will get characters that we can split the text into what we consider distinct words. However, once we have a set of words we have our first problem: Say you typed the word ‘companies’ but that really means the same thing as ‘company’ but they aren’t the same word, how do we determine that they mean the same thing?

We first solved this using a concept called Stemming, which, simply put, takes in a word, such as ‘companies’ and reduces it to a base word, such as ‘compani’ for easy comparision. Since we build everything in Python here at SurveyMonkey, we are using an open source stemming library called PyStemmer.  This library works in a multitude of difficult languages, which is great!  It allows us to support English and Dutch, which are the two languages we currently support in Question Bank.

Step 2: Checking Question Bank

Once we have stemmed all the given words, we then determine which Question Bank questions these words appear in, and create some sort of ‘rank’ or ‘importance.’  We used a form of inverse frequency matching to create our rankings for each word. Here’s a good Wikipedia article for a detailed explanation on inverse frequency matching.  In short, it gives higher relevance to words that appear less often.  The assumption is that they’re more important. For example, the stemmed version of the word “company” might appear in 10 of 100 documents, whereas “the” might appear in 90 of 100 documents, therefore the word “company” should always have a higher “score” than the word “the”. By using this algorithm we create a unique score for every word in our Question Bank.

Step 3:  Checking Your Question History

Another feature we added to make your life easier is incorporating your own question history to the autocomplete. When you type a question, we look at a significant portion of your last questions added and look for matches against those. We also try to keep this as up-to-date as possible, so any changes you make to existing questions, such as editing, and adding new questions will automatically refresh your question history so we can continue to give you accurate question suggestions.

As you type words into the Question Builder, we stem all the words you’ve typed.  For each normalized word, we compute a score based on the pre-computed inverse frequency for that word.  The tally of the scores is what determines the order in which we show you Question Bank questions. Then, we use your input text to find suggested questions from your question history.  We do all of this in a blazingly fast average of 10 milliseconds!

Did you get lost on pre-computed inverse frequency?  Don’t worry.  We’ve done all the hard work for you!  Your job is to enjoy the new feature and tell us what you think!

Doug Morgan is a Software Engineer/Code Monkey for SurveyMonkey and a Hell of an Engineer!

Tags: , , ,

30 thoughts on “A Technical Look: How We Created Question Builder Suggested Questions

  1. Engida Alemu says:

    I got your website when i was filled the survey question ACCA exam feed back. it makes easy to reach different customer & to analyze data properly. i also likes to uses survey monkey for future for my oxford BSC degree research!

    1. Bennett P says:

      Engida – Thanks so much! Good luck with your degree

  2. Excellent read, I just passed this onto a colleague who was doing some research on that. And he actually bought me lunch since I found it for him smile Therefore let me rephrase that: Thank you for lunch!|Good write-up, I’m normal visitor of one’s web site, maintain up the excellent operate, and It’s going to be a regular visitor for a lengthy time.|I would like to thank you for the efforts you have put in writing this web site. I’m hoping the same high-grade website post from you in the upcoming as well. Actually your creative writing skills has encouraged me to get my own web site now. Actually the blogging is spreading its wings quickly. Your write up is a good example of it.|Good blog! I really love how it is easy on my eyes and the data are well written. I am wondering how I could be notified whenever a new post has been made. I’ve subscribed to your RSS feed which must do the trick! Have a nice day!|I am really glad I’ve found this info. Today bloggers publish just about gossips and web and this is actually annoying. A good site with exciting content, that’s what I need. Thanks for keeping this site, I’ll be visiting it. Do you do newsletters? Can not find it.|I was just looking for this information for some time. After 6 hours of continuous Googleing, at last I got it in your web site. I wonder what’s the lack of Google strategy that don’t rank this type of informative websites in top of the list. Generally the top websites are full of garbage.|Hello there, just become aware of your blog thru Google, and located that it is really informative. I’m gonna watch out for brussels. I’ll be grateful for those who proceed this in future. Numerous other folks will be benefited out of your writing. Cheers!|It’s the best time to make some plans for the long run and it is time to be happy. I’ve read this post and if I could I wish to recommend you few attention-grabbing issues or suggestions. Perhaps you can write next articles referring to this article. I wish to read even more issues about it!|Great post. I used to be checking constantly this weblog and I’m impressed! Very useful info specially the last section :) I care for such info a lot. I used to be looking for this certain information for a very lengthy time. Thank you and good luck.|hello there and thanks to your information – I’ve certainly picked up something new from right here. I did on the other hand expertise a few technical issues the usage of this website, as I experienced to reload the web site a lot of times prior to I may get it to load correctly. I have been thinking about in case your hosting is OK? Now not that I am complaining, but sluggish loading instances occasions will sometimes impact your placement in google and could injury your quality rating if ads and ***********|advertising|advertising|advertising and *********** with Adwords. Well I’m adding this RSS to my email and could look out for much more of your respective interesting content. Make sure you replace this again very soon..|Fantastic items from you, man. I’ve remember your stuff prior to and you are just extremely excellent. I really like what you have acquired here, certainly like what you’re stating and the way in which you say it. You’re making it entertaining and you continue to care for to stay it wise. I cant wait to read far more from you. This is really a terrific web site.|Very nice post. I simply stumbled upon your blog and wanted to mention that I’ve really loved surfing around your weblog posts. After all I will be subscribing to your feed and I am hoping you write again very soon!|I like the valuable info you supply on your articles. I’ll bookmark your weblog and check again here regularly. I am fairly sure I’ll be told a lot of new stuff proper here! Best of luck for the following!|I think this is one of the so much important information for me. And i’m satisfied reading your article. However want to commentary on few common issues, The website style is great, the articles is truly nice : D. Excellent task, cheers|We are a bunch of volunteers and opening a new scheme in our community. Your website provided us with valuable information to work on. You’ve performed an impressive job and our whole group shall be grateful to you.|Unquestionably consider that that you said. Your favorite justification seemed to be at the internet the simplest thing to have in mind of. I say to you, I definitely get irked while other people consider concerns that they just don’t understand about. You controlled to hit the nail upon the top and also defined out the whole thing with no need side-effects , people can take a signal. Will likely be again to get more. Thank you|This is very interesting, You are an excessively skilled blogger. I have joined your rss feed and sit up for looking for extra of your fantastic post. Also, I’ve shared your site in my social networks!|Hey There. I found your blog the use of msn. That is an extremely smartly written article. I will be sure to bookmark it and come back to read more of your useful information. Thanks for the post. I will certainly comeback.|I cherished as much as you’ll receive carried out proper here. The comic strip is attractive, your authored subject matter stylish. nonetheless, you command get bought an shakiness over that you wish be delivering the following. sick unquestionably come further formerly once more since exactly the same just about very regularly within case you protect this increase.|Hello, i feel that i saw you visited my website so i got here to “return the want”.I’m trying to find things to improve my web site!I guess its ok to use a few of your ideas!!|Just wish to say your article is as astounding. The clarity on your submit is simply cool and that i can think you are an expert in this subject. Well along with your permission allow me to clutch your RSS feed to keep updated with drawing close post. Thank you one million and please continue the gratifying work.|Its like you read my mind! You appear to grasp a lot approximately this, such as you wrote the ebook in it or something. I believe that you just can do with some p.c. to force the message house a bit, however other than that, that is great blog. A great read. I will definitely be back.|Thank you for the good writeup. It in truth was once a entertainment account it. Glance complex to far added agreeable from you! By the way, how can we communicate?|Hi there, You have done a great job. I will definitely digg it and in my view recommend to my friends. I am confident they’ll be benefited from this website.|Excellent beat ! I wish to apprentice at the same time as you amend your site, how could i subscribe for a weblog web site? The account helped me a acceptable deal. I had been a little bit familiar of this your broadcast offered vivid clear concept|I’m extremely inspired together with your writing talents as well as with the layout for your weblog. Is this a paid theme or did you modify it your self? Anyway stay up the nice high quality writing, it is rare to look a great weblog like this one these days..|Pretty section of content. I just stumbled upon your web site and in accession capital to assert that I acquire in fact loved account your weblog posts. Anyway I will be subscribing to your augment and even I achievement you access consistently fast.|My brother suggested I would possibly like this blog. He used to be totally right. This put up truly made my day. You can not consider simply how much time I had spent for this info! Thanks!|I do not even know how I stopped up right here, however I assumed this submit was once good. I don’t recognize who you are however certainly you’re going to a well-known blogger if you are not already ;) Cheers!|Heya i’m for the first time here. I found this board and I find It truly helpful & it helped me out a lot. I am hoping to offer something back and help others such as you helped me.

    1. Hanna J says:

      Great, we’re so glad you liked the post! If you’d like to subscribe to our newsletter, just sign up for a free account, and we’ll send the newsletter to the email address you sign up with. You can opt out at any time. Thanks, and have a great day!

  3. I?m certain you’ll find a great deal of added good instances in the long term for individuals who study your internet site.

  4. Wow!! Great topic and Fantastic Post, really I appreciate this.It will be very helpful for us. Waiting for more articles, blogs like this.I’m going bookmark your blog.

    1. Hanna J says:

      Hi – We’re so glad you like our blog. We’ll continue to post the latest in survey tips, tricks, and ideas, so check back in. Thanks :)

  5. sareserve says:

    A Technical Look: How We Created Question Builder Suggested Questions | SurveyMonkey Blog I was suggested this blog by my cousin. I am not sure whether this post is written by him as nobody else know such detailed about my problem. You are incredible! Thanks! your article about A Technical Look: How We Created Question Builder Suggested Questions | SurveyMonkey BlogBest Regards Cassetta

    1. Hanna J says:

      Hi Cassetta – Your cousin sounds like a smart guy. We’re glad you both enjoy our blog! We’ll continue to post about the latest developments here at SurveyMonkey, so check back in for more. Thanks!

  6. nfs world says:

    My brother recommended I would possibly like this blog. He was entirely right. This post actually made my day. You cann’t believe just how much time I had spent for this information! Thank you!

    1. Hanna J says:

      We’re so glad you like our blog! We’ll continue to post the latest survey tips, tricks, methodology, and suggestions, so check back in for the latest. Thanks!

  7. scrapetest says:

    Very interesting blog. Im happy I checked it out….will follow.

  8. Harland says:

    You’ve made some good points there. I’ve been looking online all afternoon for a second opinion and haven’t been able to find one.

    1. Hanna J says:

      Hi Harland – We admire your inquisitive spirit! Have a great weekend.

  9. Youre so cool! I dont suppose Ive read something like this before. So good to search out somebody with some original thoughts on this subject. realy thanks for beginning this up. this web site is one thing that’s needed on the internet, someone with a bit of originality. useful job for bringing something new to the internet!

    1. Hanna J says:

      Hi Carmen – Thanks so much for your nice comment :) You made this Monkey’s day! We hope you’re having a great day too. We’ll continue to post about all things survey, so check back in for the latest. Thanks!

  10. Julane M says:

    Hi guys, I am a very enthusiastic SM user. But, I have a question about the UI. If I want my survey takers to be able to rank items in importance, right now there is the forced-ranking option on the matrix. I think it’s a bit awkward and I know that other survey engines have something different for this, where a list of items is presented and the user can drag items up and down to indicate most to least important (or whatever). Here’s a link to an example (sorry to bring a competitor into the mix!).

    Will SM have something like this for forced ranking sometime soon?

    1. Hanna J says:

      Hi Julane – Thanks for the feedback! We’re working on overhauling Create and Analyze to make them much more flexible and easy for you to use. We hope you approve of what we come up with. Be sure to test it out and let us know. Thanks!

  11. click here says:

    Is it okay to insert a portion of this on my website if perhaps I submit a reference to this website?

    1. Hanna J says:

      As long as you give us credit for the information and direct people to this blog for more information, absolutely! Monkeys think everyone should have access to survey information.

  12. Hey! I just would like to give an enormous thumbs up for the great information you will have here on this post. I shall be coming again to your blog for more soon.

    1. Hanna J says:

      Hi Raymon – Thanks so much for your kind words! Its readers like you that get Monkeys so excited to create blog content. We’ll keep working to bring you the most up to date survey information, tips, tricks, and suggestions. Have a great weekend.

  13. click says:

    If you could message me with any tips on how you made your blog look like this, Id be appreciative!

    1. Hanna J says:

      Hi, We use the WordPress platform to create our blog. It’s relatively easy to use, and you can customize it with your colors, logo, etc. Good luck making your blog! Leave us a link in the comment section once you’re done so we can check it out. Thanks!

  14. Is it okay to insert part of this in my personal website if perhaps I publish a reference point to this site?

    1. Hanna J says:

      As long as you give us credit for the information and direct people to this blog for more information, absolutely! Monkeys think everyone should have access to the latest survey news and information.

  15. I think one of your ads triggered my internet browser to resize, you might well want to place that on your blacklist.

    1. Hanna J says:

      Hi – Yikes, sorry that happened to you! We don’t think our ads are that powerful, but we’ll look into it. Thanks for letting us know!

  16. Tia Eiland says:

    A formidable share, I just given this onto a colleague who was doing slightly evaluation on this. And he actually bought me breakfast because I found it for him.. smile. So let me reword that: Thnx for the treat! However yeah Thnkx for spending the time to debate this, I feel strongly about it and love studying extra on this topic. If doable, as you grow to be expertise, would you thoughts updating your weblog with extra particulars? It is extremely useful for me. Large thumb up for this blog put up!

    1. Hanna J says:

      Hi Tia – We’re thrilled to be the (indirect) source of your free breakfast! And happy to continue posting about this and similar topics. Just keep bringing us your feedback, and we’ll continue providing you with relevant survey information. Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *

Inspired? Create your own survey.

Make your own survey

Sign In

Sign up to get started

PRO Sign Up 
Sign Up FREE

Latest Features

Curious about what we’ve been working on?

View New Features

Get expert survey help

Get expert survey help


Best practices for planning expert surveys

Planning makes writing your survey easy.

Download eGuide

How many responses do you need?

Use our sample size calculator to find out!

Calculate now

Analyze survey data like a pro

Learn to slice and dice data using the Analyze tool.

Download eGuide