Online Surveys = Ongoing Insights

How We Keep the Bugs Away: An Installment from the Code Monkey Corner

How We Keep the Bugs Away: An Installment from the Code Monkey Corner
Here's Alex!

It’s really something how quickly time flies and just a little bit has flown by since our last Code-Monkey Corner installment. But that’s only because our engineering monkeys have been hard at work keeping our engines here at SurveyMonkey running. We’re happy to have one of our code monkeys, Alex, who not only speaks French but fluent Python, run us through how he squashes pesky bugs from, well, bugging our SurveyMonkey systems.

Ready for some code conversation? Take it away, Alex!

Everyday, our engineers add, change and remove lines of code. With hard work and a few monkey tricks, all that code is turned into what you use and rely on for making better decisions–SurveyMonkey. Code can produce amazing software, but it has its weaknesses too–bugs. Don’t we all hate bugs? We’ve all been victims of computer programs that suddenly stop responding, act silly, or simply close without notice–anyone ever lose unsaved work before? Yeah, it hurts.

The code which makes up a program is often designed to obey mouse clicks, keystrokes and touch gestures. But sometimes under certain situations, errors can surface and affect a program in unexpected ways. This is actually quite common. Errors happen all the time, however we have skilled engineers who can handle them gracefully. But since software engineering is so complex, how can one prevent an error from happening if it was never observed in the first place? How can we minimize the risks for bugs?

The most popular technique is to write automated tests. Carefully crafted, tests are tiny little programs that interact with the software under test to ensure it works as intended. Some tests can even simulate mouse clicks just as if a real live person was using it. So say you have a test that interacts with a drop-down menu…You want to ensure the menu opens on click, that hovered items are highlighted, and that each clicked items do what they’re supposed to do!

At SurveyMonkey, we run tests often. In fact, as soon as code is modified, we run our test suite to check if we didn’t accidentally break your favorite Monkey feature. For this to work effectively, we’ve set up what’s called a Continuous Integration server which does exactly this. Any submitted code that doesn’t pass all tests will be rejected. If all tests pass, the server makes sure that the new code is covered by new tests before merging the author’s changes into the master code base. Each bug fix deserves a new test in order to avoid regressions. Over time, hundreds, thousands of tests–if not more–are accumulated and make the software even stronger.

So let’s dive in, shall we? I’ll run you through the four major steps you’ll need to set up a Continuous Integration server on your own.

1) Install Jenkins

With Ubuntu, Jenkins installs smoothly:


Jenkins will start automatically on port 8080.

2) Set up a Jenkins job


Create a new job and call it something without spaces! Jenkins creates a directory of the same name on the filesystem, but pip (or virtualenv?) will choke on spaces. Then select “Build a free-style software project”.

To clone a git repo, you can use the Jenkins GIT plugin. Once installed, go to “Configure System” and under “Git plugin” set the variables for Global “Config” and “Global Config” (required by the plugin to tag the source on each build).

Jenkins Git Global Config

Return to your project and under “Source Code Management”, configure the repo you want to clone. I pass the git Read-Only URL.

Git Repositories

If you run “Build Now”, it should clone your Github repo. You can make sure the files show up under your project’s “Workspace”.


3) Run tests, build reports

Now we have the Python source code, we want to install it in a virtualenv. Under “Build”, add a build step “Execute shell”

Add build step

… and tweak the following code to fit your needs:


As much as I love virtualenvwrapper, don’t use it since it really doesn’t make sense for automated tasks. Moreover, you want your job’s workspace to be self-contained, whereas virtualenvwrapper wants to keep your virtualenvs in a common place, typically ~/.virtualenvs.

The environment variable $WORKSPACE is made available during the script execution, it’s the path to where Jenkins creates your build, e.g., /var/lib/jenkins/jobs/foo/workspace/.

The package nosexcover (don’t misread it) is a nose plugin that introduces the –with-xcoverage nose option. It generates Cobertura-style XML reports.

4) Interpreting reports

When nose runs, it will generate report files in your workspace that Jenkins has to interpret.

  • nosetests.xml: This file is generated by the –with-xunit option and can be interpreted by checking “Publish JUnit test result report”

Test report XMLs

  • coverage.xml: The coverage file is generated by –with-xcoverage. Jenkins can interpret this report after installing the Cobertura plugin. It will be available as “Publish Cobertura Coverage Report”, and the report pattern must be **/coverage.xml.

Publish Cobertura

  • pylint.out: For this one, you have to install the Violations plugin, which will make the checkbox “Report Violations” available. The plugin supports a bunch of different reports. Look for “pylint” and enter **/pylint.out as the “XML filename pattern”.

XML Filename pattern

Run “Build Now” and tweak the Shell script until nose runs all the way through (regardless if your tests pass). Make sure the report files are generated. Eventually, you’ll get nice charts on your job’s page.




Github, notify Jenkins!

Finally, we want to run the build every time your Github project receives new code (push). Let’s have Jenkins ready to receive notifications from Github. We need to check “Trigger builds remotely (e.g., from scripts)”.

Authentication Token

NOTE: This checkbox only shows up if you check “Enable security” at the server configuration level and set a “Security Realm”.

Once enabled, set the “Authentication Token” to, e.g., GIT_PUSH_NOTIFY.

Now go to your Github project page, click the “Admin” button to configure a “Post-Receive URLs” service hook, and copy/paste the URL given under the “Authentication Token” input field.

Post-Receive URLs

Finally, replace the JENKINS_URL part by the URL to your own Jenkins server. Github has a “Test hook” button to simulate a code push and trigger an actual HTTP call. Jenkins should start building your Python project automatically.

And there you have it! I now leave you to my good friend, Philosoraptor, with a meta-question to unleash your inner geek. Enjoy.


Questions for Alex? Let him know in the Comments section below.

Want to join him in the Python/Monkey fun? We’re hiring, check out our Jobs page and apply today!

  • MarcAbramowitz

    Great post! Thanks, Alex!

Inspired? Create your own survey.

Inspired? Create your own survey.

PRO Sign Up Sign Up FREE or Sign in

Write Surveys Like a Pro

Write Surveys Like a Pro

Ever wonder what SurveyMonkey’s really made of?

Ever wonder what SurveyMonkey's really made of?

Read our engineering blog »