Pull Request Review Tools Social Media Survey We are Omar Elazhary, Matthieu Foucault, and Margaret-Anne Storey, computer science researchers from the University of Victoria. Our team is investigating the usage of automated tool support, and how those tools impact pull request evaluation and the pull request submission workflow. We’d be grateful if you could help us understand this further by completing the survey below. It should take approximately 25 minutes of your time.The tools we are investigating include all those that are associated to a pull request or commit, whether they appear as checks, automated comments, or via any other medium. We are also interested in other tools you might be using that do not officially show up on your software repository provider's platform, but also help you throughout the pull request evaluation process. This survey will help us in understanding how software reviewers use the tools available to them, and whether these tools fulfill their needs. We will openly publish the results so everyone can benefit from them, but will anonymize everything before doing so. Your responses will be handled confidentially. There will be an opportunity near the end of the survey to sign up for an interview if you would like to contribute more. Also, we will be sending out a report detailing the effects of integration usage in software repositories based on the results of this survey to all respondents who provide their contact information at the end of the survey. This survey focuses on the usage of automated tools from the perspective of the reviewer of a pull request. If you also used such tools as a contributor, i.e. as a submitter of a pull request, you are more than welcome to take part in our other survey here: Contributor Survey Question Title * 1. What are the benefits and challenges? Benefits Challenges DISCLAIMERThis study has been reviewed and approved by the Human Ethics Research Board at the University of Victoria. If you wish to read more details and review the consent form for this survey, please click here. Question Title * 2. How long have you been involved in developing software? Less than and up to 2 years 3 to 5 years 6 to 8 years More than 8 years Question Title * 3. How many projects (open source and others) are you actively reviewing contributions for? Question Title * 4. How many pull requests do you typically review on a monthly basis? About Integrations/Tools:We consider an integration to be a tool that integrates with the repository management system and provides feedback within the user interface of that system. Examples of feedback that these tools provide can be (1) a badge in your README.md (e.g., code climate), (2) minimal feedback within a pull request (e.g. TravisCI or Codacy), or (3) detailed feedback within a pull request like codecov. Question Title Example of minimal feedback on a pull request Question Title Example of detailed feedback on a pull request Question Title * 5. Which of the following tools have you used over the past 3 months, either locally on your development machine, as part of continuous integration (CI), or as an integration that was required by your projects' contribution guidelines?Check all that apply. Locally on my machine Acts as part of an automated CI build Integrated with your repo management system (no feedback on pull requests) Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) Integrated with your repo management system (with detailed feedback on pull requests) Travis CI Travis CI Locally on my machine Travis CI Acts as part of an automated CI build Travis CI Integrated with your repo management system (no feedback on pull requests) Travis CI Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) Travis CI Integrated with your repo management system (with detailed feedback on pull requests) AppVeyor AppVeyor Locally on my machine AppVeyor Acts as part of an automated CI build AppVeyor Integrated with your repo management system (no feedback on pull requests) AppVeyor Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) AppVeyor Integrated with your repo management system (with detailed feedback on pull requests) coveralls coveralls Locally on my machine coveralls Acts as part of an automated CI build coveralls Integrated with your repo management system (no feedback on pull requests) coveralls Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) coveralls Integrated with your repo management system (with detailed feedback on pull requests) CircleCI CircleCI Locally on my machine CircleCI Acts as part of an automated CI build CircleCI Integrated with your repo management system (no feedback on pull requests) CircleCI Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) CircleCI Integrated with your repo management system (with detailed feedback on pull requests) codecov codecov Locally on my machine codecov Acts as part of an automated CI build codecov Integrated with your repo management system (no feedback on pull requests) codecov Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) codecov Integrated with your repo management system (with detailed feedback on pull requests) scrutinizer scrutinizer Locally on my machine scrutinizer Acts as part of an automated CI build scrutinizer Integrated with your repo management system (no feedback on pull requests) scrutinizer Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) scrutinizer Integrated with your repo management system (with detailed feedback on pull requests) StyleCI StyleCI Locally on my machine StyleCI Acts as part of an automated CI build StyleCI Integrated with your repo management system (no feedback on pull requests) StyleCI Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) StyleCI Integrated with your repo management system (with detailed feedback on pull requests) Hound Hound Locally on my machine Hound Acts as part of an automated CI build Hound Integrated with your repo management system (no feedback on pull requests) Hound Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) Hound Integrated with your repo management system (with detailed feedback on pull requests) Codeship Codeship Locally on my machine Codeship Acts as part of an automated CI build Codeship Integrated with your repo management system (no feedback on pull requests) Codeship Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) Codeship Integrated with your repo management system (with detailed feedback on pull requests) Jenkins Jenkins Locally on my machine Jenkins Acts as part of an automated CI build Jenkins Integrated with your repo management system (no feedback on pull requests) Jenkins Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) Jenkins Integrated with your repo management system (with detailed feedback on pull requests) Wercker Wercker Locally on my machine Wercker Acts as part of an automated CI build Wercker Integrated with your repo management system (no feedback on pull requests) Wercker Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) Wercker Integrated with your repo management system (with detailed feedback on pull requests) Semaphore CI Semaphore CI Locally on my machine Semaphore CI Acts as part of an automated CI build Semaphore CI Integrated with your repo management system (no feedback on pull requests) Semaphore CI Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) Semaphore CI Integrated with your repo management system (with detailed feedback on pull requests) Code Climate Code Climate Locally on my machine Code Climate Acts as part of an automated CI build Code Climate Integrated with your repo management system (no feedback on pull requests) Code Climate Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) Code Climate Integrated with your repo management system (with detailed feedback on pull requests) Codacy Codacy Locally on my machine Codacy Acts as part of an automated CI build Codacy Integrated with your repo management system (no feedback on pull requests) Codacy Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) Codacy Integrated with your repo management system (with detailed feedback on pull requests) TeamCity TeamCity Locally on my machine TeamCity Acts as part of an automated CI build TeamCity Integrated with your repo management system (no feedback on pull requests) TeamCity Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) TeamCity Integrated with your repo management system (with detailed feedback on pull requests) QuantifiedCode QuantifiedCode Locally on my machine QuantifiedCode Acts as part of an automated CI build QuantifiedCode Integrated with your repo management system (no feedback on pull requests) QuantifiedCode Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) QuantifiedCode Integrated with your repo management system (with detailed feedback on pull requests) BitHound BitHound Locally on my machine BitHound Acts as part of an automated CI build BitHound Integrated with your repo management system (no feedback on pull requests) BitHound Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) BitHound Integrated with your repo management system (with detailed feedback on pull requests) Shippable Shippable Locally on my machine Shippable Acts as part of an automated CI build Shippable Integrated with your repo management system (no feedback on pull requests) Shippable Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) Shippable Integrated with your repo management system (with detailed feedback on pull requests) VersionEye VersionEye Locally on my machine VersionEye Acts as part of an automated CI build VersionEye Integrated with your repo management system (no feedback on pull requests) VersionEye Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) VersionEye Integrated with your repo management system (with detailed feedback on pull requests) SonarQube SonarQube Locally on my machine SonarQube Acts as part of an automated CI build SonarQube Integrated with your repo management system (no feedback on pull requests) SonarQube Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) SonarQube Integrated with your repo management system (with detailed feedback on pull requests) ReviewNinja ReviewNinja Locally on my machine ReviewNinja Acts as part of an automated CI build ReviewNinja Integrated with your repo management system (no feedback on pull requests) ReviewNinja Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) ReviewNinja Integrated with your repo management system (with detailed feedback on pull requests) SourceGraph SourceGraph Locally on my machine SourceGraph Acts as part of an automated CI build SourceGraph Integrated with your repo management system (no feedback on pull requests) SourceGraph Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) SourceGraph Integrated with your repo management system (with detailed feedback on pull requests) Zappr Zappr Locally on my machine Zappr Acts as part of an automated CI build Zappr Integrated with your repo management system (no feedback on pull requests) Zappr Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) Zappr Integrated with your repo management system (with detailed feedback on pull requests) Sensio Labs Insight Sensio Labs Insight Locally on my machine Sensio Labs Insight Acts as part of an automated CI build Sensio Labs Insight Integrated with your repo management system (no feedback on pull requests) Sensio Labs Insight Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) Sensio Labs Insight Integrated with your repo management system (with detailed feedback on pull requests) Hakiri Hakiri Locally on my machine Hakiri Acts as part of an automated CI build Hakiri Integrated with your repo management system (no feedback on pull requests) Hakiri Integrated with your repo management system (with minimal feedback on PRs i.e. a checkmark) Hakiri Integrated with your repo management system (with detailed feedback on pull requests) If you use tools not mentioned above, please add them here and let us know how you use them using the previous criteria. For the following questions, please consider the repository where you review the most pull requests. Question Title * 6. Where are the guidelines (e.g. a style guide) for contributing to the project specified? CONTRIBUTING.md / Contribution guide Within various integration checks Implicit knowledge (personal communication, mailing list, ...) No guidelines Other (please specify) Question Title * 7. If the project does have contribution guidelines defined, please provide examples. Question Title * 8. How do you ensure that submitters' contributions follow the project guidelines? I rely on the integrations' check results I rely on CI build results that run integrations tools I rely on CI build results that run non-integration tools I manually check the contribution against the contribution guide I don't Other (please specify) Question Title * 9. Are there tools that are not required by the contribution guidelines, but that you still use when evaluating a contribution? Yes No Question Title * 10. Are there tools that are not required by the contribution guidelines, but that you still use when evaluating a contribution?If yes, which tools do you use, and why do you use them? Question Title * 11. Please rate your level of agreement with the following statements. Strongly Disagree Disagree Agree Strongly Agree "The usage of integrations makes the review process objective." "The usage of integrations makes the review process objective." Strongly Disagree "The usage of integrations makes the review process objective." Disagree "The usage of integrations makes the review process objective." Agree "The usage of integrations makes the review process objective." Strongly Agree "The usage of integrations makes the review process transparent." "The usage of integrations makes the review process transparent." Strongly Disagree "The usage of integrations makes the review process transparent." Disagree "The usage of integrations makes the review process transparent." Agree "The usage of integrations makes the review process transparent." Strongly Agree Question Title * 12. Please elaborate on why you agree/disagree with the statements above. Question Title * 13. Below are information sources commonly accessed to reason about the quality of a contribution. Please rank them from most used to least used. 1 2 3 4 The integration's log dump for the latest pull request build 1 2 3 4 The CI log dump for the latest pull request build 1 2 3 4 The visual/annotated information the integration provides (in the pull request interface) 1 2 3 4 The visual/annotated information the integration provides (in 3rd party integration interface) Question Title * 14. Are there other resources that you use but are not mentioned above? If yes, which do you most frequently use? Question Title * 15. Which steps do you need to perform locally while reviewing a pull request, as opposed to having a tool/integration do them? Running a test suite Running a linter Running a dependency checker Other (please specify) Question Title * 16. If you selected an option in the previous question, why do you perform them locally? Question Title * 17. Which of the tools/integrations that you previously chose/listed do you feel best helps you evaluate incoming pull requests according to the guidelines specified by the project you review for? Question Title * 18. Do you use any integrations/tools whose checks you consider of low priority and do not have much effect on the pull request evaluation process?If yes, please provide examples. Question Title * 19. Do contributors ask you to help them with their failing tool/integration results? If yes, please give example(s). Question Title * 20. Approximately how many such requests have your received in the previous month? Question Title * 21. What functionality would you like to have as an integration? Question Title * 22. ”Tools that enable the automation of tasks within the pull request submission process do not provide as much value to the review process as they do the implementation process.”Do you agree with the above statement? Why or why not? Question Title * 23. Do you have any additional comments? ___________________________________ Question Title * 24. Would you like to be directly notified about the results of this study, and any publications that would result from it? If yes, please fill in your email below. Question Title * 25. Would you be available for a 30 minute interview over Skype or Google Hangouts? We’ll use the email filled in above to contact you. Yes No Thank you for taking the time to fill out this survey!! Done