Pull Request Submission Tools Social Media Survey We are Omar Elazhary, Matthieu Foucault, and Margaret-Anne Storey, computer science researchers from the University of Victoria. Our team is investigating the usage of automated tool support, and how those tools may impact pull request evaluation and the pull request submission workflow. We’d be grateful if you could help us understand this further by completing the survey below. It should take approximately 20 minutes of your time.The tools we are investigating include all those that are associated with a pull request or commit, whether they appear as checks, automated comments, or via any other medium. We are also interested in other tools you might be using that do not officially show up on your software repository provider's platform, but also help you throughout the pull request submission process. This survey will help us in understanding how software developers use the tools available to them, and whether these tools fulfill their needs. We will openly publish the results so everyone can benefit from them, but will anonymize everything before doing so. Your responses will be handled confidentially. There will be an opportunity near the end of the survey to sign up for an interview if you would like to contribute more. Also, we will be sending out a report detailing the effects of integration usage in software repositories based on the results of this survey to all respondents who provide their contact information at the end of the survey.This survey focuses on the usage of automated tools from the perspective of the contributor, i.e. the submitter of a pull request. If you also used such tools as a reviewer of a pull request, you are more than welcome to take part in our other survey here: Reviewer Survey DISCLAIMERThis study has been reviewed and approved by the Human Ethics Research Board at the University of Victoria. If you wish to read more details and review the consent form for this survey, please click here. Question Title * 1. How long have you been developing software? Less than and up to 2 years 3 to 5 years 6 to 8 years More than 8 years Question Title * 2. How many software development projects (open source and others) have you submitted pull requests to over the past 3 months? Question Title * 3. How often do you make pull requests (not necessarily to a single project)? At least once a day At least once a week At least once a month Infrequently throughout the year About Integrations/Tools:We consider an integration to be a tool that integrates with the repository management system and provides feedback within the user interface of that system. Examples of feedback that these tools provide can be (1) a badge in your README.md (e.g., code climate), (2) minimal feedback within a pull request (e.g. TravisCI or Codacy), or (3) detailed feedback within a pull request like codecov. Question Title Example of minimal feedback on a pull request Question Title Example of detailed feedback on a pull request Question Title * 4. Which tools have run on your code over the past 3 months, and where did you get feedback?Check all that apply. Locally on my machine Within a CI tool (e.g., in a CI build log or CI tool's web interface) On the repo's homepage (e.g., a badge in README.md) On the PR interface (with minimal feedback, e.g., a checkmark) On the PR interface (with detailed feedback, e.g., an automated comment) Travis CI Travis CI Locally on my machine Travis CI Within a CI tool (e.g., in a CI build log or CI tool's web interface) Travis CI On the repo's homepage (e.g., a badge in README.md) Travis CI On the PR interface (with minimal feedback, e.g., a checkmark) Travis CI On the PR interface (with detailed feedback, e.g., an automated comment) AppVeyor AppVeyor Locally on my machine AppVeyor Within a CI tool (e.g., in a CI build log or CI tool's web interface) AppVeyor On the repo's homepage (e.g., a badge in README.md) AppVeyor On the PR interface (with minimal feedback, e.g., a checkmark) AppVeyor On the PR interface (with detailed feedback, e.g., an automated comment) coveralls coveralls Locally on my machine coveralls Within a CI tool (e.g., in a CI build log or CI tool's web interface) coveralls On the repo's homepage (e.g., a badge in README.md) coveralls On the PR interface (with minimal feedback, e.g., a checkmark) coveralls On the PR interface (with detailed feedback, e.g., an automated comment) CircleCI CircleCI Locally on my machine CircleCI Within a CI tool (e.g., in a CI build log or CI tool's web interface) CircleCI On the repo's homepage (e.g., a badge in README.md) CircleCI On the PR interface (with minimal feedback, e.g., a checkmark) CircleCI On the PR interface (with detailed feedback, e.g., an automated comment) codecov codecov Locally on my machine codecov Within a CI tool (e.g., in a CI build log or CI tool's web interface) codecov On the repo's homepage (e.g., a badge in README.md) codecov On the PR interface (with minimal feedback, e.g., a checkmark) codecov On the PR interface (with detailed feedback, e.g., an automated comment) scrutinizer scrutinizer Locally on my machine scrutinizer Within a CI tool (e.g., in a CI build log or CI tool's web interface) scrutinizer On the repo's homepage (e.g., a badge in README.md) scrutinizer On the PR interface (with minimal feedback, e.g., a checkmark) scrutinizer On the PR interface (with detailed feedback, e.g., an automated comment) Jenkins Jenkins Locally on my machine Jenkins Within a CI tool (e.g., in a CI build log or CI tool's web interface) Jenkins On the repo's homepage (e.g., a badge in README.md) Jenkins On the PR interface (with minimal feedback, e.g., a checkmark) Jenkins On the PR interface (with detailed feedback, e.g., an automated comment) Code Climate Code Climate Locally on my machine Code Climate Within a CI tool (e.g., in a CI build log or CI tool's web interface) Code Climate On the repo's homepage (e.g., a badge in README.md) Code Climate On the PR interface (with minimal feedback, e.g., a checkmark) Code Climate On the PR interface (with detailed feedback, e.g., an automated comment) Codacy Codacy Locally on my machine Codacy Within a CI tool (e.g., in a CI build log or CI tool's web interface) Codacy On the repo's homepage (e.g., a badge in README.md) Codacy On the PR interface (with minimal feedback, e.g., a checkmark) Codacy On the PR interface (with detailed feedback, e.g., an automated comment) QuantifiedCode QuantifiedCode Locally on my machine QuantifiedCode Within a CI tool (e.g., in a CI build log or CI tool's web interface) QuantifiedCode On the repo's homepage (e.g., a badge in README.md) QuantifiedCode On the PR interface (with minimal feedback, e.g., a checkmark) QuantifiedCode On the PR interface (with detailed feedback, e.g., an automated comment) VersionEye VersionEye Locally on my machine VersionEye Within a CI tool (e.g., in a CI build log or CI tool's web interface) VersionEye On the repo's homepage (e.g., a badge in README.md) VersionEye On the PR interface (with minimal feedback, e.g., a checkmark) VersionEye On the PR interface (with detailed feedback, e.g., an automated comment) ReviewNinja ReviewNinja Locally on my machine ReviewNinja Within a CI tool (e.g., in a CI build log or CI tool's web interface) ReviewNinja On the repo's homepage (e.g., a badge in README.md) ReviewNinja On the PR interface (with minimal feedback, e.g., a checkmark) ReviewNinja On the PR interface (with detailed feedback, e.g., an automated comment) Zappr Zappr Locally on my machine Zappr Within a CI tool (e.g., in a CI build log or CI tool's web interface) Zappr On the repo's homepage (e.g., a badge in README.md) Zappr On the PR interface (with minimal feedback, e.g., a checkmark) Zappr On the PR interface (with detailed feedback, e.g., an automated comment) Sensio Labs Insight Sensio Labs Insight Locally on my machine Sensio Labs Insight Within a CI tool (e.g., in a CI build log or CI tool's web interface) Sensio Labs Insight On the repo's homepage (e.g., a badge in README.md) Sensio Labs Insight On the PR interface (with minimal feedback, e.g., a checkmark) Sensio Labs Insight On the PR interface (with detailed feedback, e.g., an automated comment) If you use tools not mentioned above, please add them here and let us know how you get your feedback using the previous criteria. For the following questions, please consider the repository to which you submit the majority of your pull requests. Question Title * 5. Where are the project contribution guidelines defined? CONTRIBUTING.md / Contribution Guide Within various integration checks Implicit knowledge (personal communication, mailing list, …) No guidelines Other (please specify) Question Title * 6. If the project does have contribution guidelines defined, please provide examples. Question Title * 7. How do you ensure that your contribution follows the project guidelines (if any)? I rely on the integrations' check results I rely on CI build results that run integrations tools I rely on CI build results that run non-integration tools I manually check the contribution against the contribution guide I don't Other (please specify) Question Title * 8. Do the tools you selected/listed previously help you follow the contribution guidelines? If yes, how? Question Title * 9. Which activities do you perform locally to evaluate your contribution before submitting it, without relying on CI or integrations? Running a test suite Running a linter Running a dependency checker Other (please specify) Question Title * 10. If you answered the previous question, why do you choose to perform these activities locally? The tool is too complicated to use/install There is no tool support for this feature It is more efficient to do it manually It provides faster feedback Other (please specify) Question Title * 11. Are there tools that are not enforced by the contribution guidelines, but that you still use? If yes, which tools, and why do you use them? Question Title * 12. Please rate your level of agreement with the following statements. Strongly Disagree Disagree Agree Strongly Agree "Using integrations makes the review process objective." "Using integrations makes the review process objective." Strongly Disagree "Using integrations makes the review process objective." Disagree "Using integrations makes the review process objective." Agree "Using integrations makes the review process objective." Strongly Agree "The usage of integrations makes the review process transparent, in that it makes reviewer reasons for accepting/rejecting a pull request clear." "The usage of integrations makes the review process transparent, in that it makes reviewer reasons for accepting/rejecting a pull request clear." Strongly Disagree "The usage of integrations makes the review process transparent, in that it makes reviewer reasons for accepting/rejecting a pull request clear." Disagree "The usage of integrations makes the review process transparent, in that it makes reviewer reasons for accepting/rejecting a pull request clear." Agree "The usage of integrations makes the review process transparent, in that it makes reviewer reasons for accepting/rejecting a pull request clear." Strongly Agree Question Title * 13. Please elaborate on why you agree/disagree with the statements above. Question Title * 14. The following information sources are commonly accessed to learn more about why a contribution failed. Please rank them from most used (1) to least used (7). 1 2 3 4 5 6 7 The integration’s log dump 1 2 3 4 5 6 7 The CI log dump 1 2 3 4 5 6 7 The visual/annotated information the integration provides (in the PR interface) 1 2 3 4 5 6 7 The visual/annotated information the integration provides (in the 3rd party integration interface) 1 2 3 4 5 6 7 Project documentation 1 2 3 4 5 6 7 Project’s core team members 1 2 3 4 5 6 7 The failing integration’s documentation Question Title * 15. Do you use other resources that are not listed above? If yes, which do you most frequently use? Question Title * 16. “Tools that enable the automation of tasks within the pull request submission process do not provide as much value to me -the contributor- as they do to the reviewer.”Do you agree with this statement? Why or why not? Question Title * 17. Do you have any additional comments? ___________________________________ Question Title * 18. Would you like to be notified about the results of this study and any publications we produce? If yes, please enter your email address below. Question Title * 19. Would you be available for a 30 minute interview over Skype or Google Hangouts? We’ll use the email filled in above to contact you. Yes No Thank you for taking the time to fill out this survey!! Done