Allocation failure testing -- results report (babb)

Thank you for trying babb, an allocation failure tester. We would appreciate it if you would please take 5-10 minutes to answer a few questions about your results.

This data about how well out-of-memory (OOM) is handled in practice today will help the C++ standards committee to evaluate active proposals for C++ standardization.

If you test multiple programs, we would appreciate it if you could fill out this short survey once for each program.

Question Title

* 1. What is a short name or label to describe the program you tested? If the program's name is confidential, please use a short descriptive label instead (e.g., "Unreal game").

Question Title

* 2. Before doing this test, what was your expectation about how well the program handled allocation failures?

My program was not expected to handle allocation failures at all, it exits immediately and at most it installed a new_handler and/or terminate_handler

Some parts of my program were carefully written to be safe for allocation failures

My program was written to be exception-safe and to use exceptions to report allocation failures, and so was expected to able to robustly handle allocation failure like any other exception

Question Title

* 3. Which of the following style of allocation failure reporting are used in the program? (select all that apply)

Throwing std::bad_alloc

Throwing another kind of exception (e.g., an in-house custom exception)

Returning null (e.g., by using new(nothrow))

Question Title

* 4. Approximately how many test executions did you try?

1-5

6-10

11-20

More than 20

Question Title

* 5. Did your program crash, meaning to terminate in any way that was not intended?

Never crashed

Crashed rarely, <5% test executions

Crashed sometimes

Crashed consistently, >90% of test executions

If applicable, please describe what fail_once_per and max_run_length settings would cause crashes most frequently and least frequently

Question Title

* 6. If failures to handle allocation failures well were observed, which code did the problems arise in? (check all that apply, as known)

My application's code

Third-party library code

Operating system calls

Mysterious/unknown

If you checked third-party code or OS calls, please describe which library/OS, and examples of function names that failed (if known and if you can share them)

Question Title

* 7. If any of the results caused you to investigate your code, did you find code that was allocation failure-unsafe that you did not know was unsafe? (select all that apply)

I found general code paths that were not robust when a allocation failure occurred

I found code paths that specifically existed in case of allocation failure (to handle it, or just to unwind safely), but did not actually work correctly if allocation failure occurred

Please briefly describe your findings.

Question Title

* 8. How would you summarize to a colleague "what happened when I tried injecting allocation failures in this program"? Please mention any patterns or any other observations we should know about that were not already covered above.

Question Title

* 9. Which of the following best describes the type of program?

Business (e.g., B2B, B2E)

Communications (e.g., networking, email)

Consumer (e.g., retail websites, mobile apps)

Developer tools (e.g., compilers, code editors)

Engineering (e.g., avionics, power management)

Entertainment (e.g., sports apps, video streaming)

Financial (e.g., trading, mortgage, asset management)

Frameworks (e.g., React, Unreal)

Gaming (e.g., console and mobile games)

Hardware/IoT (e.g., embedded systems, home automation)

Productivity (e.g., budget tracking, note taking)

Social and business networking (e.g., Facebook, Twitter)

Utility apps (e.g., calculators, simple image editors)

Other (please specify)

Question Title

* 10. What platforms did you run the tests on? (select all that apply)

Android

iOS

Embedded system

Game console

Mac

Linux

Windows

Other (please specify)

Question Title

* 11. If you selected Linux, what memory allocation settings were you using?

	Enabled	Disabled	Tested with both "enabled" and "disabled" on different executions	Used the system default and/or don't know
System-wide: Was overcommit_memory enabled (0 or 1)?	System-wide: Was overcommit_memory enabled (0 or 1)? Enabled	System-wide: Was overcommit_memory enabled (0 or 1)? Disabled	System-wide: Was overcommit_memory enabled (0 or 1)? Tested with both "enabled" and "disabled" on different executions	System-wide: Was overcommit_memory enabled (0 or 1)? Used the system default and/or don't know
For this user/program: Was a memory limit enabled (e.g., via ulimit)?	For this user/program: Was a memory limit enabled (e.g., via ulimit)? Enabled	For this user/program: Was a memory limit enabled (e.g., via ulimit)? Disabled	For this user/program: Was a memory limit enabled (e.g., via ulimit)? Tested with both "enabled" and "disabled" on different executions	For this user/program: Was a memory limit enabled (e.g., via ulimit)? Used the system default and/or don't know

Question Title

* 12. After doing this test, what is your impression about how well the program handled allocation failures?

My program did not handle allocation failures well

Some parts of my program were safe for allocation failures, and these parts typically required careful coding

Some parts of my program were safe for allocation failures, and did not require especially careful coding

My program robustly handled allocation failures

Question Title

* 13. (optional) If you would allow a C++ standards committee member to follow up with further questions about your experience, please provide your preferred email address here: