When, How, and Why Developers (Do Not) Test in Their IDEs

by Beller, Moritz and Gousios, Georgios and Panichella, Annibale and Zaidman, Andy

You can get a pre-print version from here.
You can view the publisher's page here.

Abstract

The research community in Software Engineering and Software Testing in particular builds many of its contributions on a set of mutually shared expectations. These expectations often form the motivation for research on the creation of new testing tools and the refinement of existing test processes in an attempt to support practitioners. Despite the fact that they are the basis for many publications as well as open-source and commercial testing applications, the common expectations and beliefs are rarely ever questioned. For example, Frederic Brooks’ statement that testing takes half of the development time seems to have manifested itself within the community since he first made it in the “Mythical Man Month” in 1975. With this paper, we report on the surprising results of a large-scale field study with 416 software engineers whose development activity we closely monitored over the course of five months, resulting in over 13 years of recorded work time. Our findings question several commonly shared assumptions and beliefs about testing and might be contributing factors to the observed bug proneness of software in practice: the majority of developers in our study does not test; developers rarely run their tests in the IDE; Test-driven Development is not widely practiced; and, last but not least, software developers only spend a quarter of their work time engineering tests, whereas they think they test half of their time.

Bibtex record

@inproceedings{BGPZ15,
  author = {Beller, Moritz and Gousios, Georgios and Panichella, Annibale and Zaidman, Andy},
  title = {When, How, and Why Developers (Do Not) Test in Their IDEs},
  booktitle = {Proceedings of the 10th Joint Meeting on Foundations of Software Engineering},
  series = {ESEC/FSE 2015},
  year = {2015},
  isbn = {978-1-4503-3675-8},
  location = {Bergamo, Italy},
  pages = {179--190},
  doi = {10.1145/2786805.2786843},
  acmid = {2786843},
  publisher = {ACM},
  address = {New York, NY, USA},
  url = {/pub/when-how-why-developers-do-not-test.pdf}
}