Oops, my tests broke the build: An analysis of Travis CI builds with GitHub

by Beller, Moritz and Gousios, Georgios and Zaidman, Andy

You can get a pre-print version from here.
You can view the publisher's page here.

Abstract

Continuous Integration (CI) has become a best practice of modern software development. At present, we have a shortfall of insight into the testing practices that are common in CI-based software development. In particular, we seek quantifiable evidence on how central testing really is in CI, how strongly the project language nfluences testing, whether different integration environments are valuable and if testing on the CI can serve as a surrogate to local testing in the IDE. In an analysis of 2,640,825 Java and Ruby builds on Travis CI, we find that testing is the single most important reason why builds fail. Moreover, the programming language has a strong influence on both the number of executed tests, their test run time and proneness to fail. The use of multiple integration environments leads to 10% more failures being caught at build time. However, testing in the CI does not seem to be a good surrogate for running tests in the IDE. To facilitate further research on Travis CI with GitHub, we introduce TravisTorrent.

Bibtex record

@article{BGZ16,
  title = {Oops, my tests broke the build: An analysis of Travis CI builds with GitHub},
  author = {Beller, Moritz and Gousios, Georgios and Zaidman, Andy},
  year = {2016},
  month = apr,
  keywords = {Travis CI, GitHub, TravisTorrent, GHTorrent, Testing, JUnit, Ruby, Java, IDE, Continuous Integration},
  volume = {4},
  pages = {e1984v1},
  journal = {PeerJ Preprints},
  issn = {2167-9843},
  url = {https://doi.org/10.7287/peerj.preprints.1984v1},
  doi = {10.7287/peerj.preprints.1984v1}
}

The paper