Cost-saving in Continuous Integration: Development, Improvement, and Evaluation of Build Selection Approaches

Continuous integration (CI) is a widely used practice in modern software engineering. Unfortunately, it is also an expensive practice — Google and Mozilla estimate their CI systems in millions of dollars. In this dissertation, I propose a collection of novel build selection approaches that are able to save the cost of CI. I also propose the first exhaustive comparison of techniques to improve CI including build and test granularity approaches. I firstly design a build selection approach (SMARTBUILDSKIP) for CI cost reduction in a balanceable way. The evaluation of SMARTBUILDSKIP shows that it can save a median of 30% of builds by only incurring a median delay of 1 build in a median of 15% of failing builds under its most conservative configuration. To minimize the delayed failure observation, I then propose the second build selection approach (PRECISEBUILDSKIP) that can save cost without delaying failure observation. We find that PRECISEBUILDSKIP can save a median of 5.5% of builds while capturing the majority of failing builds (100% in median) from the evaluation. After that, I evaluate the strengths and weaknesses of 10 techniques that can improve CI including SMARTBUILDSKIP. The findings of the comparison motivate my next work to design a hybrid technique (HYBRIDBUILDSKIP) that combines these techniques to produce more cost-saving while keeping a low proportion of failing builds that are delayed in observation. Finally, I design an experiment to understand how different weights of test duration among the whole build duration can influence the cost-saving of build and test selection techniques.