Show simple item record

dc.contributor.authorKazerouni, Ayaan M.en
dc.contributor.authorDavis, James C.en
dc.contributor.authorBasak, Arinjoyen
dc.contributor.authorShaffer, Clifford A.en
dc.contributor.authorServant, Franciscoen
dc.contributor.authorEdwards, Stephen H.en
dc.description.abstractAs incorporating software testing into programming assignments becomes routine, educators have begun to assess not only the correctness of students' software, but also the adequacy of their tests. In practice, educators rely on code coverage measures, though its shortcomings are widely known. Mutation analysis is a stronger measure of test adequacy, but it is too costly to be applied beyond the small programs developed in introductory programming courses. We demonstrate how to adapt mutation analysis to provide rapid automated feedback on software tests for complex projects in large programming courses. We study a dataset of 1389 student software projects ranging from trivial to complex. We begin by showing that although the state-of-the-art in mutation analysis is practical for providing rapid feedback on projects in introductory courses, it is prohibitively expensive for the more complex projects in subsequent courses. To reduce this cost, we use a statistical procedure to select a subset of mutation operators that maintains accuracy while minimizing cost. We show that with only 2 operators, costs can be reduced by a factor of 2-3 with negligible loss in accuracy. Finally, we evaluate our approach on open-source software and report that our findings may generalize beyond our educational context. (c) 2021 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (
dc.description.sponsorshipNational Science Foundation, United States of AmericaNational Science Foundation (NSF) [DUE-1625425, DLR-1740765]en
dc.rightsCreative Commons Attribution-NonCommercial-NoDerivs 4.0 Internationalen
dc.subjectSoftware testingen
dc.subjectMutation analysisen
dc.subjectSoftware engineering educationen
dc.subjectAutomated assessment toolsen
dc.titleFast and accurate incremental feedback for students' software tests using selective mutation analysisen
dc.typeArticle - Refereeden
dc.contributor.departmentComputer Scienceen
dc.description.notesWe are grateful to the National Science Foundation, United States of America for their support under grants DUE-1625425 and DLR-1740765. We thank the anonymous peer reviewers whose feedback improved this paper.en
dc.title.serialJournal of Systems and Softwareen

Files in this item


This item appears in the following Collection(s)

Show simple item record

Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International
License: Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International