Fast and accurate incremental feedback for students' software tests using selective mutation analysis

Simple item page
dc.contributor.authorKazerouni, Ayaan M.en
dc.contributor.authorDavis, James C.en
dc.contributor.authorBasak, Arinjoyen
dc.contributor.authorShaffer, Clifford A.en
dc.contributor.authorServant Cortes, Francisco Javieren
dc.contributor.authorEdwards, Stephen H.en
dc.contributor.departmentComputer Scienceen
dc.date.accessioned2021-04-20T19:39:27Zen
dc.date.available2021-04-20T19:39:27Zen
dc.date.issued2021-05en
dc.description.abstractAs incorporating software testing into programming assignments becomes routine, educators have begun to assess not only the correctness of students' software, but also the adequacy of their tests. In practice, educators rely on code coverage measures, though its shortcomings are widely known. Mutation analysis is a stronger measure of test adequacy, but it is too costly to be applied beyond the small programs developed in introductory programming courses. We demonstrate how to adapt mutation analysis to provide rapid automated feedback on software tests for complex projects in large programming courses. We study a dataset of 1389 student software projects ranging from trivial to complex. We begin by showing that although the state-of-the-art in mutation analysis is practical for providing rapid feedback on projects in introductory courses, it is prohibitively expensive for the more complex projects in subsequent courses. To reduce this cost, we use a statistical procedure to select a subset of mutation operators that maintains accuracy while minimizing cost. We show that with only 2 operators, costs can be reduced by a factor of 2-3 with negligible loss in accuracy. Finally, we evaluate our approach on open-source software and report that our findings may generalize beyond our educational context. (c) 2021 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).en
dc.description.notesWe are grateful to the National Science Foundation, United States of America for their support under grants DUE-1625425 and DLR-1740765. We thank the anonymous peer reviewers whose feedback improved this paper.en
dc.description.sponsorshipNational Science Foundation, United States of AmericaNational Science Foundation (NSF) [DUE-1625425, DLR-1740765]en
dc.format.mimetypeapplication/pdfen
dc.identifier.doihttps://doi.org/10.1016/j.jss.2021.110905en
dc.identifier.eissn1873-1228en
dc.identifier.issn0164-1212en
dc.identifier.other110905en
dc.identifier.urihttp://hdl.handle.net/10919/103062en
dc.identifier.volume175en
dc.language.isoenen
dc.rightsCreative Commons Attribution-NonCommercial-NoDerivs 4.0 Internationalen
dc.rights.urihttp://creativecommons.org/licenses/by-nc-nd/4.0/en
dc.subjectSoftware testingen
dc.subjectMutation analysisen
dc.subjectSoftware engineering educationen
dc.subjectAutomated assessment toolsen
dc.titleFast and accurate incremental feedback for students' software tests using selective mutation analysisen
dc.title.serialJournal of Systems and Softwareen
dc.typeArticle - Refereeden
dc.type.dcmitypeTexten
dc.type.dcmitypeStillImageen

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
1-s2.0-S0164121221000029-main.pdf
Size:
887.21 KB
Format:
Adobe Portable Document Format
Description:
Download

Collections

Scholarly Works, Computer Science