Towards Improving Students' Software Testing Practices using Modified Mutation Testing

Abstract

Mutation testing (MT) is a powerful technique for evaluating the quality of software test suites by introducing small faults, called ``mutations,'' into code to assess if tests can detect them. While MT has been extensively applied in the software industry, its use in programming courses faces both computational and pedagogical barriers. My research investigates the successful integration of MT in a post-CS2 Data Structures and Algorithms (DSA) course with 3-4 week long programming projects. Through a comprehensive study across multiple semesters, I investigated three key aspects: the computational demands of MT in an educational auto-grading system, the effect of MT on student test suite quality and coding practices, and the development of a framework for effectively integrating MT in programming courses. Initially, the implementation of standard MT showed mixed results due to inadequate stock feedback. This prompted me to develop a tailored approach that modified MT feedback, while also incorporating additional documentation and training materials. I also observed a noticeable increase (30-50 seconds per submission) in the auto-grader's processing time and feedback turnaround time when using MT, raising concerns about potential server overload. At the same time, the collection of changes made to the environment and requirements as part of this intervention led to an overall reduction in the number of submissions per student needed to complete the projects. My findings suggest that students using modified MT, as a group, demonstrated higher quality test suites and wrote better solution code compared to students whose test suites were graded on code coverage. This version of MT with modified feedback also showed positive results in student understanding and application of MT principles compared to MT with stock feedback. Analysis of IDE activity data, code submissions, and 38 semi-structured student interviews led me to provide a framework for introducing MT as an effective intervention. Thus, my research provides a framework for effectively integrating MT in programming courses, contributing to improved student test suite development and offering practical guidelines for instructors introducing MT in undergraduate Computer Science courses.