Disdoc: AI Teaching Assistant for Computer Science Courses

Abstract

As enrollment in Computer Science grows, traditional help-seeking opportunities for students, such as office hours and forums, become less effective due to rising student-to-teaching assistant ratios. To address this issue, research has investigated large language models (LLMs) to provide individualized help to students at scale. However, prior research primarily targets introductory computing courses, does not fully connect LLMs to course material, and does not expose relevant course material to students. As a result, existing approaches do not adapt well to advanced computing courses and limit opportunities for students to develop self-sufficiency. To address this, we present Disdoc, an LLM-based question and answer tool for students in advanced computing courses. Disdoc presents snippets of course material relevant to student questions and generates answers using an LLM. To include course-specific information in answers, we connect the LLM to all course material through retrieval-augmented generation (RAG). To ensure the RAG system retrieves the most relevant information, we organize course material into question categories. We evaluated Disdoc in a research study on a 340-student Computer Systems class at Virginia Tech, where we tracked student reviews, activity, and exit survey responses. Students indicated that Disdoc was helpful, particularly for questions about course assignments. Usage data revealed that students strongly preferred to see LLM-generated answers and rarely clicked on outgoing links, suggesting they were satisfied with the LLM-generated answers and snippets of relevant course material.