% \documentclass[nopageskip]{VTthesis}
\documentclass[12pt]{article}

% \title{\textbf{Automated Assessment of Students' Short Written Answers Using NLP and Large Language Models (LLMs)}}

% \submitdate{\today} 

\usepackage[margin=1in]{geometry}
\usepackage{setspace}
\usepackage{hyperref}
\usepackage{graphicx}
\usepackage{amsmath} 
\usepackage{enumitem}
\usepackage{float}
\usepackage{changepage}
\usepackage{caption}
\setcounter{topnumber}{10}
\setcounter{bottomnumber}{10}
\setcounter{totalnumber}{20}
\renewcommand{\topfraction}{0.95}
\renewcommand{\bottomfraction}{0.95}
\renewcommand{\textfraction}{0.05}
\setlength{\abovecaptionskip}{2pt} % default is ~10pt
\setlength{\belowcaptionskip}{2pt} % optional: reduces space below caption


\title{\textbf{Automated Assessment of Students' Short Written Answers Using NLP and Large Language Models (LLMs)}}
\author{
\textbf{Authors:}\\
        Arian Assadzadeh\\
        Demiana Attia\\
        Sanjana Ghanta\\
        Tai Phan\\
        Trey Walker\\
        }
\date{\today}



\begin{document}
\begin{figure}
\centering
    \includegraphics[width=0.7\linewidth]{image.png}
\end{figure}




\singlespacing

%----------------------
% Title Page
%---------   -------------
\maketitle
\thispagestyle{empty}
\begin{center}
        \vspace{0.5cm}


Course: CS4624 Multimedia/Hypertext\\
Virginia Tech\\
Instructor and Client: Dr. Mohamed Farag
    \vspace{0.8cm}
\end{center}
\vfill

\vfill
\clearpage

%----------------------
% Table of Contents
%----------------------
\tableofcontents
\clearpage

%----------------------
% Table of Figures
%----------------------
\listoffigures
\clearpage

%----------------------
% 1. Executive Summary / Abstract
%----------------------
\section{Executive Summary / Abstract}

This project proposes the design and implementation of an automated web-based system for automated formative assessment of student short written answer questions using Natural Language Processing (NLP) and Large Language Models (LLM). The proposed solution addresses the ever-growing challenge in modern education, where educators face increasing class sizes, different types of approach, and student abilities, and limited time to give personalized individual feedback for each student. The system utilizes Gemma3:4b LLM, hosted via Ollama, integrated with a React frontend and FastAPI backend to accomplish this task. Using recent advances in the area of machine learning, the objective of this project is to offer a scalable, efficient, and meaningful tool that can assist both students and instructors in the feedback process. The tool will feature user authentication, answer uploads, short answer grading, and grading results. 



\pagebreak
%----------------------
% 2. Introduction
%----------------------
\section{Introduction}
The Automated Short Answers Assessment system is a web-based platform designed to grade students’ short answer questions using advanced Natural Language Processing (NLP) techniques and open-source Large Language Models (LLMs). This project addresses the increasing need in modern education; it reduces the amount of time it takes to manually grade questions. Teachers no longer need to put in the cognitive effort as well when it comes to grading with the assist of the automated short answers assessment system.
\newline
\newline
Using NLP and LLMs, our system identifies and scores key ideas in student answers and evaluates how well they align with model answers or rubrics. This not only supports instructors in timely grading but also gives students immediate feedback to improve learning outcomes.


\textbf{
\subsection{Client Introduction} }
The client for this project is Dr. Mohamed Farag, a faculty member at Virginia Tech. Dr. Farag specializes in computer science education and research, with a particular interest in scalable, data-driven educational tools. Dr. Farag has received is Ph.D. in computer science from Virginia Tech in 2016, and his M.Sc. degree in computer science from the Arab Academy for Science, Technology, and Maritime Transport (AAST) in Alexandria, Egypt in 2010. As a research associate at the Virginia Tech Transportation Institute, Dr. Farag has extensive experience leading in interdisciplinary projects that brings about applied computing and education impact. His role in this project ranges from setting requirements, providing feedback during our bi-monthly meetings, and guiding the team’s progress and work with real-world instructional needs. 

\textit{}

\noindent{Contact Information: 540-231-0278, \href{mailto:mfarag@vt.edu}{mfarag@vt.edu} }

\pagebreak
%----------------------
% 3. Requirements
%----------------------
\section{Requirements}

\subsection{Functional Requirements}
These requirements define the core features that the system must support to satisfy the project's objectives. 

Here are the following features and their corresponding description:
\begin{itemize}
    \item User Login and Authentication
    \begin{itemize}
        \item Uses JWT-based authentication with NextAuth.js on the frontend and FastAPI on the backend.
    \end{itemize}
    \item Student Answer Upload
    \begin{itemize}
        \item Instructors can upload individual or a batch of student short answers through the website or by uploading a csv file.
    \end{itemize}
    \item Model Answer Input
    \begin{itemize}
        \item Instructors can specify idea/model answers per question for a comparison.
    \end{itemize}
    \item Rubric Input
    \begin{itemize}
        \item Users can define multi-level rubrics for a grading-by-criteria approach.
    \end{itemize}
    \item Concept Mapping Input
    \begin{itemize}
        \item Allows the educator, or rather user to list curriculum-relevant key ideas or phrases to check for in student answers.
    \end{itemize}
    \item Three Grading Modes
    \begin{itemize}
        \item Rubric: User provides a rubric for the LLM to grade against
        \item Concepts: User provides several concepts for the LLM to grade against
        \item Sample Answer: User provides a sample answer for the LLM to compare to
    \end{itemize}
\end{itemize}

\subsection{Non-functional Requirements}
These capture the expected performance, usability, maintainability, and other quality factors of the system. 

Here are the categories :
\begin{itemize}
    \item Performance

    \item Scalability

    \item Security
    \item Portability 
    \item Maintainability
\item Usability
\item Accessibility
\end{itemize}

\subsection{Deliverables}
Below is a table of deliverables for this semester that we would like to achieve. 

% Requires: \usepackage{graphicx}
\begin{table}[h]
    \centering

     Change to what we accomplished and didn't
    \begin{tabular}{|l|l|p{8cm}|}
        \hline
        \textbf{Deliverable} & \textbf{Status} & \textbf{Notes} \\ \hline
        Working Web Application & Completed& Working login, collections, assessment logic partially in place)\\ \hline
        Codebase (Frontend + Backend) & Completed& GitHub structure shown, work underway \\ \hline
        Preprocessing and Evaluation Scripts & Completed& Model optimization\\ \hline
        User Manual & Completed& Screens and flow exist (see slides), but not full write-up yet \\ \hline
        Developer Manual & Completed& Consistent with internal work and repo design\\ \hline
        Performance and Comparison Report& Completed& Can be generated within the admin panel\\ \hline
        Final Documentation & Completed& Milestone 5 in timeline: "write and publish documentation\\ \hline
        VTechWorks Archive Package & Completed& Will work according to timeline\\ \hline
    \end{tabular}
    \caption{Project Deliverables Status}
    \label{tab:project_status}
\end{table}
As the semester comes to an end, all the original deliverables have been met and accomplished by the team. There have been additional things added that were also accomplished, such as adding an admin panel.
\pagebreak
%----------------------
% 4. Design
%----------------------
\section{Design}
\subsection{General Approach}
The project is structured as a modular, full-stack web application that integrates front-end, back-end, database, and machine learning components into a unified pipeline. The end goal of the design is to ensure there is proper cohesiveness across components, enabling iterative development and easy future extension. 

\textit{}

\noindent{The architecture is built based on a microservice-inspired structure that uses Docker Compose for orchestration. Each part of the system (front-end, back-end, model engine) can run and be developed independently but communicates via APIs.}

\subsection{Backend}

The backend is implemented using FastAPI and provides the following endpoints:

\begin{table}[h]
\centering
\begin{tabular}{|l|l|}
\hline
\textbf{Endpoint} & \textbf{Description} \\
\hline
POST /login & Authenticates user and returns JWT token \\
POST /collections & Creates a collection for the current user \\
POST /collections/<collection\_id>/upload-questions & Accepts question uploads for a question or batch \\
POST /collections/<collection\_id>/upload-answers & Accepts student answer uploads for a question or batch \\
POST /student-answers/<student\_answer\_id>/grade & Invokes the grading engine for a given answer in a collection \\
GET /results/\{collection\_id\} & Retrieves a formatted table of graded results \\
\hline
\end{tabular}
\caption{API Endpoints and Descriptions}
\end{table}

The backend also handles:
\begin{itemize}
    \item Schema validation with Pydantic
    \item Role-based permissions
    \item Input salinization
    \item Logging for traceability
\end{itemize}

\subsection{Frontend}
Built using React.js, the frontend is organized into:
\begin{table}[h]
    \centering
    \begin{tabular}{|l|l|}
        \hline
        \textbf{Component}& \textbf{Functionality}\\ \hline
        LoginPage& Handles secure login flow via NextAuth.js\\ \hline
        Upload Form& Lets instructors upload student answers and select grading mode\\\hline
    \end{tabular}
    \caption{Responsive design and accessibility are prioritized using semantic HTML}
\end{table}

\pagebreak
%----------------------
% 5. Implementation
%----------------------
\section{Implementation}
\subsection{Overview}
\begin{figure}[h]
    \centering
    \includegraphics[width=0.7\linewidth]{SystemDesign.png}
    \caption{System Architecture}
    \label{fig:enter-label}
\end{figure}
The core components:
\textbf{Frontend} (React.js + NextAuth.js)
    \begin{itemize}
        \item Handles login, answer upload, and display of results
        \item Interacts with back-end via RESTful API calls
        \item Manages authentication using NextAuth.js and JWT tokens
    \end{itemize}
\textbf{Backend} (FastAPI)
    \begin{itemize}
        \item Acts as the core controller and orchestrator of the system
    \end{itemize}
    \begin{itemize}
        \item Manages authentication, routes, and validation
    \end{itemize}
    \begin{itemize}
        \item Calls scoring engine and processes its results
        \item Interacts with the PostgreSQL database to store/retrieve users, answers, and collections
    \end{itemize}
\textbf{LLM Model:} Gemma3:4b
    \begin{itemize}
        \item Lightweight and efficient open-source LLM optimized for reasoning and instruction-following tasks
        \item Hosted locally through Ollama, which handles model downloading 
        \item Receives API calls from backend, evaluates student answers based on:
        \begin{itemize}
            \item Semantic similarity to a model answer
            \item Rubric-based alignment
            \item Presence of key concepts
        \end{itemize}
    \item Responds with structured outputs
    \end{itemize}
\textbf{Evaluation Metrics:} Mean Absolute Error (MAE) is used to evaluate the performance of models in grading known questions and answers.

\subsection{Backend}
This project utilizes a FastAPI backend to create a RESTful application that contains several API endpoints to interact with the backend. At its core, the backend uses a REST API to perform CRUD operations on collections in the database and integrates LLM functionalities through HTTP calls. The FastAPI application tracks authentication using JWT tokens and employs password hashing for security in the database. Lastly, it uses SQLAlchemy to simplify database access.

\textit{}

A large portion of the backend codebase is dedicated to maintaining robust API endpoints so that the frontend and backend can communicate effectively using HTTP calls. To achieve this, a workflow was followed for each entity identified in the design phase of the project:
\begin{enumerate}
    \item Create SQLAlchemy model in \texttt{models/} to represent the business logic object
    \item Create a Pydantic schema for that model to standardize data validation for HTTP calls and responses
    \item Create desired CRUD operations for that model in the \texttt{crud.py} entrypoint to the database
    \item Create routes with error handling in the \texttt{api/} folder using the \texttt{crud.py} operations for that model
\end{enumerate}

\textit{}

The backend is also the entry point into the automated grading functionality by accessing and managing large-language models through Ollama. To reduce dependencies on a specific model, the backend provides functionality to switch between different downloaded models, as well as allowing custom prompts to be used so that the application can function seamlessly if any of these components must be changed.

\subsection{Frontend}

The frontend is a React-based web application designed to manage and assess students' short answers. It provides an interface for teachers/instructors to create exam collections, add questions, manage students, and collect student answers. The application follows a component-based architecture with a clear separation of concerns.

\subsubsection{Core Technologies}
\begin{itemize}
  \item \textbf{React:} The application is built using React (version 19.0.0)
  \item \textbf{React Router:} Used for navigation and routing (version 7.4.0)
  \item \textbf{Axios:} For making HTTP requests to the backend API
\end{itemize}

\subsubsection{Application Structure}
The frontend follows a standard React application structure:

\subsubsection{Entry Points}
\begin{itemize}
  \item \texttt{index.js}: The application entry point
  \item \texttt{App.js}: Main component defining the routing structure
\end{itemize}

\subsubsection{Pages}
High-level components representing entire views:
\begin{itemize}
  \item \texttt{Login.js}: Handles user authentication
  \item \texttt{Home.js}: Dashboard for viewing and managing collections
  \item \texttt{Students.js}: Interface for managing student data
\end{itemize}

\subsubsection{Components}
Reusable UI elements:
\begin{itemize}
  \item \texttt{CollectionDetail.js}: Displays and manages collection information and questions
  \item \texttt{QuestionForm.js}: Form for creating and editing questions
  \item \texttt{StudentForm.js}: Form for adding and editing students
  \item \texttt{StudentDetail.js}: Displays student information and answers
  \item \texttt{StudentAnswerForm.js}: Interface for submitting student answers
\end{itemize}

\subsubsection{Services}
API communication layer:
\begin{itemize}
  \item \texttt{api.js}: Contains all API calls to the backend
\end{itemize}


\subsubsection{Authentication}
\begin{itemize}
  \item The application starts at the login page (\texttt{Login.js})
  \item Users can login with username and password or register a new account
  \item Authentication uses JWT tokens stored in localStorage
\end{itemize}

\subsubsection{Collection Management}
\begin{itemize}
  \item After login, users are directed to the home page (\texttt{Home.js})
  \item Teachers can view their exam collections
  \item They can create new collections with a name and description
  \item Clicking on a collection opens the \texttt{CollectionDetail} component
\end{itemize}

\subsubsection{Question Management}
\begin{itemize}
  \item Within \texttt{CollectionDetail}, users can:
  \begin{itemize}
    \item View collection details
    \item Add, edit, and delete questions
    \item Each question has text content and an example answer
  \end{itemize}
\end{itemize}

\subsubsection{Student Management}
\begin{itemize}
  \item The Students page (\texttt{Students.js}) allows users to:
  \begin{itemize}
    \item View all students
    \item Add new students with name and school ID
    \item Edit and delete existing students
    \item View student details including submitted answers
  \end{itemize}
\end{itemize}

\subsubsection{Answer Submission System}
\begin{itemize}
  \item Users can submit answers for students through:
  \begin{itemize}
    \item The "Submit Answers" tab in \texttt{CollectionDetail}
    \item Select a student, answer questions, and submit
  \end{itemize}
  \item \texttt{StudentAnswerForm} handles the submission of answers
  \item The system supports bulk submission of multiple answers at once
\end{itemize}

\subsubsection{API Communication}
\begin{itemize}
  \item The frontend communicates with the backend through a REST API \\(hosted at \texttt{http://localhost:8001/api})
  \item All API calls are organized in the \texttt{api.js} file, categorized by resource type:
  \begin{itemize}
    \item Collection operations
    \item Question operations
    \item User operations
    \item Student operations
    \item Student answers operations
  \end{itemize}
\end{itemize}

\subsubsection{User Interface Design}
\begin{itemize}
  \item The application uses custom CSS styling for each component
  \item Modal dialogs are used for forms and detail views
  \item The interface includes a top taskbar for navigation between main sections
  \item Responsive design elements for forms and listings
\end{itemize}

\subsubsection{Data Flow}
\begin{enumerate}
  \item User authenticates and receives a JWT token
  \item Token is used for all subsequent API requests
  \item UI components fetch data from the API when needed
  \item User actions (create/edit/delete) trigger API calls and UI updates
  \item Component state manages temporary data and UI rendering
\end{enumerate}

This architecture enables a complete workflow for managing

\pagebreak
%----------------------
% 6. Testing/Evaluation/Assessment
%----------------------
\section{Testing/Evaluation/Assessment}

We conducted extensive model selection to determine the most suitable language model for grading. Eleven different models were evaluated using a set of 20 diverse questions, each associated with a range of acceptable answers scored from 0.0 to 1.0.

The evaluation prompt format was:

\begin{quote}
\ttfamily
Question: \{question\}

Correct Answer: \{model\_answer\}

Student's Answer: \{student\_answer\}

Grade the student's answer based on the correct answer from (0.0 - 1.0)
\end{quote}

We compared the grades produced by each model against reference (human) scores and used accuracy (1.0 - absolute error) as the main metric. Based on this, we selected \textbf{gemma3:4b}, which achieved an average accuracy of \textbf{89\%}.

\textit{}

\noindent{In addition to semantic and lexical similarity, we evaluated:}
\begin{itemize}
    \item \textbf{Grade Accuracy}: Calculated as $1.0 - |\text{model grade} - \text{reference grade}|$, which measures how close the model's output is to the human-labeled score (with 1.0 being a perfect match).
    \item \textbf{Grade Stability}: Standard deviation of grades across repeated evaluations, converted into a consistency score scaled from 0.0 (inconsistent) to 1.0 (perfect consistency).
\end{itemize}

\pagebreak
\section{Comparison with Previous System}

To contextualize our evaluation results, we compared the performance of our selected model, \textbf{gemma3:4b}, against the prior system used by the previous team. Both systems were evaluated on the same set of 20 diverse questions with 100 total grading attempts. The performance metric is calculated with MAE by taking the absolute value of the extracted grade minus the model grade.

\begin{table}[h!]
\centering
\begin{tabular}{lccc}
\toprule
\textbf{Metric} & \textbf{Previous System} & \textbf{Gemma3:4b (Ours)} \\
\midrule
Questions Evaluated & 20 & 20 \\
Grading Attempts & 100 & 100 \\
Overall Performance & 0.669 & \textbf{0.89} \\
Consistency Score & \textbf{1.00} & 0.92 \\
Average Response Time (s) & \textbf{1.84} & 2.96 \\
\bottomrule
\end{tabular}
\caption{Performance comparison between previous system and Gemma3:4b.}
\end{table}

While the previous system achieved faster response times and perfect consistency, our model significantly outperformed it in grading performance improving from 66.9\% to 89\%, a relative gain of over 33\%. This indicates that Gemma3:4b better approximates human graders, despite a modest trade-off in latency and slightly reduced stability. The previous system achieved a perfect consistency score of 1.00 because it produced identical outputs for repeated inputs. This is due to its rule-based grading mechanism, which relied solely on deterministic similarity thresholds and fixed phrase matching using the SentenceTransformer model, without any stochastic elements or sampling variability.

\textit{}


\noindent{Ultimately, the enhanced semantic reasoning capabilities of Gemma3:4b make it the more reliable and effective grader for open-ended short-answer assessment tasks.}


\pagebreak
%----------------------
% 7. Feature Overview 
%----------------------
\section{Feature Overview}

This section outlines the core functionalities and capabilities of the system.

\subsection{User Authentication and Access}
\begin{itemize}
    \item \textbf{User Login:} Registered users can log in using their unique username and password.
    \item \textbf{Account Creation:} New users can easily register for an account directly through the login interface.
    \item \textbf{JWT-Based Security:} Authentication is handled securely using JWT tokens, ensuring user sessions are protected.
    \item \textbf{Administrator Access:} Administrator accounts are supported, which have access to more features than normal end-users
\end{itemize}

\subsection{Navigation and Interface}
\begin{itemize}
    \item \textbf{Intuitive Navigation Bar:} A persistent top navigation bar allows easy access to the main sections: Home (Collections), Students, and Admin Panel if the user is an Admin.
    \item \textbf{User Profile Access:} User-specific options are accessible via a user icon, located in the top-right corner.
    \item \textbf{Card-Based Displays:} Collections and students are presented using clear, informative cards for quick overview and selection.
    \item \textbf{Modal Dialogs:} Forms for creating/editing items (collections, questions, students) and viewing details are presented in modal windows.
\end{itemize}

\subsection{Collection Management}
\begin{itemize}
    \item \textbf{Assessment Organization:} Users can create and manage 'Collections' to group related questions and student answers. A 'Collection' usually represents a specific exam or assignment
    \item \textbf{Collection Creation:} Create new collections by providing a name and an optional description.
    \item \textbf{Centralized Viewing:} The Home page serves as a dashboard displaying all created collections.
\end{itemize}

\subsection{Question Management}
\begin{itemize}
    \item \textbf{Question Authoring:} Within a collection, users can add new short answer questions.
    \item \textbf{Model Answer Input:} Each question requires associated text and a corresponding model or example answer used for grading comparison.
    \item \textbf{CRUD Operations:} Full support for Creating, Reading, Updating, and Deleting questions within their respective collections.
\end{itemize}

\subsection{Student Management}
\begin{itemize}
    \item \textbf{Student Roster:} Maintain a list of students, each identified by name and a unique ID.
    \item \textbf{Student Creation/Editing:} Add new students or modify existing student information (name, ID).
    \item \textbf{Student Deletion:} Remove student records as needed.
    \item \textbf{Centralized Student View:} A dedicated 'Students' section lists all registered students.
    \item \textbf{Individual Student Details:} View detailed information for each student, including their profile and a history of submitted answers across collections.
\end{itemize}

\subsection{Answer Submission and Grading}
\begin{itemize}
    \item \textbf{Manual Answer Entry:} Provides an interface within a collection's detail view to manually enter answers for a selected student across all questions in that collection.
    \item \textbf{CSV Upload (Mentioned Elsewhere):} Provides support for bulk upload of questions, answers, and student answers.
    \item \textbf{Answer Viewing:} View previously submitted answers and graded scores. 
    \item \textbf{Grading Trigger:} Grade a single student answer at a time or grade all student answers at once
\end{itemize}

\subsection{Admin View}
\begin{itemize}
    \item \textbf{LLM Management:} Provides an interface to view and download LLMs supported by Ollama.
    \item \textbf{Prompt Management:} Provides support for the creations of custom prompts for the LLM's.
    \item \textbf{LLM-Prompt Pairs:} Gives the ability to pair specific LLM's with custom prompts
    \item \textbf{Robust Testing Wizard:} Testing Wizard allows testing of multiple LLM's and prompts with a specific question set to assess optimal combinations. The results of the test is displayed as the Mean Absolute Error of the LLM assessment on the specified question set. 
\end{itemize}
\newpage
%===============================================================================
% User Guide for Automated Student Short Answer Assessment System

\section{User Guide for Short Answer Assessment System}

This guide will walk you through the complete functionality of the Automated Student Short Answer Assessment system.

% PART A Login / Sign‑up
\subsection{Getting Started: Login and Account Creation}

\begin{enumerate}
\item Navigate to the application at \textbf{http://localhost:3000/}
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(0).jpeg}
    \caption{Initial login screen - The system welcomes you with a clean login interface}
\end{figure}

\item Click the \textbf{"New user? Sign up"} link at the bottom of the login form
\begin{figure}[H]
    \vspace{-1.5cm}
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(1).jpeg}
    \caption{Sign-up option - Access the registration form for new users}
\end{figure}

\item Enter your desired username in the Username field
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(2).jpeg}
    \caption{Registration form - Enter your credentials in the designated fields}
\end{figure}

\item For an administrator account, enter \textbf{"dev"} as your username and create a secure password
\begin{figure}[H]
    \vspace{-1.5cm}
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(3).jpeg}
    \caption{Successful registration - The system confirms your account creation}
\end{figure}

\item Click the \textbf{"Sign Up"} button to complete your registration

\item After successful registration, enter your username and password again to log in

\item Once logged in, you'll arrive at the home page where you can manage your collections
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(7).jpeg}
    \caption{Home page - Central dashboard for accessing all system features}
\end{figure}
\end{enumerate}

\clearpage
% PART B Create first Collection
\subsection{Creating and Managing Collections}

Collections allow you to organize sets of questions and student answers for grading.

\begin{enumerate}
\setcounter{enumi}{7}
\item Click the \textbf{"+ Create Collection"} button on the home screen
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(8).jpeg}
    \caption{Collection creation - Initiate the process of creating a new assessment collection}
\end{figure}

\item Enter a name for your collection in the \textbf{"Collection Name"} field
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(9).jpeg}
    \caption{Collection naming - Provide a clear, descriptive name for your collection}
\end{figure}

\item For this guide, enter \textbf{"Exam 1 demo"} as the collection name

\item Add an optional description in the \textbf{"Description"} field
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(10).jpeg}
    \caption{Adding description - Enter contextual information about the collection (optional)}
\end{figure}

\item Use the grading pair drop-down to select which model and prompt will assess student answers
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(11).jpeg}
    \caption{Grading pair selection - Choose the AI model and prompt combination for assessment}
\end{figure}

\item Click \textbf{"Create"} to finalize your collection
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(13).jpeg}
    \caption{Collection created - Your new "Exam 1 demo" collection is now ready for use}
\end{figure}
\end{enumerate}

% PART C Upload CSVs & inspect student list
\subsection{Uploading Questions and Student Answers}

\begin{enumerate}
\setcounter{enumi}{13}
\item Click on your newly created collection to access its details
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(14).jpeg}
    \caption{Collection view - Access the detailed view of your "Exam 1 demo" collection}
\end{figure}

\item Click the \textbf{"Upload Question CSV"} button to import your assessment questions
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(15).jpeg}
    \caption{Question upload option - Begin the process of importing questions from a CSV file}
\end{figure}

\item Select your question CSV file from your local storage, ensuring it follows the required format
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(16).jpeg}
    \caption{File selection - Choose your properly formatted question CSV file for upload}
\end{figure}

\item Click \textbf{"Upload"} to import the questions into your collection
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(17).jpeg}
    \caption{Questions loaded - Your assessment questions have been successfully imported}
\end{figure}

\item Now import student answers by clicking the \textbf{"Upload Student Answer CSV"} button
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(18).jpeg}
    \caption{Student answer upload - Begin the process of importing student responses}
\end{figure}

\item Select your student answers CSV file and click \textbf{"Upload"}
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(19).jpeg}
    \caption{Student file selection - Choose your properly formatted student answers CSV file}
\end{figure}
\end{enumerate}

\begin{enumerate}
\setcounter{enumi}{19}
\item Verify that student answers have been successfully imported
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(20).jpeg}
    \caption{Student answers imported - All student responses are now ready for assessment}
\end{figure}

\subsection{Grading Student Answers}

\item To grade an individual answer, locate a student (for this guide, "Alex Johnson") and click the \textbf{"Grade"} button
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(21).jpeg}
    \caption{Grade button - Initiate the grading process for an individual student response}
\end{figure}

\item If the required AI model isn't already downloaded, the system will automatically download it
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(22).jpeg}
    \caption{Model downloading - Progress indicator shows the AI model being prepared}
\end{figure}

\item Wait for the model download to complete
\begin{figure}[H]
    \vspace{-1cm}
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(23).jpeg}
    \caption{Download complete - The required AI model is now ready for assessment}
\end{figure}

\item The system will process and display the student's grade (refresh page if not shown)
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(24).jpeg}
    \caption{Grading result - Automated assessment displayed below the student's answer}
\end{figure}

\item To grade all student answers simultaneously, use the \textbf{"Grade All"} button at the top of the page
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(25).jpeg}
    \caption{Grade All option - Process all student responses with a single click}
\end{figure}

\item Return by clicking the \textbf{"$\leftarrow$ back"} button or \textbf{"Home"} in the navigation bar
\end{enumerate}

% PART D Admin → Download TinyLlama model
\subsection{Admin Panel: Managing Models}

\begin{enumerate}
\setcounter{enumi}{26}
\item From the home page, access administrative functions by clicking the \textbf{"Admin"} button
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(26).jpeg}
    \caption{Home page navigation - Locate the Admin button in the navigation bar}
\end{figure}

\item Click the \textbf{"Admin"} button in the navigation bar to open the administrator panel
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(27).jpeg}
    \caption{Admin button - Access advanced system configuration options}
\end{figure}

\item The Admin Panel provides access to model management and other advanced features
\begin{figure}[H]
    \vspace{-1cm}
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(28).jpeg}
    \caption{Admin Panel - Central interface for system configuration and management}
\end{figure}

\item To add a new AI model, enter its name in the model field (model names are available at \href{https://ollama.com/library}{https://ollama.com/library})
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(29).jpeg}
    \caption{Model download - Enter "tinyllama" to install the TinyLlama AI model}
\end{figure}

\item Click the \textbf{"Pull model"} button to begin downloading the selected model
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(30).jpeg}
    \caption{Download progress - System shows the download and installation status}
\end{figure}

\item Wait for the download and installation process to complete
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(31).jpeg}
    \caption{Installation complete - The TinyLlama model is now ready for use}
\end{figure}

\item View all installed models in the \textbf{"Available Models"} section of the Admin Panel
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(32).jpeg}
    \caption{Available models - List of all AI models currently installed (tinyllama and gemma3:4b)}
\end{figure}
\end{enumerate}

% PART E Create a grading Prompt
\subsection{Admin Panel: Managing Prompts}

\begin{enumerate}
\setcounter{enumi}{33}
\item Access prompt management by clicking the \textbf{"Manage Prompts"} button in the Admin Panel
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(33).jpeg}
    \caption{Manage Prompts option - Access tools for creating and editing assessment prompts}
\end{figure}

\item The Manage Prompts page allows you to create or modify prompts for testing and assessment
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(34).jpeg}
    \caption{Prompts management - Overview of existing prompts and creation options}
\end{figure}

\item Click the \textbf{"Create New Prompt"} button to begin creating a custom assessment prompt
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(35).jpeg}
    \caption{Prompt creation - Interface for designing a new assessment prompt}
\end{figure}

\item Enter a descriptive name for your prompt (for this guide: \textbf{"Demo User Guide"})
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(36).jpeg}
    \caption{Prompt details - Enter the required information for your new prompt}
\end{figure}

\item Select an appropriate category for your prompt (for this guide: \textbf{"Chemistry"})
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(37).jpeg}
    \caption{Category selection - Choose "Chemistry" for chemistry-related assessment questions}
\end{figure}

\item Enter the prompt text that will guide the AI in assessing student answers
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(38).jpeg}
    \caption{Prompt configuration - Define the instructions for the AI assessment process}
\end{figure}

\item Ensure your prompt maintains the required format for referencing questions, correct answers, and student responses
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(39).jpeg}
    \caption{Prompt created - Your new prompt is now available for testing and use}
\end{figure}
\end{enumerate}

% PART F Testing Wizard
\subsection{Admin Panel: Testing Prompts and Models}

\begin{enumerate}
\setcounter{enumi}{40}
\item Navigate to the Test Results page and click \textbf{"Create New Test"} to evaluate your prompt-model combination
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(40).jpeg}
    \caption{Test Results page - Starting point for creating and viewing assessment tests}
\end{figure}

\item The Testing Wizard will guide you through setting up an assessment test
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(41).jpeg}
    \caption{Testing Wizard - Step-by-step interface for configuring assessment tests}
\end{figure}

\item Select the AI model you wish to test (for this guide: \textbf{"tinyllama"})
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(42).jpeg}
    \caption{Model selection - Choose the AI model to evaluate in this test}
\end{figure}

\item Select the prompt you created earlier for testing with the chosen model
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(44).jpeg}
    \caption{Prompt selection - Choose which assessment prompt to evaluate}
\end{figure}

\item Enter a name for your test and prepare to upload test questions
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(45).jpeg}
    \caption{Test configuration - Set up basic parameters for your assessment test}
\end{figure}

\item Name your test (for this guide: \textbf{"UserGuide Demo"}) and add an optional description
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(46).jpeg}
    \caption{Test naming - Provide a clear identifier for your test configuration}
\end{figure}

\item Upload a CSV file containing questions for testing, following the required format
\begin{figure}[H]
    \vspace{-1.5cm}
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(47).jpeg}
    \caption{CSV upload - Select a properly formatted question file for testing}
\end{figure}

\item Ensure your CSV follows the specified format with quotes around string values (see example files for reference)

\item After uploading your CSV, click \textbf{"Next"} and then click the \textbf{"Run"} button to execute the test
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(49).jpeg}
    \caption{Test execution - Initiate the assessment test with the Run button}
\end{figure}

\item Monitor the test progress as the system evaluates the model-prompt combination
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(50).jpeg}
    \caption{Test in progress - System shows status as the test executes}
\end{figure}

\item Review the detailed test results when the assessment is complete
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(51).jpeg}
    \caption{Test results - Comprehensive analysis of the model's performance}
\end{figure}

\item The test results are saved and can be accessed later from the Test Results page
\begin{figure}[H]
    \vspace{-1.5cm}
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(53).jpeg}
    \caption{Saved test results - Your test configuration and outcomes are permanently stored}
\end{figure}

\item Return to the home page by clicking the \textbf{"Home"} button in the navigation bar
\end{enumerate}

% PART G Edit Prompt, create Prompt‑Model Pair & reuse it
\subsection{Creating and Managing Pairs}

\noindent{Pairs are combinations of prompts and models used for grading student answers.}

\begin{enumerate}
\setcounter{enumi}{52}
\item From the home page, access pair management features
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(55).jpeg}
    \caption{Home page navigation - Access the system's main interface}
\end{figure}

\item Click \textbf{"Pairs"} in the navigation bar to open the Pairs management section
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(56).jpeg}
    \caption{Pairs section - Interface for creating model-prompt combinations}
\end{figure}

\item Modify the prompt template as needed (e.g., adjusting the scoring scale from 0-1 to 0-100)
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(58).jpeg}
    \caption{Template editing - Customize the prompt for specific assessment needs}
\end{figure}

\item Select your desired AI model from the dropdown menu (for this guide: \textbf{"tinyllama:latest"})
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(59).jpeg}
    \caption{Model selection - Choose which AI model to pair with your prompt}
\end{figure}

\item Click \textbf{"Next"} and provide a name and optional description for your model-prompt pair
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(61).jpeg}
    \caption{Pair configuration - Navigate through the setup process}
\end{figure}

\item Review the summary and click \textbf{"Save"} to create your prompt-model combination
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(63).jpeg}
    \caption{Navigation options - Return to the home page after configuration}
\end{figure}

\item Verify your newly created pair is now available in the Pairs section
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(65).jpeg}
    \caption{Saved pairs - List of all available prompt-model combinations}
\end{figure}

\item Return to the home page to create a new collection using your custom pair
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(66).jpeg}
    \caption{Home page - Return to the main dashboard to create a new collection}
\end{figure}

\item When creating a new collection, select your custom pair from the \textbf{"Grading Pair"} dropdown menu
\begin{figure}[H]
    \hspace*{-2.5cm}
    \includegraphics[width=1.29\textwidth]{imagesFinal/ascreenshot(67).jpeg}
    \caption{Pair selection - Choose your custom model-prompt combination for assessment}
\end{figure}
\end{enumerate}


%----------------------
% 8. Developer Manual (Initial Version)
%----------------------
\section{Developer Manual}
\noindent You may choose to install and run the system locally, through terminal, or through Docker (with 1 or 4 images).

% Requires \usepackage{hyperref} in the preamble
\subsubsection{Local Installation}

\begin{enumerate}
  \item \textbf{Install prerequisites}
    \begin{enumerate}
      \item Git: \url{https://git-scm.com/downloads/win} (select the 64-bit installer and accept all default options).
      \item Python: \url{https://www.python.org/downloads/}
      \item Ollama: \url{https://ollama.com/download/windows} (run the installer).
      \item Node.js (if \texttt{npm} is unavailable): \url{https://nodejs.org/en/download} (choose the “Windows Installer” and accept defaults).
      \item PostgreSQL 17.4 (x86-64): \url{https://www.postgresql.org/download/windows/}. During setup choose a memorable password.
    \end{enumerate}

  \item \textbf{Clone the repository}\\
  Open a Windows \texttt{cmd} window, navigate to the directory where you want the project, and run:
\begin{adjustwidth}{-1cm}{}
\begin{verbatim}
    git clone https://github.com/f74dragon/Automated-Students-short-answers-assessment
\end{verbatim}
\end{adjustwidth}


  \item \textbf{Configure the project}
    \begin{enumerate}
      \item In the project root open \texttt{.env} and set \texttt{DOCKERIZED=false}.
      \item Create a Python virtual environment and activate it:
\begin{verbatim}
cd Automated-Students-short-answers-assessment
python -m venv venv
\end{verbatim}
      \item Run the Postgres configuration script and enter the password you created during installation twice:
\begin{verbatim}
configPostWin.bat
\end{verbatim}
    \end{enumerate}

  \item \textbf{Build and launch the front-end}
\begin{verbatim}
cd frontend
npm install
npm run build
npm start
\end{verbatim}
  If prompted by Windows Firewall, click \emph{Allow}.
\clearpage
  \item \textbf{Launch the back-end} (in a new command prompt)
\begin{verbatim}
venv\Scripts\activate
cd backend
pip install -r requirements.txt
uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
\end{verbatim}
\end{enumerate}

%==================


%------------------------------------------------------------
\subsubsection{Local Installation (Terminal‑Only Workflow)}
%------------------------------------------------------------
\noindent
The following steps assume you will perform \emph{all} actions from a terminal (PowerShell on Windows, Terminal on macOS, or any shell on Linux). Replace the commands with the variant appropriate for your platform as shown.

\begin{enumerate}
  \item \textbf{Install prerequisites from the command line}

    \begin{description}[style=nextline, leftmargin=1.5cm]
      \item[Windows (PowerShell \texttt{winget})]
\begin{verbatim}

winget install --id Git.Git -e
winget install --id Python.Python.3.12 -e
winget install --id OpenJS.NodeJS.LTS -e
winget install --id PostgreSQL.PostgreSQL -e
winget install --id Ollama.Ollama -e
# Re‑open PowerShell so new PATH entries are active
\end{verbatim}
      \item[macOS (\texttt{brew})]
\begin{verbatim}

brew update
brew install git python@3.12 node postgresql ollama
\end{verbatim}
      \item[Ubuntu / Debian]
\begin{verbatim}

sudo apt-get update

sudo apt-get install -y git python3 python3-venv nodejs npm \
    postgresql postgresql-contrib
    
curl https://ollama.ai/install.sh | sh
\end{verbatim}
    \end{description}

\noindent\textit{Tip:} After installation, ensure each tool is visible on your \texttt{PATH} by running\\
    \texttt{git --version}, \texttt{python --version}, \texttt{node --version}, \texttt{psql --version}, and \texttt{ollama --help}.

\begin{description}
            \item[\textbf{Run Ollama}]: Make sure Ollama is running. Run the following command: 
\end{description}
\vspace{-0.5cm}
\begin{verbatim}
Run Ollama in the background
Linux/MacOs:
        ollama serve &
Windows:
        start "" ollama serve
Note: you can just use "ollama serve" but it will occupy the terminal. 
A new terminal must be opened to continue.
\end{verbatim}

  \item \textbf{Clone the repository}
\begin{adjustwidth}{-1cm}{}
\begin{verbatim}
    git clone https://github.com/f74dragon/Automated-Students-short-answers-assessment
    cd Automated-Students-short-answers-assessment
\end{verbatim}
\end{adjustwidth}


  \item \textbf{Configure environment variables}\\
  Copy the template and edit as needed (e.g.\ to match your local Postgres or Ollama ports):
\begin{verbatim}
cp .env.local.example .env.local   # Linux/macOS
copy .env.local.example .env.local # Windows
# Then open .env.local in a text editor (e.g. code .env.local) and set:
DOCKERIZED=false
\end{verbatim}

  \item \textbf{Initialise PostgreSQL (first run only)}
    \begin{enumerate}
      \item \textit{Windows:} execute the helper script from the project root:
\begin{verbatim}
configPostWin.bat
\end{verbatim}
      \item \textit{macOS / Linux:} create the database and user manually:
\begin{verbatim}
# Switch to the 'postgres' system user and enter the psql shell
sudo -u postgres psql

# Then run the following SQL commands inside psql:
CREATE ROLE "user" LOGIN PASSWORD 'mypassword';
CREATE DATABASE mydatabase OWNER "user";

# Exit the psql shell
\q
\end{verbatim}


 \item \textbf{Create and activate a Python virtual environment}
\begin{verbatim}
python -m venv venv
# Windows
venv\Scripts\activate
# macOS / Linux
source venv/bin/activate
\end{verbatim}

  \item \textbf{Install Python dependencies and launch the back‑end}
\begin{verbatim}
cd backend
pip install -r requirements.txt
uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
\end{verbatim}



  \item \textbf{Build and serve the front‑end (in a second terminal)}
\begin{verbatim}
cd frontend
npm install
npm run build          # generates static assets
npm start              # serves at http://localhost:3000
\end{verbatim}
  If the terminal warns about blocked ports or permissions, resolve them, then re‑run the command.

\end{enumerate}

\noindent Once both terminals show their respective “\texttt{Listening on …}” \\ messages, open \texttt{http://localhost:3000} in a browser to use the application.

%=======================
\subsubsection{All-in-One Docker Image}

\noindent This option allows you to run the entire project stack (frontend, backend, database, and Ollama) from a \textbf{single custom Docker image}, ideal for Linux users seeking a streamlined setup. To install Docker, go to \hyperref[sec:docker-install]{Section~\ref*{sec:docker-install}} below.

\begin{enumerate}
  \item Clone the repository:

\begin{adjustwidth}{-2cm}{}
\begin{verbatim}

git clone https://github.com/f74dragon/Automated-Students-short-answers-assessment
cd ./Automated-Students-short-answers-assessment/LinuxDocker

\end{verbatim}
\end{adjustwidth}


  \item Ensure Docker is installed and running. To install Docker, go to \hyperref[sec:docker-install]{Section~\ref*{sec:docker-install}}.

  
  \item Build and run the all-in-one Docker container:

\begin{verbatim}
For Linux/Mac:
./rebuild.sh

For Windows 
./rebuild.bat
\end{verbatim}
\item After setup completes, access the system in your browser at:
\begin{verbatim}
http://localhost:3000
\end{verbatim}

\noindent This method automatically installs dependencies, configures PostgreSQL, builds the frontend, installs the backend, and launches all services.
\end{enumerate}


Note: Make sure that the setup is complete. In the terminal, the last thing will be \textbf{uvicorn} running. Should see something like this: \begin{verbatim}
    INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
\end{verbatim}
\subsubsection{Docker Installation 4 Images (Recommended)}
\label{sec:docker-install}

This project uses Docker Desktop version 4.40.0 and Docker Engine 28.0.4. Any later versions are not currently tested and may not work with this version of the project. Refer to the \hyperref[sec:docker]{section~\ref*{sec:docker}} for more information on using Docker, running the compose file, and Docker commands. \\

\clearpage
\noindent\textbf{Windows Installation}
\begin{enumerate}
    \item Download Docker Desktop for Windows from the official Docker website: \url{https://www.docker.com/products/docker-desktop}
    \item Double-click the installer file \texttt{Docker Desktop Installer.exe}
    \item Follow the installation wizard instructions, keeping the default settings
    \item When prompted, ensure the options to install required Windows components for WSL 2 are selected
    \item Click \texttt{Finish} to complete the installation
    \item Restart your computer when prompted
    \item \textbf{Enable Virtualization in BIOS (if not already enabled):}
    \begin{enumerate}
        \item Reboot your computer and enter the BIOS/UEFI settings. This is usually done by pressing a key like \texttt{Del}, \texttt{F2}, \texttt{F10}, or \texttt{Esc} during startup (varies by manufacturer).
        \item Navigate to the \textbf{Advanced}, \textbf{CPU Configuration}, or \textbf{Security} tab—location varies by BIOS.
        \item Look for an option named \texttt{Intel Virtualization Technology} or \texttt{SVM Mode} (for AMD processors).
        \item Set this option to \texttt{Enabled}.
        \item Save your changes and exit the BIOS (usually by pressing \texttt{F10}).
        \item \textbf{Tip:} If you cannot find the virtualization setting, consult your motherboard and CPU documentation or search online for instructions specific to your hardware model.
    \end{enumerate}
    
\end{enumerate}


\noindent \textbf{macOS Installation}
\begin{enumerate}
    \item Download Docker Desktop for Mac from the official Docker website: \url{https://www.docker.com/products/docker-desktop}
    \item Open the downloaded \texttt{.dmg} file
    \item Drag the Docker icon to the Applications folder
    \item Open the Applications folder and double-click Docker to start it
    \item When prompted, enter your password to allow Docker to make system changes
    \item Wait for the Docker initialization process to complete



\end{enumerate}
\textbf{Linux Installation (Ubuntu)}
In a terminal, enter the following commands sequentially:
\begin{verbatim}
# Update package information
sudo apt-get update

# Install required dependencies
sudo apt-get install apt-transport-https ca-certificates \
curl software-properties-common

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# Add Docker repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com \
/linux/ubuntu$(lsb_release -cs) stable"

# Update package information again
sudo apt-get update

# Install Docker
sudo apt-get install docker-ce docker-ce-cli containerd.io

# Add your user to the docker group to run Docker without sudo
sudo usermod -aG docker $USER
\end{verbatim}

\noindent\textbf{Note:} For other Linux distributions, refer to the official Docker documentation.

\subsubsection{Verifying Docker Installation}
After installation, verify that Docker is working correctly:

\begin{verbatim}
docker --version
docker run hello-world
\end{verbatim}

\texttt{docker compose up --build}

\subsubsection{Running the Project with Docker}
Clone the repo first:
\begin{adjustwidth}{-1cm}{}
\begin{verbatim}
git clone https://github.com/f74dragon/Automated-Students-short-answers-assessment
\end{verbatim}
\end{adjustwidth}

After configuring Docker and cloning the repository, navigate to the project folder using the terminal:

\begin{verbatim}
cd Automated-Students-short-answers-assessment
\end{verbatim}

\noindent{If you are running the project on a computer equipped with an \textbf{NVIDIA GPU}, no further changes are needed. However, if your system does \textbf{NOT} have an \textbf{NVIDIA GPU}, you must do the following:}

\begin{enumerate}
    \item Rename the default \texttt{docker-compose.yml} file to some other name.
    \item Rename the file \texttt{docker-compose-no-nvidia.yml} to \texttt{docker-compose.yml}.
\end{enumerate}
This ensures the project uses the correct Docker configuration based on your hardware.

\noindent{Once this is set up, run the following command to build and start all the services defined in the Docker configuration:}

\begin{verbatim}
docker compose up --build
\end{verbatim}

\noindent{This command builds the Docker images (if it is not already built) and starts the containers.} 

\noindent{After it completes, the frontend server should be running and accessible at \texttt{http://localhost:3000}.}


\subsection{Technologies Used}
Here is a list of the technologies and frameworks used in the current iteration of this project:
\begin{itemize}
    \item FastAPI
    \item React.js
    \item Docker
    \item SQLAlchemy
    \item PostgreSQL
\end{itemize}

\subsection{Project Structure}
The project follows a modular architecture to enhance decoupling, ensuring that individual components can be seamlessly replaced or upgraded with alternative implementations as needed:
\begin{verbatim}
Automated-Students-short-answers-assessment/
|-- backend/
|-- examples/
|-- frontend/
|-- .dockerignore
|-- .env
|-- .env.local
|-- .gitignore
|-- README.md
|-- configPostWin.bat
|-- docker-compose.yml
|-- docker-compose-no-nvidia.yml
\end{verbatim}

The \texttt{backend/}, \texttt{frontend/}, with \texttt{llm/} subdirectories contain all source code for each respective component's functionalities. The \texttt{examples/}contain questions for testing and the data pertaining to the collections. The \texttt{Dockerfile} that specifies how to containerize these components for deployment. A \texttt{.env} file is used to specify environment variables for local development, but environmental variables should be specified in the \texttt{docker-compose.yml} file for deployment. The \texttt{configPostWin.bat/} helps set up the backend environment in a more feasible manner. Lastly, the compose files specify how to connect the different images built from the \texttt{Dockerfile}s. \texttt{docker-compose.yml} assumes a NVIDIA GPU is installed, however renaming \texttt{docker-compose-no-nivida.yml} to \texttt{docker-compose.yml} configures the project without a NVIDIA GPU.   



%==============================================
\subsection{Backend}

The backend contains a standard FastAPI project structure, with most of the functionality contained in the \texttt{app/} subdirectory. The \texttt{Dockerfile} contains information on how to containerize the backend image. \texttt{tests/} contains the folders with .gitkeep. 

\begin{verbatim}
Automated-Students-short-answers-assessment/backend/
|-- ..
|-- app/
│-- tests/
│-- Dockerfile
\end{verbatim}

\subsubsection{\texttt{app/}}

The \texttt{app/} subdirectory contains all the source code to handle API requests, database functions, and LLM requests.

\begin{verbatim}
Automated-Students-short-answers-assessment/backend/app/
|-- ..
|-- api/
|-- auth/
|-- database/
|-- models/
|-- schemas/
|-- main.py
\end{verbatim}

\subsubsection{\texttt{main.py}}

The \texttt{main.py} file serves as the entry point for the FastAPI backend application. It defines and initializes the main FastAPI app instance, configures Cross-Origin Resource Sharing (CORS) to allow frontend requests (e.g., from \texttt{http://localhost:3000}), and includes all API routers such as users, login, questions, and others.

Upon application startup, the \texttt{startup\_event()} function is triggered. This function performs two key tasks:
\begin{enumerate}
    \item Initializes the database connection by calling \texttt{init\_db()}.
    \item Checks whether any combinations exist in the \texttt{Combination} table. If none are found, it creates a default grading combination with a predefined prompt and model name (\texttt{gemma3:4b}).
\end{enumerate}

This logic ensures the system always has a fallback grading configuration upon first deployment.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.75\linewidth]{defaultPrompt.png}
    \caption{Default grading prompt template inserted into the database if no combinations exist. This prompt is used to evaluate student answers by comparing them to a reference answer using an gemma3:4b.}
    \label{fig:default-prompt}
\end{figure}





\subsubsection{\texttt{api/} }
The \texttt{api/} subdirectory contains the API router endpoints for different backend features. Each route is defined in a modular fashion to keep the application maintainable and scalable. 
\\ 
Here is the \texttt{collections.py} API route that handles collection management: 
\\ 
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.8\textwidth]{dev-manual-figures/api-collection-import.png} 
    \caption{\texttt{collections.py} API Imports}
    \label{fig:enter-label}
\end{figure} 

The imports at the top are necessary imports from FastAPI and SQLAlchemy. The rest of the imports come from other packages within the \texttt{app/} directory.
\\ 

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.8\textwidth]{dev-manual-figures/api-collection-router.png} 
    \caption{\texttt{collections.py} base router}
    \label{fig:enter-label}
\end{figure} 

    This line defines the base router for the collection endpoints as well as the prefix used to distinguish it. The base router will be imported into \texttt{main.py} to define the routes for the collection endpoints.
\\ 
 
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.8\textwidth]{dev-manual-figures/api-collection-http.png}
    \caption{\texttt{collections.py} REST API Endpoints}
    \label{fig:enter-label}
\end{figure} 


These are two sample \texttt{GET} and \texttt{POST} HTTP endpoints for use in the frontend. The \texttt{response\_model} is a Pydantic model for data validation, \texttt{crud} is a semantic way to access the database for \texttt{CRUD} operations, and \texttt{get\_db} is a function that utilizes dependency injection to obtain the current database session to mitigate database access errors. 
% example goes here

\subsubsection{\texttt{auth/}}
This directory contains scripts for authentication in the backend. It only contains one file \texttt{auth.py}, which has various methods for hashing passwords, comparing hashed passwords, creating JWT tokens, and verifying JWT tokens.


\subsubsection{\texttt{database/}}
There are two files in the \texttt{database/} subdirectory, \texttt{connection.py} and \texttt{crud.py}. \\
\texttt{connection.py} is responsible for the initial database connection and database session management, which is used every time the database is accessed. It uses a dependency injection function  \texttt{get\_db} to manage database access.\\

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.3\textwidth]{dev-manual-figures/database-getdbexample.png}
    \caption{\texttt{get\_db/} Dependency Injection Function}
    \label{fig:enter-label}
\end{figure} 

The \texttt{crud.py} module implements the CRUD operations for database access, providing an abstraction layer that simplifies interaction with the database. This allows other components of the backend to handle entities within the context of business logic more efficiently. \\

Each entity is given CRUD access through SQLAlchemy queries, which are structured similarly to SQL queries:\end{enumerate} \\

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.8\textwidth]{dev-manual-figures/database-crud-example.png}
    \caption{Example Database Query for Collections}
    \label{fig:enter-label}
\end{figure} 
In the future, it may be helpful to split up the crud file based on entities as the application grows in size to improve scalability, since the crud.py file is very large.

\subsubsection{\texttt{models/}}
This directory contains the database models used in the application. This is not to be confused with the Large Language Models used in this application. These models were made using the SQLAlchemy module to allow for easier back-end and database integration. Here is an example of the \texttt{Collections} model:

\begin{figure}[htbp]
    \centering
    \includegraphics[width=1\linewidth]{dev-manual-figures/collections-model.png} 
    \caption{Example Collections Model}
    \label{fig:enter-label}
\end{figure} 

SQLAlchemy allows table name declarations, column definitions, and foreign key constraints to reference other tables. \texttt{Collections} requires relationships with other entities such as \texttt{User}, \texttt{Student}, and \texttt{Question}, so foreign key constraints are declared at the bottom of the class.

\subsection{Introduction to Docker}
\label{sec:docker}
Docker is a platform that uses containerization technology to package applications and their dependencies into a standardized unit for software development and deployment. This section provides instructions for installing and running Docker on different operating systems.

\subsubsection{System Requirements}
\begin{itemize}
    \item \textbf{Windows:} Windows 10 64-bit: Pro, Enterprise, or Education (Build 17134 or later) with Hyper-V enabled, or Windows 10 64-bit Home (Build 19018 or later) with WSL 2 enabled
    \item \textbf{macOS:} macOS version 10.15 or newer (Catalina, Big Sur, Monterey, or newer)
    \item \textbf{Linux:} Kernel version 3.10 or higher, 64-bit installation
    \item \textbf{RAM:} At least 4GB of system RAM, 8GB recommended
    \item \textbf{CPU:} 64-bit processor with hardware virtualization support
     \item \textbf{GPU:} 8GB VRAM recommended
\end{itemize}



The first command shows the installed Docker version, and the second downloads and runs a test container to verify the installation.

\subsection{Running the Application with Docker Compose}

\subsubsection{Prerequisites}
Docker Compose is included with Docker Desktop for Windows and Mac. For Linux, you might need to install it separately:

\begin{verbatim}
sudo curl -L "https://github.com/docker/compose/releases/download/ \
v2.18.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
\end{verbatim}

\subsubsection{Starting the Application}
\begin{enumerate}
    \item Clone the application \href{https://github.com/f74dragon/Automated-Students-short-answers-assessment}{repository} or extract the application files to a local directory
    \item Open a terminal or command prompt
    \item Navigate to the application directory containing the \texttt{docker-compose.yml} file
    \item Run the following command to start all services:
\end{enumerate}

\begin{verbatim}
docker-compose up -d
\end{verbatim}

This command builds, creates, and starts the containers defined in the \texttt{docker-compose.yml} file. The \texttt{-d} flag runs containers in the background.

\subsubsection{Accessing the Application}
Once the containers are running, access the application through a web browser:
\begin{itemize}
    \item Frontend: \url{http://localhost:3000}
    \item Backend API: \url{http://localhost:8001}
\end{itemize}

\subsubsection{Stopping the Application}
To stop all running containers:

\begin{verbatim}
docker-compose down
\end{verbatim}

To stop and remove all containers, networks, and volumes:

\begin{verbatim}
docker-compose down -v
\end{verbatim}

\subsubsection{Common Docker Commands}

\begin{tabular}{|p{0.3\textwidth}|p{0.6\textwidth}|}
\hline
\textbf{Command} & \textbf{Description} \\
\hline
\texttt{docker ps} & List running containers \\
\hline
\texttt{docker ps -a} & List all containers (running and stopped) \\
\hline
\texttt{docker images} & List all images \\
\hline
\texttt{docker logs [container-id]} & View logs for a specific container \\
\hline
\texttt{docker exec -it [container-id] bash} & Open a shell inside a running container \\
\hline
\texttt{docker-compose logs} & View logs from all containers defined in docker-compose.yml \\
\hline
\texttt{docker-compose restart} & Restart all services defined in docker-compose.yml \\
\hline
\end{tabular}


\subsubsection{Common Issues}
\begin{itemize}
    \item \textbf{Port conflicts:} If ports 3000 or 8001 are already in use, modify the port mapping in the \texttt{docker-compose.yml} file
    \item \textbf{Permission issues on Linux:} Ensure your user is added to the docker group or run commands with \texttt{sudo}
    \item \textbf{WSL 2 not enabled on Windows:} Follow the prompts to enable WSL 2 during installation
    \item \textbf{Insufficient memory:} Increase the memory allocated to Docker in Docker Desktop settings
\end{itemize}

\subsubsection{Resetting Docker}
If you encounter persistent issues, you can reset Docker to its default state:

\begin{enumerate}
    \item Open Docker Desktop
    \item Go to Settings/Preferences
    \item Click on "Troubleshoot" or "Reset"
    \item Select "Reset to factory defaults"
\end{enumerate}

\subsubsection{Updating Docker}
To update Docker:
\begin{itemize}
    \item \textbf{Windows and macOS:} Docker Desktop will notify you when updates are available
    \item \textbf{Linux:} Use your package manager to update (e.g., \texttt{sudo apt-get update \&\& sudo apt-get upgrade})
\end{itemize}

\subsubsection{Additional Resources}
\begin{itemize}
    \item Official Docker Documentation: \url{https://docs.docker.com}
    \item Docker Hub: \url{https://hub.docker.com}
    \item Docker Community Forums: \url{https://forums.docker.com}
\end{itemize}
\pagebreak

%----------------------
% 11. Timeline
%----------------------
\section{Timeline}

\begin{center}
\begin{tabular}{|l|l|l|}
\hline
\textbf{Milestone} & \textbf{Target Date} & \textbf{Status} \\ \hline
Research \& Prototyping & Feb 24 &  Done \\ \hline
Basic UI \& Database & Mar 17 & Done \\ \hline
Backend \& AI Finalization & Mar 31 & Done \\ \hline
Deployment \& E2E Testing & Apr 14 & Done \\ \hline
Final Documentation & Apr 28 & Done \\ \hline
\end{tabular}
\end{center}

\begin{itemize}
    \item Feb 24 - Research \& Prototyping
    \begin{itemize}
        \item Completed. Technologies selected, initial API routes planned, and LLM models compared.
    \end{itemize}
\item Mar 17 - Basic UI \& Database Setup
\begin{itemize}
    \item Completed. Login/authentication implemented, collections page functional, PostgreSQL integrated.
\end{itemize}
\item March 31 - Backend \& AI integration
\begin{itemize}
    \item Completed. There's a default LLM chosen, however admins have the option to save and use different models from the Ollama library.
\end{itemize}
\item Apr 14 - Deployment \& Testing
\begin{itemize}
    \item Completed. Full system testing with Docker Compose, final integration of frontend/backend/LLM
\end{itemize}
\item Apr 28 - Final Documentation \& VTechWorks Submission
\begin{itemize}
    \item Completed. Final report, source code, manuals, and presentation materials archived and submitted.
\end{itemize}
\end{itemize}
\pagebreak

%----------------------
% 12. Probblem and Solutions
%----------------------
\section{Problems and Solutions}
\subsection{Technical Lessons}


\noindent\textbf{Integration between LLMs and scoring models is non-trivial} 

Early in the project, we anticipated that there wouldn't be an extremely big emphasis on what LLM to use. However, we quickly understood that wasn't the case. Model performance varied drastically in terms of output format, latency, and reliability. This led to a more comparative evaluation of multiple models like (Mistral 7b vs Gemma 3:4b for its consistency and performance in local environments.
\\

\noindent\textbf{Creating flexible grading strategies required a deeper understanding of educational rubrics and LLM prompts} 

Translating a grading rubric or concept list into logic for an LLM is not straightforward. We learn that prompt design is something that is constantly evolving and is an iterative process. As a result, we decided on allowing admins to modify and customize prompts, since different prompts will significantly change the assessment given by the LLM. Rubrics do not translate cleanly into a structured prompting logic, signifying the need for iterative prompt design.
\\

\noindent\textbf{Early decisions on API design helped streamline parallel development}

Our strategy involved concurrent development of the frontend and backend components with a defined API contract. However, we encountered setbacks during the integration phase whenever discrepancies arose in the API contract-specifically concerning expected data structures in calls or responses. This experience served as a critical lesson in documenting meaningful response formats and status codes when working with API's. Once this contract was refined and adhered to, the parallel development workflow proceeded effectively, allowing for the smooth completion of remaining project tasks with minimal integration issues.

\clearpage
\subsection{Project Process Lessons}

\noindent\textbf{Cross-Functional Communication is Critical}

Since roles were divided (LLM integration, frontend, backend, testing), there was a need for effective communication. As a team, regular sync meetings were scheduled to ensure that all members were updated on expectations, progress, and roadblocks preventing the team from making progress.
\\


\noindent\textbf{Documentation Should be Continuous}

While writing the interim report, it was immediately evident that it would be easier to document the project and progress while in development rather than after. This showed the need of constant and consistent documentation during development.
\\


\noindent\textbf{Agile Iteration}

Our team truly benefited from using a lightweight agile methodology when it came to weekly check-ins and we followed our milestones pretty well. This helped us adapt quickly when we were approached by technical blocks or other requests made by the client.
\\

\pagebreak

%----------------------
% 11. Plans
%----------------------
\section{Plans}

\subsection{Potential Future Work}

\noindent\textbf{Advanced Analytics Dashboard}

One thing future teams could do is build an advanced analytics dashboard. This would offer admins with visual insights into how their class is performing over time. It would include trends of concept mastery, invidual student progress, and overall areas of difficulty. With these capabilities, instructors could examine performance in more specific instances by filtering by a certain question or rubric criteria.
\\

\noindent\textbf{LMS Integration}

Another avenue for change is the integration of Learning Management Systems like Canvas and Blackboard. Building plugins or APIs that would allow for automatic syncing of student data, submissions, grades, and feedback would create a smoother experience for instructors and students. It would be rather helpful if combined with Single Sign-On support.
\\

\noindent\textbf{Multilingual Support}

This improvement is essential and could prove very useful. The system could be expanded to support student answers and feedback in multiple languages, which is definitely a top priority in diverse and international classrooms. To ensure that this could be met with full usability, the interface should comply with accessibility standards and offer support for screen readers, keyboard navigation, and other technologies that assist users.
\\

\noindent\textbf{Optimizing Prompts and Grading Strategies }

More future work could also focus on optimizing prompts and grading strategies. This would involve testing different prompts and LLMs to identify which combination provides the more accurate and fair grading. Developers could essentially explore a more rubric-based scoring, comparisons against ideally/partially correct answers, and concept based grading to find the best-performing approach.
\\

\noindent\textbf{Customization and Instructor Controls}

This would allow for the system to be more flexible and adaptable, allowing instructors to tailor grading behavior, feedback tone, rubric emphases based on their specific teaching style. This would all be made easy to a configurable interface.



\pagebreak
%----------------------
% 12. Acknowledgments
%----------------------
\section{Acknowledgments}
We thank our client, Dr. Mohamed Farag, for his guidance. We also acknowledge Virginia Tech and CS4624 - Multimedia/Hypertext for providing us with the opportunity and support to explore impactful applications of NLP in education.
\pagebreak
%----------------------
% 13. References
%----------------------
\section{References}
\begin{enumerate}
    \item Singh, Purushartha, et al. \emph{Automated Support to Scaffold Students’ Written Explanations in Science.} SpringerLink.
    \item Client: Mohamed Farag, \href{mailto:mmagdy@vt.edu}{mmagdy@vt.edu}
    \item Mistral 7b Documentation
    \item BERTScore: \href{https://github.com/Tiiiger/bert_score}{https://github.com/Tiiiger/bert\_score}
    \item FastAPI: \href{https://fastapi.tiangolo.com/}{https://fastapi.tiangolo.com/}
    \item React.js: \href{https://reactjs.org/}{https://reactjs.org/}
\end{enumerate}
\pagebreak

\end{document}
```