Human-in-the-Loop LLM Grading for Handwritten Mathematics Assessments

The gist The workflow spans (1) constructing solution keys, (2) developing detailed rubric-style grading keys used to guide the LLM, and (3) a grading procedure that combines automated scanning and anonymisation, multi-pass LLM scoring, automated consistency checks, and mandatory human verification<ref:2603.13083#pg2>

Introduction

Providing timely and individualised feedback on handwritten student work is highly beneficial for learning but difficult to achieve at scale<ref:2603.13083#pg2>. This challenge has become more pressing as generative AI undermines the reliability of take-home assessments, shifting emphasis toward supervised, inclass evaluation<ref:2603.13083#pg2>. Short open-ended handwritten responses remain essential for evaluating generative reasoning and structured problem solving, despite substantial operational costs in grading time and feedback timeliness<ref:2603.13083#pg3>. The paper presents a complete, scalable workflow that integrates LLMs into the grading and assessment cycle for short, supervised, pen-and-paper assessments<ref:2603.13083#pg3>.

Workflow Construction

The proposed end-to-end workflow involves several critical stages to ensure accuracy and efficiency<ref:2603.13083#pg3>. These stages include:

1. Constructing solution keys

2. Developing detailed rubric-style grading keys used to guide the LLM

3. A grading procedure that combines automated scanning and anonymisation, multi-pass LLM scoring, automated consistency checks, and mandatory human verification.

Test Setup and Data Preparation

The study was deployed in two undergraduate mathematics courses at the Vrije Universiteit Brussel using six low-stakes in-class tests. These tests were designed as low-stakes formative checkpoints to encourage continuous engagement early in the semester. To enable reliable automated processing of handwritten work at scale, a standardized answer sheet was used, which contained two clearly delineated answer boxes and a separate region for student identification and test metadata. Student identification was captured using a bubble-coded identifier printed on the answer sheets. Post-processing involved bulk-scanning the answer sheets into a single PDF, followed by an OCR and template-recognition step to identify fixed structural elements like ID bubbles and answer boxes. All regions containing identifying information were removed prior to LLM submission, yielding fully anonymised images.

LLM Grading Procedure

The grading procedure utilized GPT-5.1 operating on anonymised answer fragments extracted from the scanned test sheets. Each student’s response was graded independently five times to account for the inherent stochasticity of LLM outputs. The grading prompt used a fixed template that instructed the model to base scoring strictly on evidence in the student’s solution and not to infer non-trivial reasoning from a correct final answer except for trivial algebraic simplifications. A conservative approach was taken by selecting the maximum of five LLM-generated scores as the provisional score. Automated consistency checks were run across the five outputs, and responses exhibiting substantial disagreement were automatically flagged for review.

Results and Analysis

Empirically, LLM assistance reduces grading time by approximately 23% while achieving agreement comparable to, and in several cases tighter than, fully manual grading. The geometric-mean D/M ratio was 0.767 (95% CI: [0.595, 0.989]), corresponding to an average time reduction of 23.3% for LLM-assisted grading relative to manual grading. In terms of grade calibration, agreement between annotators and the LLM in the digital condition is comparable to, and in several cases higher than, the agreement observed between the two human annotators in the manual condition. Deviations are more tightly concentrated near zero in the digital setting than in manual grading, consistent with an anchoring effect where the LLM grade acts as a stabilising reference. This hybrid design shows that LLMs can function as supportive tools that reduce workload, stabilise grading decisions, and enable timely feedback loops at scale. The workflow demonstrates that carefully embedded human-in-the-loop LLM grading can substantially reduce workload while maintaining fairness and accuracy.

Conclusion

The results show that LLMs can support timely and consistent grading at scale, provided that human verification remains in place for rare outlier cases. The study suggests a shift in how LLMs should be positioned within educational assessment, showing they can function as supportive tools that reduce workload, stabilise grading decisions, and enable timely feedback loops at scale. The workflow itself is not mathematics-specific and could extend to other STEM contexts where short, structured tasks require students to demonstrate understanding through incremental steps rather than complete solutions.

Code Availability

The end-to-end pipeline code (processing, anonymisation, grading, and feedback generation) and analysis scripts used in this study are available at https://github.com/Integrated-Intelligence-Lab/llm assisted grading.git. The full implementation of our LLM-assisted grading pipeline is available at https://github.com/Integrated-Intelligence-Lab/llm assisted grading.git.

References

[1] John Hattie. Influences on student learning

[2] Darren Paul Fisher, Gaelle Brotto, Iris Lim, and Colette Southam. The impact of timely formative feedback on university student motivation

[3] Karen R Young, Henry E Schaffer, Jasmine B James, and Maria T Gallardo-Williams. Tired of failing students? improving student learning using detailed and automated individualized feedback in a large introductory science course

[4] Valerie J Shute. Focus on formative feedback

[5] Robert L Bangert-Drowns, James A Kulik, and Chen-Lin C Kulik. Effects of frequent classroom testing

[6] Paul Black and Dylan Wiliam. Developing the theory of formative assessment

[7] Frank C Leeming. The exam-a-day procedure improves performance in psychology classes

[8] Scott Freeman, Sarah L Eddy, Miles McDonough, Michelle K Smith, Nnadozie Okoroafor, Hannah Jordt, and Mary Pat Wenderoth. Active learning increases student performance in science, engineering, and mathematics

[9] Daniel Schwarcz and Dion Farganis. The impact of individualized feedback on law student performance

[10] Benjamin S Bloom. The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring

[11] Batya Elbaum, Sharon Vaughn, Marie Tejero Hughes, and Sally Watson Moody. How effective are one-to-one tutoring programs in reading for elementary students at risk for reading failure? a meta-analysis of the intervention research

[12] Samuel A Livingston. Constructed-response test questions: Why we use them; how we score them

[13] Andrew C Butler. Multiple-choice testing in education: Are the best practices for assessment also good for learning

[14] Henry L Roediger III and Elizabeth J Marsh. The positive and negative consequences of multiple-choice testing

[15] Yasuhiro Ozuru, Stephen Briner, Christopher A Kurby, and Danielle S McNamara. Comparing comprehension measured by multiple-choice and open-ended questions

[16]

1. Teacher-Aligned Grading Key Refinement for LLMs: Developing stable scoring criteria is paramount to reliable LLM performance, as grading keys must decompose solutions into small, explicitly enumerated steps, each associated with a fixed and relatively low point value (typically 2–3 points out of 10 per step). This prevents the model from assigning credit for mathematically irrelevant operations, as illustrated by the example in Figure 1.

2. Robust Anonymization Pipeline: Enhance student identification security by replacing student numbers with identifiers explicitly designed to maximize pairwise Hamming distance, which functions as a simple error-correcting identifier, allowing the intended value to be recovered reliably even when individual bubbles are partially filled. This mitigates OCR failure risks mentioned in the limitations section.

3. Adaptive LLM Aggregation Strategy: Implement a dynamic aggregation rule for five independent LLM evaluations based on context, moving beyond simple maximum selection to mitigate outliers. The paper suggests that selecting the maximum of the five LLM-generated scores protects students from accidental underscoring, but this should be balanced by using the median score in certain high-stakes contexts to achieve slightly higher κ values when using median-based aggregation.

4. Human Verification Trigger Optimization: Use quantitative metrics derived from LLM output variance to dynamically prioritize human review. The paper notes that responses exhibiting substantial disagreement across evaluations were automatically flagged for review, which should be operationalized by setting a threshold based on the computed anomaly score rather than relying solely on the presence of large outliers.

5. Contextual Prompt Grounding: Refine the grading prompt to explicitly enforce reasoning logic over pattern matching, instructing the model to Base all scoring strictly on evidence in the student’s solution; if evidence is missing, assign zero for that criterion and to ignore steps that do not follow from solid mathematical reasoning. This directly combats the LLM's tendency toward overly literal execution of underspecified criteria.

Providing timely and individualised feedback on handwritten student work is highly beneficial for learning but difficult to achieve at scale. This challenge has become more pressing as generative AI undermines the reliability of take-home assessments, shifting emphasis toward supervised, in-class evaluation. We present a scalable, end-to-end workflow for LLM-assisted grading of short, pen-and-paper assessments. The workflow spans (1) constructing solution keys, (2) developing detailed rubric-style grading keys used to guide the LLM, and (3) a grading procedure that combines automated scanning and anonymisation, multi-pass LLM scoring, automated consistency checks, and mandatory human verification. We deploy the system in two undergraduate mathematics courses using six low-stakes in-class tests. Empirically, LLM assistance reduces grading time by approximately 23% while achieving agreement comparable to, and in several cases tighter than, fully manual grading. Occasional model errors occur but are effectively contained by the hybrid design. Overall, our results show that carefully embedded human-in-the-loop LLM grading can substantially reduce workload while maintaining fairness and accuracy.