Machine-human agreement in marking undergraduate assignments is specific to model, prompt and institution
A · Strong evidence Cambridge-led multi-site study of 761 formally marked assignments from three UK universities, three frontier models and a fully crossed prompt design, with code and replication data released.
Summary based on the full text·Original: about 45 minutes to read
In short
Giulio Corsi, Maryam Abo-Tabik and colleagues tested whether GPT-5.4, Claude Opus 4.6 and Gemini 3 Flash can mark undergraduate psychology assignments the way university markers do. Agreement with human marks depended on the model, the prompt and the institution, and no combination was best everywhere. All three models pulled marks towards the middle, over-marking weak work and under-marking strong work. Because the models made the same errors, averaging them did not help.
Key numbers
- 761undergraduate psychology assignments from Cambridge, Manchester Metropolitan and Nottingham
- 0.616 vs 0.345best quadratic weighted kappa with human marks at Cambridge vs Manchester Met
- 63.4%assignments where all three models erred in the same direction, 3.4 times the 18.8% expected if errors were independent
- 0.74-0.78partial correlations between the models' signed errors after controlling for the human mark
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- The best method reached a QWK of 0.616 at Cambridge, 0.420 at Nottingham and 0.345 at Manchester Met.
- No model dominated: GPT was strongest at Cambridge (Spearman 0.671) but weakest at Manchester Met; Claude led at Manchester Met; Gemini was the least biased everywhere.
- At Cambridge, machine-human rank agreement sits within the range human markers reach with each other (0.47-0.93); at the other two sites it falls towards or below the lower bound.
- All models compressed the scale. Signed error correlated with the human mark at -0.79 to -0.83.
- Errors were shared: all three erred in the same direction on 63.4% of assignments, against 18.8% expected by chance. The weighted-mean ensemble was statistically indistinguishable from the best single model.
- Machine marks rewarded linguistic features more strongly than human marks did.
- A simple post-hoc rescaling improved agreement (pooled QWK 0.393 to 0.459), but requires human-marked calibration data.
How they got it
761 assignments marked under each degree's standard process. Each model marked zero-shot under a 3 x 3 x 3 factorial prompt design varying criterion detail and other prompt factors, with four aggregation strategies. Agreement was measured with QWK, exact and adjacent accuracy, rank correlation and error metrics; error independence with partial correlations.
Limits
- One discipline (psychology) and three UK institutions.
- Zero-shot marking only; fine-tuned or few-shot setups with local examples were not the main test.
- Human marks are treated as ground truth, though human markers also disagree.
Why it matters for companies
The result applies to any use of LLMs to grade people's work, including hiring assessments, written tests in training, and performance write-ups. Validation in one setting does not transfer to another, and a panel of models does not cancel bias when they share it. Organisations need local calibration against their own human ratings, and should watch for compression that rewards the average and penalises the best.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Work and AI, L&d, Evaluation