---
id: "lib-2026-10-05-verified-not-generated-expert-verified-ai-study"
type: "library"
title: "Verified, not generated: expert-verified AI study materials and the distribution of learning gains"
lang: "EN"
org: "arXiv"
source_type: "Paper"
date: "2026-10-05"
week: "2026-W41"
url: "https://nosov.ai/library/lib-2026-10-05-verified-not-generated-expert-verified-ai-study/"
original_url: "https://arxiv.org/abs/2610.07097"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "King's Business School working paper with a two-cohort difference-in-differences design on real exam records, explicit robustness checks and honest reporting of a fragile mean, but one course and 170 students."
summary_basis: "full-text"
original_minutes: "60"
tags: ["enablement", "l&d", "learning gains"]
---

# Verified, not generated: expert-verified AI study materials and the distribution of learning gains

Tier A (Strong evidence): King's Business School working paper with a two-cohort difference-in-differences design on real exam records, explicit robustness checks and honest reporting of a fragile mean, but one course and 170 students.

Summary based on the full text.

## In short

A working paper by Canh Thien Dang and An Nguyen (King's Business School, King's College London) on AI-generated study materials in a compulsory first-year economics course. The macroeconomics half got NotebookLM podcasts, FAQs and quiz-based study guides, each checked by a named graduate teaching assistant before release and labelled "AI-generated and academically verified". The average gain was small and statistically fragile, but the share of weak marks fell sharply and roughly three-quarters of the gain sat in the bottom fifth of students. The authors argue the decisive factor is who carries the "judgement burden" of checking AI output.

## Key numbers

- -24.7 pp: change in share of marks below the upper-second boundary, relative to the counterfactual (p = 0.001)
- 2.34 marks: average advantage on a 50-mark exam component for the half with verified AI materials
- ~75%: share of the average gain that came from the bottom quintile (+8.9 marks)
- 1.8 min: staff verification time per student (about 2.5 hours across five weeks)

## What it reports

- 170 students across two cohorts of 85, 340 exam marks; the 2024/25 Part 2 cell was the only treated cell.
- Access was associated with a 2.341-mark advantage on a 50-mark component (p = 0.045 HC1, 0.013 clustered), equal to 0.33 SD of the pre-period Part 2 distribution.
- The effect is protection from a decline: treated-half marks rose 0.73 (not significant) while the untreated half fell 1.61 (p = 0.021).
- Marks below 30/50 (the upper-second boundary) moved from 23.5% to 10.6% in the treated half and from 7.1% to 18.8% in the untreated half: a -24.7 pp difference, about 21 students in a cohort of 85.
- Effects were significant at every threshold from 23 to 31 marks and at none above; the 10th percentile gained 11.0 marks, the 20th 5.0, the 30th upward near zero.
- Part 2 standard deviation fell from 7.09 to 5.03 (Brown-Forsythe p = 0.004).
- Dropping the lowest 3 to 8 pre-period students pulls the mean to 1.88-1.41 marks and out of significance; the threshold effect stays between -19.9 and -22.9 pp (p ≤ 0.006).
- Each podcast reached more than 70 unique listeners (about 82%); FAQs and guides were opened by about 65% and 60%. Verification took about 30 minutes per weekly set.

## How they got it

OLS difference-in-differences on exam marks, with the untreated microeconomics half of the same course as a contemporaneous control and the prior cohort as the baseline. Same lecturer, teaching assistant, rubric and marker in both years; the 2024/25 paper was drafted alongside the 2023/24 paper. Distributional tests (threshold models, quantile DiD with 1,000 bootstrap draws) plus field notes from 36 student interviews.

## Limits

- One course, one institution, one pre-period cohort; parallel trends cannot be tested and effort substitution between halves is not ruled out.
- Verification is bundled with new multimodal material, so its separate contribution is not identified; no individual usage logs.
- High-attaining cohort with no failures, so nothing is learned about failure rates.

## Why it matters for companies

Corporate learning teams are generating AI training content at near-zero cost. This study suggests the value lands on weaker learners only when an accountable, named expert checks the material before release. Verification is the real cost line and scales with content volume, not headcount. Evaluations that report only average scores can miss whether the intended group gained at all.

Original: https://arxiv.org/abs/2610.07097