Automated Feedback to Peer Tutors Increases Their Talk Moves and Learner Reasoning but Not the Balance of Talk
A · Strong evidence A preregistered randomized controlled trial led by Dorottya Demszky (Stanford) with Cornell and Schoolhouse.world, replicated across two cohorts, with interviews and a frank limitations section; causal evidence on behaviour, not on learning.
Summary based on the full text·Original: about 30 minutes to read
In short
An arXiv paper (4 November 2026) by Stanford, Cornell and Schoolhouse.world authors on a randomized trial in two four-week SAT maths bootcamps on a free online peer-tutoring platform. Volunteer tutors received no feedback, automated feedback on their own talk from session transcripts, or that plus participation feedback sent to their learners. Tutor feedback raised the targeted talk moves and learner reasoning by 8-15%, replicated in both bootcamps. Tutors still did about 90% of the talking, and learner-facing feedback added nothing.
Key numbers
- 1,030 / 7,443 / 6,508volunteer tutors, learners and recorded sessions in the trial
- 8%increase in tutors eliciting learner ideas (1.41 more times per hour over a control mean of 17.16)
- 15%increase in learner reasoning (0.24 more times per hour)
- 91.5%tutor share of talk time in the control group, unchanged by any feedback condition
- 77.9% and 56.0%share of treated learner-sessions where learner feedback was actually delivered, by bootcamp
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Four arms: control, tutor feedback, and tutor plus learner feedback under a social or a goal-oriented framing; 1,199 tutor assignments across 1,030 distinct tutors.
- Tutor feedback: eliciting learner ideas +1.41 per hour (8%), uptake of learner contributions +0.52 per hour (9%), learner reasoning +0.24 per hour (15%); each about 0.12 control standard deviations, significant after multiple-comparison correction.
- Share of learners speaking rose 2.4 points (p=.054), not conventionally significant.
- No effect on tutor talk time (control mean 91.5%), and no effect of learner-facing feedback on discourse; practice test performance did not change.
- Delivery of learner feedback was incomplete: 77.9% of treated learner-sessions in the first bootcamp, 56.0% in the second.
- Interviews with 17 tutors: they adopted the recommended moves but were limited by learners who did not respond. Interviews with 13 learners: feedback reported a group-level statistic, some preferred to listen, and learner-to-learner talk was rare.
- Suggestions in the feedback were generated by GPT-4 Turbo.
How they got it
Tutors were randomised across conditions; the analysis plan was registered (AEARCTR-0014050) before outcome data were analysed, and deviations are reported. Discourse measures come from classifiers applied to session transcripts; estimates are intention-to-treat regressions pooled across bootcamps, with robustness checks. Qualitative interviews explain mechanisms.
Limits
- Four-week window and no subsequent SAT scores, so effects on learning are unknown.
- Classifiers detect the presence of a move, not its quality; only two were validated on this platform's transcripts.
- No log data on whether tutors opened feedback; some control tutors had received feedback before the trial.
- Volunteer, mostly high-school tutors on one platform; the framing comparison is underpowered.
Why it matters for companies
Automated feedback built from meeting or session transcripts can change specific behaviours at low cost, which applies to coaching, sales calls, onboarding and internal training as well as tutoring. The trial also shows the limit: it changed how tutors asked, not how much others spoke, and feedback to participants did nothing. Programmes that promise engagement gains from AI feedback alone should expect small, specific effects and test them with a control group before scaling.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org