The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial (IZA DP 18989)
A · Strong evidence A University of Maryland team ran an instructor-level randomised trial with 2,379 students and 30 instructors, reporting intention-to-treat effects on grades and platform activity with the design and its weak spots stated.
Summary based on the full text·Original: about 50 minutes to read
In short
IZA Discussion Paper 18989 (October 2026) by Jing Liu and colleagues at the University of Maryland. In Fall 2025, instructors were randomly given a GPT-4o tutor with retrieval over their own course materials, built into the learning platform. Students in treated sections got lower final grades and engaged much less with the course site. Losses were larger for first-generation students.
Key numbers
- 0.37 standard deviationsdrop in final grades from tutor access, sections of the same course (0.27-0.37 SD across samples)
- 0.90 standard deviationsdrop in recorded participation in instructor-designed online activities
- 14.63%treated students who ever used the AI tutor
- 73.8%tutor requests seeking information, explanations or solutions; 0.7% asked for feedback on own work
As reported by the source.
English·PDF·opens docs.iza.org
What it reports
- 2,379 undergraduates and 30 instructors across STEM, social sciences, humanities and education; randomisation by instructor within blocks of similar courses.
- Final grades fell 0.27-0.37 SD, clearest in the exact-match sample of sections of the same course. Homework effects were null or negative.
- In the exact-match sample: course page views -0.37 SD, active days -0.51 SD, participation in online activities (discussions, submissions, quizzes) -0.90 SD; similar in the full sample.
- Only 14.63% of treated students ever used the tutor. Use peaked around exams.
- Estimated grade losses for first-generation students were more than twice those of continuing-generation students.
- Requests: 73.8% sought information, explanations or solutions, 11.3% practice and test preparation, 0.7% feedback on own work.
- Tutor users reported asking instructors fewer content questions and sending fewer messages.
- Only two instructors switched from direct-answer mode to the guided tutoring mode.
How they got it
A randomised controlled trial during one 16-week semester, with intention-to-treat estimates on grades and learning-platform logs, plus end-of-term surveys of students and instructors and a classification of tutor requests. The authors also report instrumental-variable estimates to probe mechanisms.
Limits
- 34 clusters, so cluster-robust standard errors may overstate precision; the authors flag this.
- With 15% uptake, the grade effect may come partly from how instructors introduced the tool, not from tutor use itself.
- Students in both arms had free access to ChatGPT and Gemini through the university.
- Instructors could read student-tutor exchanges, which may have changed what students asked.
- One semester, one university, mostly direct-answer configuration.
Why it matters for companies
Embedding an answer engine into an existing learning system can displace the activities that produced learning, while measured use stays low. Corporate learning and onboarding teams adding AI assistants to their platforms should measure outcomes, not usage, and keep guided modes on by default. The participation drop is the warning sign to track: it appeared even among people who barely used the tool.
Summary based on the full text. Read the original for the full argument and data.Open PDF on docs.iza.org