Scaling Up Counseling with AI: Evidence from a Nationwide Experiment
A · Strong evidence Inter-American Development Bank economists with McGill and Boston University co-authors run a registered, IRB-approved nationwide field experiment comparing an AI chatbot with trained human counsellors, with costs and per-counsellor spread reported.
Summary based on the full text·Original: about 90 minutes to read
In short
IDB Working Paper IDB-WP-01887 (October 2026) by Nicolás Ajzenman, Ana Teresa del Toro Mijares, Gregory Elacqua, Catalina Hermosilla and Santiago Veleda. Chile's teacher shortage motivated a 10-week campaign to steer high-school seniors toward education degrees, delivered either by trained human counsellors or by Kai, a GPT-4o mini chatbot, both on WhatsApp. Among students already interested in teaching, the chatbot raised first-choice education applications; the human arm's effect was smaller and not significant. The chatbot's main advantage was uniform quality, which the authors call controlled variance.
Key numbers
- N≈42,000Chilean high-school seniors in the nationwide experiment, reached over WhatsApp
- 1.4 percentage pointsrise in ranking an education major first in the AI arm, among students with baseline interest (control mean 26.1%)
- roughly two-thirdsshare of the 45 human counsellors that the chatbot matched or outperformed
- $6.20 vs $1.95cost per student contacted, human arm versus chatbot including amortised development
- $223 vs $123cost per additional student induced to rank an education major first, human versus chatbot
As reported by the source.
English·Web page·opens d.repec.org
What it reports
- Among students with baseline interest, the AI arm raised the probability of ranking an education major first by 1.4 pp (control mean 26.1%) and the share of education majors in the choice set by 1.1 pp (control mean 24.4%), both significant at 5%.
- Human arm: 1.1 pp on first choice, not significant; choice-set effect about half the chatbot's point estimate.
- No effect among students without baseline interest, as the targeting predicted.
- Quality varied widely across the 45 human counsellors; the best ones were highly effective, and Kai matched or beat roughly two-thirds of them.
- Cost: $6.20 per student contacted for humans, $1.95 for Kai with development amortised over five years ($0.95 without). Per induced student: about $223 versus $123 ($60 without development).
- Kai conversations were faster and more semantically coherent and focused on facts such as admission rules and scholarships. Counsellors scored much higher on empathy and motivation, and their conversations were slightly longer (8 versus 7 messages).
- Enrolment effects are not interpreted causally, because centralised admission links treated and control students' outcomes.
How they got it
A randomized experiment with an NGO partner (Elige Educar), timed to Chile's centralised college application cycle. Scripts and training materials were shared across arms, and Kai's prompts were tuned in pilots to match counsellor style. Outcomes come from administrative application data. The study was approved by an institutional review board and registered (AEARCTR-0014942). Conversation text was analysed for speed, coherence, topic breadth and rater-coded empathy, though conversations could not be linked to individual outcomes for privacy reasons.
Limits
- Effects are small in absolute terms (one to two percentage points) and apply to one decision on one margin.
- The human-arm null may partly reflect power, though the authors argue the sample was large enough; the cost comparison takes an insignificant point estimate at face value.
- Students knew from the first message whether they were talking to a person or a chatbot.
- Conversation features cannot be tied to outcomes, so the mechanism evidence is descriptive.
Why it matters for companies
The result transfers to any conversation-based programme that has to scale through a broad pool of staff: onboarding, career guidance, benefits enrolment, internal mobility nudges. The gain came less from a better average than from removing the weak tail of human delivery. The design lesson is a hybrid: the chatbot handles fast, consistent factual delivery, and the strongest human staff handle the empathy and motivation work where they kept a clear advantage. Measure the spread across human deliverers before deciding where automation pays.
Summary based on the full text. Read the original for the full argument and data.Read the original on d.repec.org
- Tags
- Work and AI