Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program
A · Strong evidence Economics working paper with a formal decision model and 2,611 rubric-scored production interviews, overlapping reviewers, a changelog and a ticket-based control series; single firm and preliminary.
Summary based on the full text·Original: about 60 minutes to read
In short
A working paper (arXiv econ.GN, 4 October 2026) by Liu Zhang and Mark Esposito. It studies how a firm should evaluate a deployed AI agent over time, using data from Zara, a voice-and-video agent that runs live job interviews for the recruiting firm micro1. The main finding is that raw human ratings can hide real change: a stricter reviewer joining flattened the trend, while reviewer-adjusted reliability rose. Evaluation evidence also goes stale within weeks because the agent drifts between logged releases.
Key numbers
- 0.79 SDseverity gap between two reviewers scoring the same batches on the composite index
- 0.53 SDrise in reviewer-adjusted reliability from March to August 2026
- about 5 weekstime after which forecast uncertainty reaches the largest observed deploy effect
- about 10% per monthfall in interview-related support tickets relative to technical-issue tickets
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- 2,611 production interview recordings scored in 14 batches between 26 March and 24 August 2026 on a nine-family rubric (structure, interruptions, latency, pacing, coherence, audio-visual sync, freezes, audio quality, outcome).
- The agent ran roughly 32,000 to 91,000 interviews a week, with volume close to doubling between March and September 2026.
- Two reviewers on the same batches differed by 0.79 standard deviations (SE 0.05) on the composite index. After adjusting for reviewer effects, reliability rose 0.53 standard deviations.
- Critical failures peaked at 6.3% and fell to 2.5%; 73% of critical failures were audio-visual sync failures.
- Three production deploys (12 April, 16 July, 29 July); the 12 April deploy moved the index +0.20 SD. Vendor model launches on 23 April, 7 July and 9 July served as placebo dates.
- Forecast standard deviation reaches 0.18 SD after four weeks, 0.25 after eight and 0.30 after twelve; it matches the largest deploy effect after about 5 weeks (range 1 to 21 weeks).
- Interview-related tickets (mean 46 per day) fell about 10% per month relative to technical-issue tickets (mean 312 per day).
How they got it
The authors treat reliability as a hidden state measured by raters with fixed individual severity, and build a small decision model of evaluation and repair. Overlapping reviewers within batches identify severity; a state-space model smooths batch estimates and estimates week-to-week drift with and without deploys. Support tickets serve as an outside check, with technical-issue tickets as control.
Limits
- No unevaluated comparison period, so the paper does not show that evaluation caused the improvement.
- One firm, one agent, essentially two reviewers; the drift horizon rests on a few step-like changes and is imprecise.
- Some critical failures may be recording failures rather than agent failures; vendors can change models served under a fixed name.
Why it matters for companies
Any company tracking an AI agent with human ratings should check who did the scoring before reading a trend: a change of reviewers can look like a change in quality. Evaluation results have a shelf life of weeks, even without code releases, so periodic re-evaluation should be scheduled, not triggered only by deploys. The paper also shows that evaluation only pays off if there is engineering capacity to act on findings. Linking scores to an operational signal such as complaints gives a check that does not depend on the raters.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org