---
id: "lib-2026-10-04-reliability-of-ai-agents-rater-effects-drift-and"
type: "library"
title: "Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program"
lang: "EN"
org: "arXiv (micro1 data)"
source_type: "Paper"
date: "2026-10-04"
week: "2026-W41"
url: "https://nosov.ai/library/lib-2026-10-04-reliability-of-ai-agents-rater-effects-drift-and/"
original_url: "https://arxiv.org/abs/2610.07003"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "Economics working paper with a formal decision model and 2,611 rubric-scored production interviews, overlapping reviewers, a changelog and a ticket-based control series; single firm and preliminary."
summary_basis: "full-text"
original_minutes: "60"
tags: ["ai technology", "enablement", "agent evaluation", "ai interviewer", "hr tech"]
frames: ["verification-capacity"]
---

# Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program

Tier A (Strong evidence): Economics working paper with a formal decision model and 2,611 rubric-scored production interviews, overlapping reviewers, a changelog and a ticket-based control series; single firm and preliminary.

Summary based on the full text.

## In short

A working paper (arXiv econ.GN, 4 October 2026) by Liu Zhang and Mark Esposito. It studies how a firm should evaluate a deployed AI agent over time, using data from Zara, a voice-and-video agent that runs live job interviews for the recruiting firm micro1. The main finding is that raw human ratings can hide real change: a stricter reviewer joining flattened the trend, while reviewer-adjusted reliability rose. Evaluation evidence also goes stale within weeks because the agent drifts between logged releases.

## Key numbers

- 0.79 SD: severity gap between two reviewers scoring the same batches on the composite index
- 0.53 SD: rise in reviewer-adjusted reliability from March to August 2026
- about 5 weeks: time after which forecast uncertainty reaches the largest observed deploy effect
- about 10% per month: fall in interview-related support tickets relative to technical-issue tickets

## What it reports

- 2,611 production interview recordings scored in 14 batches between 26 March and 24 August 2026 on a nine-family rubric (structure, interruptions, latency, pacing, coherence, audio-visual sync, freezes, audio quality, outcome).
- The agent ran roughly 32,000 to 91,000 interviews a week, with volume close to doubling between March and September 2026.
- Two reviewers on the same batches differed by 0.79 standard deviations (SE 0.05) on the composite index. After adjusting for reviewer effects, reliability rose 0.53 standard deviations.
- Critical failures peaked at 6.3% and fell to 2.5%; 73% of critical failures were audio-visual sync failures.
- Three production deploys (12 April, 16 July, 29 July); the 12 April deploy moved the index +0.20 SD. Vendor model launches on 23 April, 7 July and 9 July served as placebo dates.
- Forecast standard deviation reaches 0.18 SD after four weeks, 0.25 after eight and 0.30 after twelve; it matches the largest deploy effect after about 5 weeks (range 1 to 21 weeks).
- Interview-related tickets (mean 46 per day) fell about 10% per month relative to technical-issue tickets (mean 312 per day).

## How they got it

The authors treat reliability as a hidden state measured by raters with fixed individual severity, and build a small decision model of evaluation and repair. Overlapping reviewers within batches identify severity; a state-space model smooths batch estimates and estimates week-to-week drift with and without deploys. Support tickets serve as an outside check, with technical-issue tickets as control.

## Limits

- No unevaluated comparison period, so the paper does not show that evaluation caused the improvement.
- One firm, one agent, essentially two reviewers; the drift horizon rests on a few step-like changes and is imprecise.
- Some critical failures may be recording failures rather than agent failures; vendors can change models served under a fixed name.

## Why it matters for companies

Any company tracking an AI agent with human ratings should check who did the scoring before reading a trend: a change of reviewers can look like a change in quality. Evaluation results have a shelf life of weeks, even without code releases, so periodic re-evaluation should be scheduled, not triggered only by deploys. The paper also shows that evaluation only pays off if there is engineering capacity to act on findings. Linking scores to an operational signal such as complaints gives a check that does not depend on the raters.

Original: https://arxiv.org/abs/2610.07003