On the estimation and validity of AI time horizons: a statistical look at the METR plot
A · Strong evidence UC Berkeley statisticians re-fit METR's public data on 228 tasks and 26 AIs with cross-validated proper scoring rules; rigorous method on the most cited capability chart, but a single arXiv preprint rather than a field-shaping report.
Summary based on the full text·Original: about 45 minutes to read
In short
A statistical critique of METR's "time horizon" plot, the chart that shows AI agents handling ever-longer software tasks, by Drew T. Nguyen and William Fithian of the UC Berkeley Department of Statistics (arXiv, 8 October 2026). Using METR's public data, they replace the assumption that AI difficulty rises linearly with the log of human task time by spline and item-response-theory models. Their models predict success better on held-out data and show that the relationship is nearly flat between 2 and 30 minutes of human time. The practical result: a jump from 3 to 30 minutes is much easier than one from 30 minutes to 5 hours, even though both are 10x.
Key numbers
- 2-30 minband of human task time where AI difficulty is nearly flat, so horizon jumps there mean less than elsewhere
- 7.2 vs 4.4 minGPT-4o 50% time horizon under METR's baseline fit versus the authors' Model 2
- R2 = 0.996fit between Model 2 Rasch abilities (rescaled) and METR's published time horizons
- 6.0 monthsdoubling time of 50% time horizons under Model 2
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Data: 228 hand-designed software tasks, 79 task families, 26 AIs, released through Claude Mythos in May 2026.
- Two new specifications (Model 1, Model 2) beat the shared-slope logistic baseline on all 12 metrics in a 5-fold cross-validated suite (marginal log score, Brier score, elementary scores at q=0.5 and 0.8, three weightings).
- The shared-slope baseline itself beats METR's original per-AI logistic fit on 11 of 12 metrics.
- AI difficulty is nearly flat in log human time between about 2 and 30 minutes and close to linear elsewhere.
- GPT-4o's 50% horizon moves from 7.2 minutes (baseline) to 4.4 minutes (Model 2); some estimates for AIs in the 2-15 minute range move outside METR's original 95% confidence intervals.
- Most of the gain in fit comes from AIs with horizons of 2-15 minutes.
- Model 2 horizons still double every 6.0 months (R2 = 0.931). Rescaled Model 2 abilities track METR's horizons almost exactly (R2 = 0.996), so METR's line may be measuring a general "Rasch ability" more than a true time horizon.
- Current tasks reach up to 30 hours of human time; the authors see no reason the time-to-difficulty relationship stays linear for longer tasks.
How they got it
They treat the METR method as item-response theory with human time as an external anchor. They fit per-AI success curves, invert them to get horizons, and compare models out of sample with proper scoring rules and Murphy diagrams, folding by task family. They add two diagnostic charts: a time-to-difficulty conversion plot and conditional success trajectories at fixed human times.
Limits
- Only METR's software task suite; other domains (video, broader economic tasks) already show weak links between human time and AI success.
- The paper improves estimates and interpretation; it does not dispute that capabilities are rising fast.
- Preprint, not yet peer reviewed.
Why it matters for companies
The METR chart is used in board decks and policy debates as a forecast of what AI agents will soon do. This paper shows the y-axis is a statistical estimate whose meaning varies by task length: progress across short tasks is cheaper than it looks, progress on multi-hour work is harder. A manager planning automation of half-day or multi-day work should not extrapolate the doubling line directly. Ask what kind of tasks sit behind a horizon number, not only the number.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- AI capability