---
id: "lib-2026-10-08-on-the-estimation-and-validity-of-ai-time"
type: "library"
title: "On the estimation and validity of AI time horizons: a statistical look at the METR plot"
lang: "EN"
org: "UC Berkeley"
source_type: "Paper"
date: "2026-10-08"
week: "2026-W41"
url: "https://nosov.ai/library/lib-2026-10-08-on-the-estimation-and-validity-of-ai-time/"
original_url: "https://arxiv.org/abs/2610.12466"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "UC Berkeley statisticians re-fit METR's public data on 228 tasks and 26 AIs with cross-validated proper scoring rules; rigorous method on the most cited capability chart, but a single arXiv preprint rather than a field-shaping report."
summary_basis: "full-text"
original_minutes: "45"
tags: ["ai technology", "capability forecasting", "metr"]
---

# On the estimation and validity of AI time horizons: a statistical look at the METR plot

Tier A (Strong evidence): UC Berkeley statisticians re-fit METR's public data on 228 tasks and 26 AIs with cross-validated proper scoring rules; rigorous method on the most cited capability chart, but a single arXiv preprint rather than a field-shaping report.

Summary based on the full text.

## In short

A statistical critique of METR's "time horizon" plot, the chart that shows AI agents handling ever-longer software tasks, by Drew T. Nguyen and William Fithian of the UC Berkeley Department of Statistics (arXiv, 8 October 2026). Using METR's public data, they replace the assumption that AI difficulty rises linearly with the log of human task time by spline and item-response-theory models. Their models predict success better on held-out data and show that the relationship is nearly flat between 2 and 30 minutes of human time. The practical result: a jump from 3 to 30 minutes is much easier than one from 30 minutes to 5 hours, even though both are 10x.

## Key numbers

- 2-30 min: band of human task time where AI difficulty is nearly flat, so horizon jumps there mean less than elsewhere
- 7.2 vs 4.4 min: GPT-4o 50% time horizon under METR's baseline fit versus the authors' Model 2
- R2 = 0.996: fit between Model 2 Rasch abilities (rescaled) and METR's published time horizons
- 6.0 months: doubling time of 50% time horizons under Model 2

## What it reports

- Data: 228 hand-designed software tasks, 79 task families, 26 AIs, released through Claude Mythos in May 2026.
- Two new specifications (Model 1, Model 2) beat the shared-slope logistic baseline on all 12 metrics in a 5-fold cross-validated suite (marginal log score, Brier score, elementary scores at q=0.5 and 0.8, three weightings).
- The shared-slope baseline itself beats METR's original per-AI logistic fit on 11 of 12 metrics.
- AI difficulty is nearly flat in log human time between about 2 and 30 minutes and close to linear elsewhere.
- GPT-4o's 50% horizon moves from 7.2 minutes (baseline) to 4.4 minutes (Model 2); some estimates for AIs in the 2-15 minute range move outside METR's original 95% confidence intervals.
- Most of the gain in fit comes from AIs with horizons of 2-15 minutes.
- Model 2 horizons still double every 6.0 months (R2 = 0.931). Rescaled Model 2 abilities track METR's horizons almost exactly (R2 = 0.996), so METR's line may be measuring a general "Rasch ability" more than a true time horizon.
- Current tasks reach up to 30 hours of human time; the authors see no reason the time-to-difficulty relationship stays linear for longer tasks.

## How they got it

They treat the METR method as item-response theory with human time as an external anchor. They fit per-AI success curves, invert them to get horizons, and compare models out of sample with proper scoring rules and Murphy diagrams, folding by task family. They add two diagnostic charts: a time-to-difficulty conversion plot and conditional success trajectories at fixed human times.

## Limits

- Only METR's software task suite; other domains (video, broader economic tasks) already show weak links between human time and AI success.
- The paper improves estimates and interpretation; it does not dispute that capabilities are rising fast.
- Preprint, not yet peer reviewed.

## Why it matters for companies

The METR chart is used in board decks and policy debates as a forecast of what AI agents will soon do. This paper shows the y-axis is a statistical estimate whose meaning varies by task length: progress across short tasks is cheaper than it looks, progress on multi-hour work is harder. A manager planning automation of half-day or multi-day work should not extrapolate the doubling line directly. Ask what kind of tasks sit behind a horizon number, not only the number.

Original: https://arxiv.org/abs/2610.12466