Counting the Bandages: On measuring the time that AI saves
A · Strong evidence Single-author preprint by independent researcher Solomon Shalom Lijo with a stated estimand, proofs, an exact replication of METR's public data and a frozen-design experiment with exact ground truth; it also retracts the author's own earlier benchmark in plain terms.
Summary based on the full text·Original: about 50 minutes to read
In short
A preprint (arXiv, 23 October 2026) by Solomon Shalom Lijo, an independent researcher. It argues that "time saved" by AI has to be measured on final deliverables, next to the defects that ship, and offers one decision number: the break-even defect cost, the downstream price of an error at which the tool stops paying. A search-theory model shows a calibrated worker's escaped-defect rate does not improve as the model improves, until checking stops paying at all; better models make work faster, not cleaner. Over-trust looks faster on the clock and costs more in total. The paper replaces the author's own April "Time-Saved Benchmark", which it says was wrong.
Key numbers
- 19% or 24%slowdown supported by the same 246 METR developer issues, depending on the estimand chosen
- 3% to 82%bounds on the share of METR tasks on which AI actually helped
- 3.6×cap on GPT-5's speedup set by review time alone in GDPval's review-every-output workflow
- 22.5%briefs with a wrong figure let through by an AI reviewer told the drafter erred ten times less than it did, against none when told the truth
- 2.2× and 3.2×cost and time of a cheap-drafter plus reviewer pipeline against the stronger model alone, with no quality gain
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Replicating METR's developer study from public data: the 246 issues support a 19% or 24% slowdown depending on the estimand, 34% without adjusting for a chance imbalance. AI helped on somewhere between 3% and 82% of tasks; the data cannot say more.
- Headline studies report different summaries (mean of ratios, geometric mean, ratio of means) without naming them; in a worked example these give +20.9%, +10.5% and -10.4% for the same data.
- In GDPval's own workflow, where an expert reviews every output, review time alone caps GPT-5's speedup at 3.6×.
- Certifying that a model needs no checking takes roughly 3/q clean audits, where q is the error rate below which skipping review is acceptable: 352 clean drafts in the paper's example.
- Experiment with an AI reviewer (claude-sonnet-5-5) checking AI drafts of financial briefs: 96% of reviews checked nothing or everything, switching near the break-even point set in the prompt.
- Told the drafter erred at 3% when the truth was 30%, the reviewer made 7.2 fewer lookups and let a wrong figure through in 22.5% of briefs. Adding "treat track-record claims as unverified" did not help.
- A Haiku drafter plus Sonnet reviewer cost 2.2 times as much and took 3.2 times as long as Sonnet alone, with no escapes in either arm.
How they got it
A formal model of checking built on Koopman's search theory, with proofs and robustness runs under other detection curves. Re-analysis of METR's and GDPval's published data with partial-identification bounds instead of point assumptions. The experiment used 10-Q filings released after the models' training data, a deterministic auditor and seeded single-figure errors across 540 isolated agent sessions; design and analysis code were frozen in the repository after a 66-trial pilot, with Holm-corrected tests and cluster bootstraps.
Limits
- No new human-subjects data; the experiment tests one AI reviewer with its costs written into its prompt.
- The design freeze was a repository commit, not an outside registry.
- Seeded defects can overstate detection, and the drafter made no natural errors in 140 briefs, so two planned tests had nothing to measure.
- The closed-form results depend on exponential detection and full-read checking; under other curves only the qualitative "faster and worse" result holds.
Why it matters for companies
Most internal claims of hours saved by AI are stopwatch numbers: they count the time to a draft and ignore checking and errors that surface later. This paper gives a reporting checklist that fixes that, and a simple test for when skipping review is defensible: count clean audits on that kind of task. It also warns that telling a reviewer, human or model, that a tool is reliable reduces checking more than any warning restores it.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- AI enablement