Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone
A · Strong evidence Amazon researchers build a benchmark of 280 twenty-request chains from 140 open-source projects, test ten coding agents with eight attempts per chain and decompose the failure; clear method and checkable numbers.
Summary based on the full text·Original: about 100 minutes to read
In short
A preprint (1 November 2026) by Alejandro Cuadron and Arpit Gupta at Amazon. Coding agents are usually tested one request at a time on human-written code. BOBA-Bench chains 20 consecutive requests, each built on whatever the agent left. All ten agents tested do far worse on chains. The cause is mistakes the agents' own tests miss, which later requests then build on.
Key numbers
- 95% vs 51%average completion of requests taken one at a time vs requests chained together
- 53%how often agents are right when asked whether an implementation is correct
- 51% to 59%chain completion after one external 'that's wrong' message and one repair round per failed request
- 5,600requests in BOBA-Bench (280 chains of 20, from 140 Python repositories)
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Ten agents, from GLM-5.3-Flash to Claude Opus 5.5, complete 95% of requests on average one at a time but 51% when chained; only 39% of chains finish without a single missed request or broken earlier test.
- On verified code, agent-written code is nearly as buildable as the developers' own, within 4 points for every model. Unverified states are the problem.
- Agents declare a request done once their own tests pass; the next request then fails about twice as often, and the mistake is rarely found later.
- Asked to review whether an implementation is correct, agents are right 53% of the time.
- One "verification failed" message plus one repair round lifts chain completion from 51% to 59% on average, with per-model gains from 1 to 22 points, at the cost of 3 to 9 points more chains with a broken test.
- Told only that verification failed, agents fix a tenth to a third of their misses; a quarter to a half when the failing test is named.
How they got it
Chains built from real commit histories of 140 Python repositories, with hidden tests after every request to separate verified work from silent misses. Pass@1 is estimated from eight attempts per chain. The degradation is decomposed by who wrote the underlying code and whether it was verified.
Limits
- Python only; chains derive from commit histories, which may be more linear than real feature work.
- Hidden tests define correctness, so misses the tests also miss are not counted.
- The paper's own figures differ by measure: 95% single-request completion in the abstract, 66-84% for the individual requests inside chains.
Why it matters for companies
Agents left to run long sequences without independent checks accumulate errors that nobody sees, and asking the agent to review itself barely helps. Budget for external verification after each change: tests the agent did not write, CI, or a human check. The return on even a bare failure signal is large, which makes verification infrastructure the cheapest reliability investment for agentic coding.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org