Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success
A · Strong evidence Single-author preprint (Naman Vats, BIT Mesra) with a pre-specified, placebo-controlled design, confidence intervals and released code; a narrow but cleanly measured negative result on cheap-model control of agents.
Summary based on the full text·Original: about 15 minutes to read
In short
An arXiv paper (17 October 2026) by Naman Vats. Agent builders increasingly hand small control decisions (stop, verify, change approach) to cheap models to save money. The author held a Claude Sonnet 5.5 terminal agent fixed and varied only the controller: none, fixed rules, a small LLM (Claude Haiku 5.5), a commercial typed decision model (Jev 1.13), and a placebo. The decision model made runs 19% more expensive with no change in success. It ranked finished runs better than the alternatives offline but was badly calibrated, so it kept asking the agent to re-check work that was already correct.
Key numbers
- 19%higher cost per attempt when the Jev decision model controlled the agent (ratio 1.19, 95% CI 1.02-1.36)
- 73.0% vs 73.0%pass rate with and without the decision-model controller on held-out tasks
- 315trials on 21 held-out Terminal-Bench 2.1 tasks across five controller arms
- 0.50 vs 87%decision model's mean probability a run was complete vs the share of those runs that actually passed
- AUROC 0.78decision model's offline ranking of finish requests, vs 0.62 for rules and 0.60 for a small LLM
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Held-out results (63 attempts per arm): no controller 73.0% pass at $0.177 per attempt; rules 73.0% at $0.191; small LLM 76.2% at $0.213; Jev 73.0% at $0.206.
- Jev vs no controller: cost ratio 1.19 (95% CI 1.02-1.36), pass rate identical. Jev was cheaper than the small-LLM controller (ratio 0.81) but 22% more expensive than the placebo.
- No pass-rate difference between arms was detectable.
- Offline, Jev ranked finish requests best (AUROC 0.78 vs 0.62 rules, 0.60 small LLM), but its mean probability that a run was complete was 0.50 when 87% of those runs passed.
- No controller ever chose to abandon a run. A Jev decision cost about $0.00008.
How they got it
Pre-specified study on Terminal-Bench 2.1 with one shared agent loop in Daytona sandboxes via Harbor. Four deterministic triggers (repeated failure, no progress, finish request, 10-step checkpoint) are logged in every arm; controllers can only append fixed advisory messages, finish or abandon, and guards override them. The placebo samples Jev's own development-phase choice frequencies at the same moments. Costs are provider-reported; comparisons use bootstrap intervals with Holm adjustment.
Limits
- 21 tasks, mostly easy, so pass-rate intervals are about ±11-13 points; the cost result is more robust than the success result.
- One reasoning model, one decision model, one prompt wording; an open-weight replication could not be run.
- Sonnet 5.5 cannot be run at fixed temperature and hosts were not pinned.
- The placebo rests on a small development sample.
Why it matters for companies
Adding a cheap "supervisor" model to an agent loop is a common cost-saving idea, and this is a clean test showing it can raise cost instead. A signal that ranks well is not the same as one you can act on: if confidence is poorly calibrated, verification prompts fire on work that is already done. Teams should measure any controller end to end on cost and success against a no-controller baseline and a random-intervention placebo before rolling it out.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Cost of AI