---
id: "lib-2026-10-17-good-judgment-bad-advice-decision-model"
type: "library"
title: "Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success"
lang: "EN"
org: "arXiv"
source_type: "Paper"
date: "2026-10-17"
week: "2026-W43"
url: "https://nosov.ai/library/lib-2026-10-17-good-judgment-bad-advice-decision-model/"
original_url: "https://arxiv.org/abs/2610.24649"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "Single-author preprint (Naman Vats, BIT Mesra) with a pre-specified, placebo-controlled design, confidence intervals and released code; a narrow but cleanly measured negative result on cheap-model control of agents."
summary_basis: "full-text"
original_minutes: "15"
tags: ["ai technology", "model routing", "agent cost", "negative results"]
---

# Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success

Tier A (Strong evidence): Single-author preprint (Naman Vats, BIT Mesra) with a pre-specified, placebo-controlled design, confidence intervals and released code; a narrow but cleanly measured negative result on cheap-model control of agents.

Summary based on the full text.

## In short

An arXiv paper (17 October 2026) by Naman Vats. Agent builders increasingly hand small control decisions (stop, verify, change approach) to cheap models to save money. The author held a Claude Sonnet 5.5 terminal agent fixed and varied only the controller: none, fixed rules, a small LLM (Claude Haiku 5.5), a commercial typed decision model (Jev 1.13), and a placebo. The decision model made runs 19% more expensive with no change in success. It ranked finished runs better than the alternatives offline but was badly calibrated, so it kept asking the agent to re-check work that was already correct.

## Key numbers

- 19%: higher cost per attempt when the Jev decision model controlled the agent (ratio 1.19, 95% CI 1.02-1.36)
- 73.0% vs 73.0%: pass rate with and without the decision-model controller on held-out tasks
- 315: trials on 21 held-out Terminal-Bench 2.1 tasks across five controller arms
- 0.50 vs 87%: decision model's mean probability a run was complete vs the share of those runs that actually passed
- AUROC 0.78: decision model's offline ranking of finish requests, vs 0.62 for rules and 0.60 for a small LLM

## What it reports

- Held-out results (63 attempts per arm): no controller 73.0% pass at $0.177 per attempt; rules 73.0% at $0.191; small LLM 76.2% at $0.213; Jev 73.0% at $0.206.
- Jev vs no controller: cost ratio 1.19 (95% CI 1.02-1.36), pass rate identical. Jev was cheaper than the small-LLM controller (ratio 0.81) but 22% more expensive than the placebo.
- No pass-rate difference between arms was detectable.
- Offline, Jev ranked finish requests best (AUROC 0.78 vs 0.62 rules, 0.60 small LLM), but its mean probability that a run was complete was 0.50 when 87% of those runs passed.
- No controller ever chose to abandon a run. A Jev decision cost about $0.00008.

## How they got it

Pre-specified study on Terminal-Bench 2.1 with one shared agent loop in Daytona sandboxes via Harbor. Four deterministic triggers (repeated failure, no progress, finish request, 10-step checkpoint) are logged in every arm; controllers can only append fixed advisory messages, finish or abandon, and guards override them. The placebo samples Jev's own development-phase choice frequencies at the same moments. Costs are provider-reported; comparisons use bootstrap intervals with Holm adjustment.

## Limits

- 21 tasks, mostly easy, so pass-rate intervals are about ±11-13 points; the cost result is more robust than the success result.
- One reasoning model, one decision model, one prompt wording; an open-weight replication could not be run.
- Sonnet 5.5 cannot be run at fixed temperature and hosts were not pinned.
- The placebo rests on a small development sample.

## Why it matters for companies

Adding a cheap "supervisor" model to an agent loop is a common cost-saving idea, and this is a clean test showing it can raise cost instead. A signal that ranks well is not the same as one you can act on: if confidence is poorly calibrated, verification prompts fire on work that is already done. Teams should measure any controller end to end on cost and success against a no-controller baseline and a random-intervention placebo before rolling it out.

Original: https://arxiv.org/abs/2610.24649