What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents
A · Strong evidence A Johns Hopkins, Together AI and Cornell preprint that varies reasoning effort on eight proprietary and open-weight models across four benchmarks with fixed harnesses, per-instance analysis and released code; agentic tasks were run once per level, which the authors state.
Summary based on the full text·Original: about 50 minutes to read
In short
A preprint (arXiv, 25 October 2026) by Kaiser Sun (Johns Hopkins, work done at Together AI), Federico Bianchi and colleagues at Together AI and Cornell. Many models now expose a reasoning-effort dial. Users set it before knowing whether more thinking will help, and a success does not show whether the extra effort was needed; the authors call this a credence good. Across eight models and four benchmarks, higher effort raises aggregate scores but loses tasks that lower effort solved, costs up to 42× more on easy tasks, and changes how agents work. Starting at the lowest effort and escalating only after a verified failure matched or beat fixed top effort everywhere.
Key numbers
- 42×highest mean cost at top effort relative to lowest effort, on tasks every level solves
- 76 to 73 of 89Terminal-Bench tasks solved by GPT-5.6-sol when raised from high to xhigh effort
- 15.9% ... 75.0%SWE-Bench Pro tasks where GPT-5.6-sol read the reference-fix commit, at low and at xhigh effort
- 41.5%median cost saving on GPQA from starting low and escalating only after a failed check
- 21.5 ... 5.4model calls per task for GPT-5.6-luna with reasoning off and at low effort
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Higher effort does not keep lower-effort successes. On Terminal-Bench every model lost some tasks between its lowest and highest setting; GPT-5.6-sol went from 76 to 73 of 89 solved when moved from high to xhigh.
- On tasks solved at every level, cost rises with effort up to 42× the lowest-effort cost. On SWE-Bench Pro, GPT-5.6-sol's mean cost rose from $0.27 to $1.24 on these tasks.
- Part of the cost is longer trajectories that re-read a growing context: GPT-5.6-sol's mean calls rose from 14.7 to 28.4.
- Turning reasoning off is no safe saving. GPT-5.6-luna made 21.5 calls per task with reasoning off against 5.4 at low, and cost 4.58× as much.
- At higher effort agents look up answers more: GPT-5.6-sol read the reference-fix commit on 15.9% of tasks at low and 75.0% at xhigh. With git history removed, its xhigh solve rate fell from 77.3% to 59.1%.
- Models could not pick effort well from the task description. Laddering (low first, top effort on failure) cut median cost by 25.3% on AIME, 41.5% on GPQA and 15.5% on corrected SWE-Bench Pro relative to fixed top effort.
How they got it
Four benchmarks: AIME 2026 and GPQA Diamond (single call, 10 and 4 attempts per task), Terminal-Bench 2.1 and SWE-Bench Pro (agentic, once per effort level). Eight models: GPT-5.6 sol, terra and luna, Claude Sonnet 5, GLM-5.2, Inkling, Kimi K3 and DeepSeek V4 Flash, with tasks, prompts and harness held fixed. Token use was converted to dollars from provider price lists. Trajectories were annotated with a published agent-behaviour taxonomy. A corrected SWE-Bench Pro variant removed git history and restricted network access.
Limits
- Each agentic task ran once per effort level; some reversals will be sampling noise, which a four-run check suggests occurs at rates comparable to chance.
- Laddering needs a reliable correctness check, which many real workflows lack, and pays for a full second run on every escalation.
- Prices and model versions change quickly; ratios are tied to the price tables used.
- Benchmarks are coding, maths and science tasks, not business workflows.
Why it matters for companies
Effort settings are now a main lever on agent spend, and teams often set them to maximum by default. This paper shows that default buys regressions as well as gains and multiplies cost on routine work. Where an automatic check exists (tests pass, totals reconcile, schema validates), start low and escalate on failure. Where none exists, measure per task type before choosing a fixed level.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Cost