What Does a Harness Buy? Tokens, Mostly
A · Strong evidence Controlled arXiv preprint from Shandong University that reruns identical configurations to measure noise, uses paired tests and power analysis, and releases its trial table and scripts.
Summary based on the full text·Original: about 45 minutes to read
In short
An arXiv preprint (3 October 2026) by Yangze Liu and Zhongyi Han of Shandong University. It tests how much the coding-agent harness (system prompt, tools, context management) changes results when the model is held fixed, running five models through Claude Code, mini-SWE-agent and OpenCode on SWE-bench Verified and rerunning identical configurations. The harness barely moves pass rate: swapping it changes outcomes as much as rerunning the same setup. What it does change is cost, by up to 3x.
Key numbers
- 13%median share of hard tasks whose outcome flips, both when swapping the harness and when rerunning the same one
- up to 3xcost per task difference across harnesses with the same model, tasks and price list
- 16,581 / 7,025 / 829tokens of fixed preamble sent every step by Claude Code, OpenCode and mini-SWE-agent
- 71%of early web-enabled DeepSeek and Qwen trials that fetched the upstream fix from GitHub
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- On 447 tasks, Claude Code and mini-SWE-agent differ by -1.8 points (Qwen3.6-35B-A3B) and -1.4 points (Qwen3.8-27B), equivalent within a 5-point margin.
- On the 45 hardest tasks, harness swaps and reruns each flip 13% of task outcomes at the median; swapping the model flips 22%.
- The only effect that clears the noise is a loss: OpenCode trails by +7.9 to +9.2 points on Qwen3.8-27B; 33 OpenCode trials there failed at an output cap with no recovery prompt, against one in Claude Code.
- Fixed preamble per step: 16,581 tokens (Claude Code), 7,025 (OpenCode), 829 (mini-SWE-agent). On GLM-5.3-Flash a trial costs 1.81 CNY, 0.97 CNY and 0.56 CNY respectively.
- 96% to 99% of input tokens are cache hits; cache pricing (Zhipu 0.287x, DeepSeek 0.033x of miss price) scales the bill without changing the harness ranking.
- DeepSeek's own harness on its own model passed 21 of 45, the same as Claude Code, at 1.04 CNY per trial against 0.98.
- A Claude Code release from a year earlier scored within the same band at 0.44x the list-price cost because extended thinking was off by default.
- Statistical resolution: 45 tasks catch a 12.9-point gap half the time; 447 tasks resolve 5.2 points at 80% power. Claude Opus 5 in Claude Code scored 36 of 45 (80%).
How they got it
Three pinned harness versions, five models (two served locally with vLLM, three via vendor APIs) plus Claude Opus 5 as an anchor, 300-step budget, run offline in containers via Harbor 0.22.0. Scoring used McNemar tests, Newcombe intervals and equivalence tests, with ten rerun pairs as the noise yardstick. Cost was computed from each trial's own token counts under one price list.
Limits
- One benchmark (SWE-bench Verified) plus one transfer check; general-purpose agents may behave differently.
- The closed frontier model ran only in Claude Code, so harness gaps for top closed models are untested.
- Small hard-task set limits power; OpenCode had shorter wall-clock time in some arms.
Why it matters for companies
Vendor claims that a harness or agent wrapper raises accuracy by a few points are often within run-to-run noise, so they should be checked with repeat runs before purchase decisions. The harness choice does drive the token bill, mostly through the fixed prompt it resends each step and the number of steps. For procurement, ask vendors for preamble size, step counts and repeat-run variance, not single benchmark scores. The finding that 71% of networked trials retrieved the answer from GitHub is also a warning about contaminated internal evaluations.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org