Why harnesses matter for energy (and token) efficiency
B · Useful, limited Neuralwatt's CEO reports measured per-request energy from 30 days of production traffic plus a recorded lab test of four agent harnesses; method is shown, but the data is the vendor's own and it sells the dashboard.
Summary based on the full text·Original: about 14 minutes to read
In short
Chad Gibson, CEO of inference provider Neuralwatt (16 October 2026), shows that the agent harness, the client software that builds prompts, decides much of the GPU work and energy per agent session. Prompt caching only works if earlier text stays byte-identical; harnesses that edit the start of a prompt force the server to recompute it. Most cache breaks found were fixable with settings or small code changes.
Key numbers
- 1.6% to 11.1%uncached input share across agent clients, 8 September to 7 October 2026
- about 2.7 timesprefill work at 92% cached vs 97% cached
- roughly twiceenergy of a mostly uncached request vs a cached one of the same size (about 2.5x on GLM-5.3)
- 65 of 539follow-up requests in lab tests that broke the cache, all caused by the harness editing earlier prompt text
As reported by the source.
English·Web page·opens neuralwatt.com
What it reports
- Uncached input ranged from 1.6% to 11.1% by client over 30 days. A cleaner measure, breaks per turn: about 1 in 100 for deepseek-harness vs 3 in 100 for OpenCode and Pi.
- A mostly uncached request uses roughly twice the energy of a cached one at the same size; about 2.5 times on GLM-5.3.
- Past 128k tokens, about 3 turns in 1,000 re-read more than half the prompt; they account for 75% of re-read tokens at that depth, each costing as much prefill as about 600 normal turns.
- Lab test of OpenCode, Pi, omp and Kilo Code: 88% of 539 follow-up requests were pure appends; the other 65 broke the cache, every one from the harness changing text it had already sent. Idle time, restarts and sub-agents caused none.
- Causes: OpenCode 1.x plan-mode reminders, omp pruning re-read files, switching reasoning effort (written in the first tokens), MCP tools inserted mid-list, dates in the system prompt, and compaction.
- In production, tool-list or effort changes make a break more than 20 times as likely; tool changes on about 0.5% of turns account for 19% of re-read tokens.
How they got it
Request metadata and measured GPU energy from production traffic (no prompt text logged), plus scripted sessions through a recording proxy against DeepSeek V4.1 Flash and GLM-5.3, comparing each request's tokens with the previous one.
Limits
- Vendor data from one provider's customer mix; the post promotes its own dashboard.
- Server side cannot always separate harness edits from cache eviction.
- Lab sessions were short scripted runs on a small Python project.
Why it matters for companies
Agent bills and energy use depend on the harness as much as the model. Keeping system prompts static, appending changes at the end, keeping tool lists stable and choosing reasoning effort once per session cut cost without changing the model. Teams standardising on coding agents should test cache hit rates of candidate harnesses on their own workloads before rollout.
Summary based on the full text. Read the original for the full argument and data.Read the original on neuralwatt.com
- Tags
- Inference cost, Caching