Cache Engineering for Agents
B · Useful, limited A Tsinghua systems paper with open code and measured results on four agent benchmarks and three open models; solid engineering, but a method proposal evaluated by its authors and relevant mainly to teams that run their own model serving.
Summary based on the full text·Original: about 45 minutes to read
In short
A preprint (arXiv, 20 October 2026) by Tianyu Fu, Ruoxi Wang, Zihan Min and colleagues at Tsinghua University and CUHK. Long agent runs make the stored attention cache (the KV cache) the dominant cost. Today's harnesses cut context by trimming or summarising history, which forces the model to recompute everything after the edit. The authors let the agent drop selected cache entries and shift positions directly, and build a serving engine (AKV) that handles the bookkeeping. Accuracy goes up slightly and cost falls by more than a quarter.
Key numbers
- 82.4%share of total API cost taken by KV cache reads in the authors' agent cost breakdown
- 1.7 pointsaverage benchmark score gain of a tool-response rolling-drop policy over threshold summarisation
- 27.7%average cost reduction against conventional threshold-triggered summarisation
- 1.5×speedup in P95 request latency with the AKV serving engine
- 7.1%further cost reduction from cache policies refined by recursive self-improvement over expert design
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- In the authors' cost breakdown, KV cache reads account for 82.4% of total API cost for agents.
- Two new operations: drop (exclude selected cache entries from future attention) and repos (adjust position encodings of retained keys), alongside the normal append.
- Identical token sequences can produce different cache states under different operation histories, so standard prefix matching cannot reuse caches correctly; AKV manages this behind a stateless API.
- A rolling-drop policy keeping the latest 12 tool responses improves average scores by 1.7 points and cuts cost 27.7% versus threshold summarisation, and 21.9% versus text-level trimming.
- Serving gains: up to 1.2x output throughput and 1.5x faster P95 latency.
- Policies refined by recursive self-improvement cut cost a further 7.1% over expert design.
How they got it
Experiments on MiniMax-M2.7, GPT-OSS-120B and MiroThinker-1.7-Mini across BrowseComp-Plus, xbench-DeepSearch, the GAIA text subset and SWE-Verified, comparing against standard threshold summarisation and text-level context engineering. Costs use published token prices, with repositioning charged conservatively. Serving was measured on eight A800 GPUs on SGLang.
Limits
- Requires control of the serving stack; it cannot be applied through a closed vendor API.
- Tested on three open models; linear-attention models are not supported.
- The accuracy gain is small and the comparison baselines are the authors' implementations of common harness behaviour.
Why it matters for companies
For firms buying agents through vendor APIs, the main lesson is economic: long-running agents are priced by cache reads and re-prefill, so how a harness compresses history drives the bill more than token counts suggest. Teams self-hosting open models for agent workloads can test cache-level policies for cost savings of this order. For everyone else, it is a sign that vendors may expose cache controls as an API feature.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Inference cost, Caching