Cache Engineering for Agents
In the authors' cost breakdown, KV cache reads account for 82.4% of total API cost for agents.
82.4%share of total API cost taken by KV cache reads in the authors' agent cost breakdown
Inference cost
Week 43 · 19-25 Oct
3 sources filed · EN 3
B A Understanding KV Residency in Production-Scale Agentic LLM Workloads
In Deep Research the application, not the person, issues 89% of model calls; median context grows from about 10K tokens to 68K by the 30th call.
~10K to 68Kmedian context tokens at the first call vs the 30th call in Deep Research
B We fine-tuned a 235B model on 4 GPUs. Then we didn't need it
Task: find entities in long structured documents and label them from a fixed set; scored per tenant with F1.
75.47 to 83.10average F1 across nine tenants, base vs fine-tuned Qwen3-235B-A22B
Week 42 · 12-18 Oct
2 sources filed · EN 1 · JA 1
B Why harnesses matter for energy (and token) efficiency
Uncached input ranged from 1.6% to 11.1% by client over 30 days.
1.6% to 11.1%uncached input share across agent clients, 8 September to 7 October 2026
A Moegi: sharing service-specific language models internally
Public pre-trained models lack service-specific vocabulary, spelling variants and model numbers; separate pre-training by each product duplicates data, training, evaluation and maintenance work.
0.895 / 0.853 / 0.760Moegi bert-110m scores on Shopping query, Shopping product and Search query classification