Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents
A · Strong evidence A multi-university software-engineering study (Wake Forest, Griffith, NTU, UNSW, SUSTech) that codes 120 real defects with three independent raters, builds a detector with an ablation, and validates it on 571 unseen repositories with upstream confirmations; recall on the labelled set is in-sample, which the authors state.
Summary based on the full text·Original: about 40 minutes to read
In short
A paper (arXiv, October 2026, v2 dated 27 October) by Ying Zhang (Wake Forest), Ruoqi Guo and Yi Liu (Griffith) and colleagues. Providers offer prompt caching to cut the cost of resending identical prefixes, but each provider configures it differently and the rules change. The authors studied 120 caching defects in ten popular open-source agent applications and found that almost all fail silently: the agent still answers correctly, it just pays full price. They built CacheLint, a static checker that found 32 previously unknown defects across 571 other projects.
Key numbers
- 120prompt-caching defects found in merged pull requests of 10 popular open-source agent applications
- 108of those defects that raised no error and showed only as higher cost and latency
- 37defects caused by hard-coded model-name checks, the largest single root cause
- 0.86precision of the CacheLint detector on 571 unseen agent projects (32 true findings)
- 18upstream fixes from the findings, including one in Anthropic's .NET SDK (21 confirmed)
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- 97 of 120 defects (81%) are on the request side, where the agent builds the request so the cache cannot be used; 23 are on the read side, where usage figures are misparsed.
- 108 defects produced no error; they appear only as higher cost and latency in usage metrics.
- Largest causes: caching switched on by hard-coded model names, so a renamed or new model silently loses caching (37 defects); prefixes rebuilt in a different order on each call (15); a per-request value such as a timestamp placed inside the cached prefix (13).
- Fixes are small and local, for example truncating a timestamp to the date so the prefix stays stable for a day.
- Provider changes also break hit rates in unchanged code; the paper cites Anthropic cutting the default cache lifetime from one hour to five minutes.
- CacheLint scored 0.71 precision and 0.78 recall on the labelled set and 0.86 precision on 571 unseen repositories. Without its LLM guard check precision fell to 0.31; without its dated provider-capability table, to 0.05.
- Of reported findings, 21 were confirmed and 18 fixed upstream, including a null-handling bug in Anthropic's official .NET SDK that hid cache-usage counts.
How they got it
From 148,947 merged pull requests in ten agent applications (among them Cline, OpenHands, Aider, Goose and AutoGPT), keyword search gave 622 candidates; three authors screened them to 120 defects and coded root cause, symptom and fix independently, resolving disagreements by consensus. CacheLint finds caching call sites with CodeQL and checks each precondition against a dated table of provider capabilities or with an LLM check. Each warning on the 571 repositories was inspected by hand and reported upstream.
Limits
- Recall on the 571 repositories is unknown; the 0.78 recall is measured on the corpus that shaped the rules.
- Open-source GitHub projects only; closed-source agents may differ.
- Provider capabilities and LLM judgements change; the capability table is a dated snapshot.
- CacheLint covers 80% of the defect types in the dataset.
Why it matters for companies
Prompt caching is one of the largest cost levers for agents that resend long system prompts and context. This paper shows the saving is easy to lose without any visible failure: a timestamp in the system prompt or a model rename can switch it off. Teams running agents in production should monitor cache-hit rates as a cost metric, check that usage fields are parsed, and gate caching on provider capability rather than model names.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Prompt caching, Cost of AI