---
id: "lib-2026-10-16-understanding-kv-residency-in-production-scale"
type: "library"
title: "Understanding KV Residency in Production-Scale Agentic LLM Workloads"
lang: "EN"
org: "arXiv (IISc, Microsoft)"
source_type: "Paper"
date: "2026-10-16"
week: "2026-W43"
url: "https://nosov.ai/library/lib-2026-10-16-understanding-kv-residency-in-production-scale/"
original_url: "https://arxiv.org/abs/2610.22863"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "IISc and Microsoft M365 Research characterise four production agentic workloads across more than 121 million model calls and test memory policies by trace replay; rare production-scale evidence with a clear method, though the traces cannot be shared and memory sizes rest on an assumed model layout."
summary_basis: "full-text"
original_minutes: "50"
tags: ["ai technology", "inference cost", "caching", "production traces"]
---

# Understanding KV Residency in Production-Scale Agentic LLM Workloads

Tier A (Strong evidence): IISc and Microsoft M365 Research characterise four production agentic workloads across more than 121 million model calls and test memory policies by trace replay; rare production-scale evidence with a clear method, though the traces cannot be shared and memory sizes rest on an assumed model layout.

Summary based on the full text.

## In short

An arXiv paper (16 October 2026) by Mayank Arya and Yogesh Simmhan (Indian Institute of Science) with Microsoft M365 Research. Agent applications turn one request into a long chain of model and tool calls over a growing context, and keeping the computed key-value (KV) state in GPU memory lets later calls skip recomputing it. Using four production traces, the authors show this state is large, idle most of the time, and often lost when calls are routed to a different deployment. Compressing idle state works better than smarter eviction alone, and routing matters as much as memory.

## Key numbers

- 121 million: model calls across four Microsoft M365 production traces (Deep Research, Enterprise Assistant, Coding Agent, Enterprise Chat)
- ~10K to 68K: median context tokens at the first call vs the 30th call in Deep Research
- 5.6%: share of the time retained context state is actually used by model inference
- 89% to 29%: drop in cached-context reuse when the next call to the same model lands on a different deployment
- 63% vs 13.6%: Deep Research calls recomputing context under LRU eviction vs int4 demotion (MRR) at one tenth of peak memory

## What it reports

- In Deep Research the application, not the person, issues 89% of model calls; median context grows from about 10K tokens to 68K by the 30th call.
- About 80% of context tokens are served from the prefix cache; a predictable class of models accounts for 67% of call-context KV.
- Using a public Llama-2-70B layout as a stand-in, one 99th-percentile Deep Research session exceeds the 80 GB memory of an H100 GPU. Holding all live sessions at peak would need an estimated 501.8 TB of GPU memory, while inference uses that state only 5.6% of the time it is held.
- A one-hour idle cap cuts that ceiling 3.6x while raising recomputing calls only from 4.07% to 4.53%.
- At one tenth of peak memory, LRU eviction makes 63% of Deep Research calls recompute; a clairvoyant policy gets 31%; demoting idle sessions to int4 (MRR) gets 13.6%.
- When the next call goes to another deployment in the region, cache reuse falls from 89% to 29%. Sessions touch 4.5 deployments on average, lifting the recompute floor from 5.4% to 20.2% regardless of memory.

## How they got it

Telemetry from four M365 workloads between late June and early August 2026 (Deep Research: 17M calls in 691k sessions over eight days; Chat: 104M calls over 24 hours), with session structure, token accounting, model identity and, for three workloads, the deployment serving each call. Memory needs were estimated with a fixed public KV layout because production models are confidential. Policies were compared by trace-driven replay.

## Limits

- Four workloads at one company over short windows; the authors warn against treating results as universal.
- Byte figures scale with the assumed per-token KV size of a public model, not the real production models.
- The deployment label may hide worker-level placement or cache sharing; several results depend on analysis choices.
- Production traces cannot be released; a synthetic generator (AgenticTraceSim) is planned, not yet available.

## Why it matters for companies

Agent workloads cost more per request than chat partly because of this context handling, and much of the cost depends on infrastructure choices buyers rarely see. Firms running their own inference should look at session-aware routing and compressed caching before buying more GPUs. Firms buying from providers should ask how prompt caching behaves across long agent sessions and whether cached-token discounts survive routing, since that decides the real price of agentic use.

Original: https://arxiv.org/abs/2610.22863