Context Is a Pipeline, Not a Window: Information Attrition in LLM Agent Harnesses
A · Strong evidence A short paper by Sehgal, Panahandeh and Ezzati-Jivan (Brock University, University of Ottawa) that measures where tool output is lost inside one agent harness, tests the effect with predictions written down in advance, reports a refuted prediction, and ships a replication package; small samples, but the mechanism is shown and checkable.
Summary based on the full text·Original: about 20 minutes to read
In short
An arXiv paper (26 October 2026) on a failure that context-window size and answer accuracy both hide: tool output deleted by the harness before the model reads it. The authors studied their own agent, which diagnoses injected faults from Linux kernel traces. The window was nearly empty, yet a third of tool results were cut by slicing the JSON text at a fixed length, which removed whichever fields came last. Diagnosis accuracy split along the same line as the lost fields. In a controlled test the cut mostly cost extra work rather than wrong answers, because the agent found another route to the culprit. Two of seven popular agent frameworks cut tool output the same way.
Key numbers
- 5.75%share of the context window used by the largest prompt across 720 agent runs
- 3,827 of 11,189tool results cut by the harness before reaching the model, every one leaving broken JSON
- 4.7% vs. 99.3%delivery rate of two fields sitting in the same tool results, decided only by their position
- 43%rise in tool calls when the truncation defect is reintroduced with keys reversed, at equal accuracy
- 0.61 to 0.31fall in yield (findings recorded per computation run) under the defect
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Across 720 runs the largest prompt was 23k tokens against a 400k window (5.75%). Still, 3,827 of 11,189 tool results were truncated, and each became invalid JSON.
- Within the cut results, one container list reached the model in 4.7% of 742 calls and a neighbouring list in 99.3%. Faults that start a new process appear in the surviving list and scored 40-55 out of 60; faults in already-running containers appear in the lost list and scored 0-24 out of 60.
- Replaying 864 recorded results: every result over the budget broke under the prefix slice and none under a row-aware cut. Only 2.5% of calls were affected, which is why the defect goes unnoticed.
- On one production-scale trace (21 containers, a 33,370-character result, 5.6 times the budget), the shipped cut delivered 0 of 128 records of the evidence class that held the culprit.
- Controlled experiment (145 runs): with the defect, the agent lost a median 27 results per run and still named the spawned culprit in 15 of 15 runs. Tool calls rose from 53 to 68 to 76, computations from 18 to 32 to 36, and yield fell from 0.61 to 0.34 to 0.31.
- Other harness boundaries also leak: 51% of 554 workers stopped exactly at the step limit, and 98 passed on nothing, which the system read as "nothing found".
- Framework survey: langchain-core keeps the first 500 characters of serialized tool output; smolagents deletes the middle of observations over 20k characters. Only one structure-aware cutter was found, and it is not a default.
How they got it
Field data from the authors' own root-cause agent, then deterministic replays of stored tool results through both cutting rules with no model involved. The controlled experiment reintroduced one defect at a time with agent, model (temperature 0), tools and task fixed, on 5 recordings per fault class with 3 repeats, scoring only an explicit named culprit against ground truth. Predictions were written before each wave and plan changes logged. A first scorer that counted mentions was found to inflate results and replaced.
Limits
- One harness, one domain (kernel traces), one model; the authors built the defect themselves.
- Five recordings per class repeated, not independent fault instances.
- Whether the truncation causes the accuracy split in field data remains open; the production-scale replay (3 of 6 against 0 of 5 correct) points the same way but is noisy.
- Yield is a narrow proxy for work reaching the decision.
Why it matters for companies
Teams evaluating agents watch accuracy, latency and window size, and this defect barely moves any of them: model wait time was 80-91% of wall clock in the traces measured. The cost shows up as extra tool calls and tokens, and as silent failure when evidence has only one route to the model. Anyone running agents over logs, CI output or database results should check how their framework truncates tool output, prefer cuts that keep whole records and say what was dropped, and log what the model received alongside what the tool produced.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org