Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance
A · Strong evidence A NeurIPS 2026 paper from Luxembourg, Tilburg, Gran Sasso and Naples researchers with a fault model mined from public incident data, a full-factorial design, paired statistics, stated limits and released artifacts; the model range is narrow.
Summary based on the full text·Original: about 60 minutes to read
In short
A paper (arXiv, 23 October 2026, NeurIPS 2026) by Antonio Ken Iannillo and colleagues at the University of Luxembourg, Tilburg University, Gran Sasso Science Institute and the University of Naples. Agent benchmarks assume tools answer correctly and on time. The authors built MCP-Injector, a middleware that injects realistic tool failures between an agent and its MCP servers, calibrated on public incident reports. Under those faults task-completion scores fell by 60%, the larger model was no more robust than the smaller one, and the agent framework's error handling made more difference than model size.
Key numbers
- 57,240issue and pull-request threads mined from 1,774 open-source MCP repositories to calibrate the faults
- 8.08 to 3.23task-completion score out of 10 before and under injected faults, a 60% drop
- p= 0.96test of whether the 120B model keeps more of its fault-free score than the 20B model
- 91.1% vs. 95.6%completion rate under faults for the 120B and 20B models
- 1,350experimental cells: two models, three agent stacks, three conditions, 25 tasks, three seeds
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Of 14,829 classified failures in the mined threads, explicit errors were the largest group (10,393). The resulting fault profile is 91.3% permanent faults.
- Pooled across stacks and models, task-completion score fell from 8.08 to 3.23 out of 10; tool selection fell 59% and planning effectiveness 67%. The drop held (57%) when only runs that completed in both conditions were compared.
- The most common outcome was partial degradation (56% of cells): the agent finishes, but with much lower quality.
- gpt-oss-120B kept no larger share of its fault-free score than gpt-oss-20B (p= 0.96) and completed fewer tasks under faults (91.1% vs. 95.6%).
- LangChain had the highest completion under faults (95 to 97%), the OpenAI Agents SDK the lowest (88 to 95%). The authors call the framework effect suggestive, not established.
- About 6.0% of runs under faults scored well while barely using tools, answering from memory; the judge rubric rewards some of these ungrounded answers.
How they got it
The authors mined 103,166 threads from 1,774 MCP server repositories, filtered to 57,240, classified them with GPT-4o and checked a sample by hand (agreement of at least 0.80). The resulting mix of failure modes and persistence drove six fault operators. They ran 25 MCP-Bench tasks on two locally served gpt-oss models across three agent stacks, with no proxy, a pass-through proxy and the fault profile, three seeds each. A GPT-4o-mini judge scored each run five times; analysis used paired tests and mixed-effects models.
Limits
- Two models from one family on one consumer hardware setup; the scale finding covers only these two.
- The benchmark is easy without faults (95 to 100% completion), so harder workloads may behave differently.
- GitHub issues over-represent persistent, reproducible failures; transient and intermittent faults fired only 9 of 1,905 times.
- Fault exposure and framework error handling are partly entangled because stacks make different numbers of tool calls.
- Single-server tasks only; no multi-agent or human-in-the-loop setups.
Why it matters for companies
Agent pilots are usually tested against tools that work. In production, connectors fail, time out and return malformed data, and this paper shows that quality can more than halve when they do, while the run still "completes". Buying a larger model does not fix it. The cheaper lever is the orchestration layer: how errors are surfaced, retried or switched. Fault injection belongs in acceptance testing for any agent that calls business systems.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Mcp, Agent reliability