Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents
A · Strong evidence A Peking University and BUPT team builds a validated 90-case benchmark from eight public datasets and runs simple harnesses, five published agents and non-LLM methods on the same cases across six models, with repeat passes and cost and time per case; method shown and numbers checkable.
Summary based on the full text·Original: about 50 minutes to read
In short
An arXiv paper (26 October 2026) testing whether hand-designed investigation procedures help LLM agents find the root cause of microservice failures. The authors compare three harnesses that leave every step to the model (one call over an evidence package, a SQL agent over raw telemetry, and the Codex coding agent) with five published root-cause agents built on standard operating procedures, fixed stages and role agents, plus statistical and causal methods. On 90 validated incidents, the simple harnesses win on every model tested. A case study shows prescribed procedures skipping the decisive evidence. A stronger model mostly buys reliability; more reasoning effort adds little.
Key numbers
- 65-77%top-1 root-cause accuracy of simple harnesses on three mid-tier models
- 16-53%top-1 accuracy of five published, hand-crafted root-cause agents on the same models
- 38.9%top-1 accuracy of the best non-LLM method (NSigma), used as the usability threshold
- 2-4 pointsaccuracy gained from more reasoning effort, at 1.5 to 4 times the wall time
- 85.6%accuracy of DeepSeek V4.1 Flash with Codex at about five cents per case
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Top-1 accuracy (PR@1) on three mid-tier models: simple harnesses 65-77%, published agents 16-53%, non-LLM methods 6-39%.
- With the strongest models tested, the SQL agent and Codex reach 84-89%; the best run (GPT-6.1 Sol at xhigh effort with Codex) gets 88.9% at about 26 cents and 277 seconds per case.
- Published agents fix between a tenth and all of their model calls in code (prescribed-call ratio 0.10 to 1.00); the simple harnesses fix none. In the case study, one agent's matched procedure never read the logs that held the answer.
- Moving from a weaker to a stronger model raised PR@1 by 14-18 points while the agents made fewer calls. Raising reasoning effort added 2-4 points at 1.5 to 4 times the wall time (the abstract says up to seven times).
- Over three passes, the stronger model cut unreliability (solved only sometimes) from 32-36% to 7-8% of cases. On the weaker model, single passes of Codex ranged from 62% to 76%, enough to flip method rankings.
- Every run on the cost and time Pareto frontier uses a simple harness. Practical picks: Codex with GPT-6 Luna at 68.9% for under one cent per case; Codex with DeepSeek V4.1 Flash at 85.6% for about five cents; SQL agent with Sol at 85.6% in 45 seconds.
How they got it
RCAtlas pools eight public RCA datasets (nine systems, from Online Boutique to a bank, a telecom and a cloud platform) into one format. Of 542 cases screened, 348 passed eight exclusion rules; 90 were sampled across 23 scenarios. Injection-tool traces were removed. One scorer maps answers to the labelled level (service, pod or node). Six models, three reasoning efforts, and three passes for two models; published methods were changed only where they could not run or had bugs.
Limits
- Codex is the only general coding harness tested; Claude Code and others may differ.
- All models are from 2026; budget excluded the top frontier models, and older, weaker models may favour prescribed workflows.
- Datasets and reimplemented methods may differ from their originals.
- The mechanism (prescription skips evidence) rests on one case study and a ratio, not an ablation.
Why it matters for companies
IT operations teams and vendors are building incident agents that encode runbooks as fixed steps. This paper suggests that, with current models, a general agent given read access to metrics, traces, logs and events and left to choose what to query is both more accurate and cheaper. Runbook knowledge is still useful as guidance the model may consult. For buyers, the practical test is to run any vendor RCA agent against a simple SQL or coding agent on your own past incidents, with several passes, before paying for the extra machinery.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Agent harnesses