Lessons Outlive Their Failures: A Case Study of Rules for AI Coding Agents
B · Useful, limited A single-author, self-described exploratory case study of one personal repository; the method is unusually explicit (failure ledger, answer-keyed audit test, counterfactual replays), but n=1 and three runs per condition keep it in the useful-practice tier.
Summary based on the full text·Original: about 35 minutes to read
In short
A preprint (arXiv, 23 October 2026) by independent researcher Hassan Hassanzadeh Aliabadi studying the rule set (AGENTS.md, skills, hooks, lint checks) that three coding-agent tools shared while editing his personal markdown repository daily. He logged every repeatable failure for six weeks, tested an LLM documentation audit against a known answer key, and replayed old failures with and without their fixes. Behavioural rules often turned out to be unnecessary on the newer model, while stale documentation kept returning and needed code checks to block it.
Key numbers
- 118agent failures logged over six weeks; stale documentation was the largest class at 48 (40.7%)
- 49.6%share of 47 known stale statements found by a single Claude Sonnet 5.5 audit run (63.8% across three runs)
- 916 to 3,543 wordsgrowth of the always-loaded AGENTS.md file in 14 days, before a lint cap cut it back
- 4,921 to 17,366 wordsgrowth of total rule text by Day 18; a manual cleanup cut it 32%, then it grew another 27%
- $3.55cost of the 30-session replay test that checked whether past fixes were still needed
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Failure classes: stale documentation 48, wrong facts stated by the agent 20, runtime or provider failures 14, plus smaller classes including destructive actions (6) and secret or privacy risks (4).
- 35 of the 48 drift failures were found on one audit day.
- A single Sonnet 5.5 audit found 49.6% of 47 real stale statements; Opus 5.5 averaged 41.1% and Haiku 5.5 26.2%. Planted errors were easier: 80.0% on Sonnet, 83.3% on Opus.
- Structural mismatches (cron schedules, renamed identifiers, dead paths) were caught reliably; inaccurate descriptions of script behaviour and an outdated deadline were missed in two-thirds of runs.
- Replays on Sonnet 5.5: the agent finished four of five tasks; in three of those the old failure did not recur with or without its fix. Only the stale-count failure returned without its lint check, and the check stopped it every time.
- A written rule on run-time estimates did not change Sonnet 4.5's overestimates; the guard hook blocked 13 of 19 recursive deletion commands.
- Rule text grew 3.5-fold in 18 days; lessons were added far faster than they were retired.
How they got it
Autobiographical single-case design. The failure ledger was built from commits, change logs, audit reports and a diary, coded in two LLM passes with disagreements resolved by the author (agreement kappa 0.65 for category, 0.89 for fix type). The audit test used 47 real stale statements and 20 planted errors across three models, three runs each. Replays ran in isolated git clones.
Limits
- One repository, one author who also built the harness and coded the data.
- Three runs per condition; the author labels results exploratory.
- 71 of 118 failures have unknown recurrence status.
- Model upgrades during the study confound whether rules or models changed behaviour.
Why it matters for companies
Teams that steer coding agents with rule files tend to keep appending rules after each mistake. This study suggests two practices: convert rules that matter into executable checks, because prose rules are advisory and documentation drift recurs; and periodically replay old failures after model upgrades to retire rules no longer needed, which here cost a few dollars. An LLM audit catches about half of stale documentation in one pass, so it supplements mechanical checks rather than replacing them.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org
- Tags
- Coding agents, Agents.md