I Deleted 168,000 Lines of Comments. My AI Agent Didn't Notice
B · Useful, limited A careful first-hand test by an OpenStation developer, with a blind judge, a hidden test and published data; well documented, but one model, two tasks and three runs per branch.
Summary based on the full text·Original: about 5 minutes to read
In short
Roberto Aranda removed every comment from his company's codebase, about a third of its source, and had Claude Code with Opus 5.5 build the same two features on both versions, three times each. Every run passed all checks, and a blind judge scored the results within noise of each other. His main lesson is methodological: run-to-run variation was so large that single runs could have supported opposite conclusions.
Key numbers
- 168,000lines of comments removed from the codebase (765,491 lines total down to 597,332)
- 7.01M to 10.64Mtoken spread across the three no-comment runs of the bigger task
- 30% / 31%opposite cost conclusions you could draw by picking single runs
- about 22%fewer tokens without comments on the small task
As reported by the source.
English·Web page·opens openstation.blog
What it reports
- Small task (tile open windows, score out of 19): 18.3 with comments, 19.0 without; tokens 2.18M vs 1.71M; cost $1.42 vs $1.04.
- Bigger task (keep a window on top, score out of 23): 21.7 vs 22.0; 9.06M vs 9.12M tokens; $3.84 vs $3.95.
- The bigger task had a trap (a PHP function dropping unknown session keys); all six runs found it, with or without comments.
- Runs with comments used 8.21M, 8.79M and 10.17M tokens; without, 7.01M, 9.70M and 10.64M. Picking one run per branch could show removal as 30% more expensive or 31% cheaper.
- Conclusion: comments neither help nor hurt the agent; keep them for human readers.
- Teaser: removing the docs/ folder and AGENTS.md did make a difference, reported in a later post.
How they got it
Two branches, compiled to verify identical code. Same prompt, model and settings; runs alternated between branches. Checks: lint, type checks, existing tests, plus a hidden test added after the agent finished. A separate Claude session scored each change against a pre-written checklist with branch names hidden. Code, diffs, rubrics and full results are linked.
Limits
- One model, two tasks, three runs per branch: enough to rule out a large effect, not a small one, as the author says.
- The judge is the same model family as the agent.
- One codebase whose comments were mostly agent-written.
Why it matters for companies
Teams measuring agent productivity or cost from one run per condition are reading noise. Budget for repeated runs before concluding that a prompt, tool or codebase change helps. On the substance, code comments are not where agent context comes from; project-level documentation appears to matter more, so that is where to invest.
Summary based on the full text. Read the original for the full argument and data.Read the original on openstation.blog
- Tags
- Coding agents, Evaluation