---
id: "lib-2026-10-13-i-deleted-168-000-lines-of-comments-my-ai-agent"
type: "library"
title: "I Deleted 168,000 Lines of Comments. My AI Agent Didn't Notice"
lang: "EN"
org: "OpenStation"
source_type: "Practitioner post"
date: "2026-10-13"
week: "2026-W42"
url: "https://nosov.ai/library/lib-2026-10-13-i-deleted-168-000-lines-of-comments-my-ai-agent/"
original_url: "https://openstation.blog/2026/10/13/i-deleted-168000-lines-of-comments-my-ai-agent-didnt-notice/"
tier: "B"
tier_name: "Useful, limited"
tier_reason: "A careful first-hand test by an OpenStation developer, with a blind judge, a hidden test and published data; well documented, but one model, two tasks and three runs per branch."
summary_basis: "full-text"
original_minutes: "5"
tags: ["coding agents", "evaluation", "context"]
frames: ["verification-capacity", "tacit-to-explicit"]
---

# I Deleted 168,000 Lines of Comments. My AI Agent Didn't Notice

Tier B (Useful, limited): A careful first-hand test by an OpenStation developer, with a blind judge, a hidden test and published data; well documented, but one model, two tasks and three runs per branch.

Summary based on the full text.

## In short

Roberto Aranda removed every comment from his company's codebase, about a third of its source, and had Claude Code with Opus 5.5 build the same two features on both versions, three times each. Every run passed all checks, and a blind judge scored the results within noise of each other. His main lesson is methodological: run-to-run variation was so large that single runs could have supported opposite conclusions.

## Key numbers

- 168,000: lines of comments removed from the codebase (765,491 lines total down to 597,332)
- 7.01M to 10.64M: token spread across the three no-comment runs of the bigger task
- 30% / 31%: opposite cost conclusions you could draw by picking single runs
- about 22%: fewer tokens without comments on the small task

## What it reports

- Small task (tile open windows, score out of 19): 18.3 with comments, 19.0 without; tokens 2.18M vs 1.71M; cost $1.42 vs $1.04.
- Bigger task (keep a window on top, score out of 23): 21.7 vs 22.0; 9.06M vs 9.12M tokens; $3.84 vs $3.95.
- The bigger task had a trap (a PHP function dropping unknown session keys); all six runs found it, with or without comments.
- Runs with comments used 8.21M, 8.79M and 10.17M tokens; without, 7.01M, 9.70M and 10.64M. Picking one run per branch could show removal as 30% more expensive or 31% cheaper.
- Conclusion: comments neither help nor hurt the agent; keep them for human readers.
- Teaser: removing the docs/ folder and AGENTS.md did make a difference, reported in a later post.

## How they got it

Two branches, compiled to verify identical code. Same prompt, model and settings; runs alternated between branches. Checks: lint, type checks, existing tests, plus a hidden test added after the agent finished. A separate Claude session scored each change against a pre-written checklist with branch names hidden. Code, diffs, rubrics and full results are linked.

## Limits

- One model, two tasks, three runs per branch: enough to rule out a large effect, not a small one, as the author says.
- The judge is the same model family as the agent.
- One codebase whose comments were mostly agent-written.

## Why it matters for companies

Teams measuring agent productivity or cost from one run per condition are reading noise. Budget for repeated runs before concluding that a prompt, tool or codebase change helps. On the substance, code comments are not where agent context comes from; project-level documentation appears to matter more, so that is where to invest.

Original: https://openstation.blog/2026/10/13/i-deleted-168000-lines-of-comments-my-ai-agent-didnt-notice/