Parsimony: which AI coding agent changes the least code?
B · Useful, limited An independent benchmark site that re-scores published SWE-bench patches for code size with a documented method; a useful new metric, but Python only, tests not rerun, and the leaderboard values were not in the saved copy.
Summary based on the full text·Original: about 3 minutes to read
In short
Parsimony, a site by lasu.dev, ranks AI coding models by how much code they add to fix a task. It takes patches that benchmarks already publish, parses both versions of each changed file, and counts net "coding units" (calls, names, literals) rather than lines. The author's point is that solve rate and cost ignore excess code, which makes codebases harder to maintain. The page states its own limits clearly, including that models writing more code also tend to solve more tasks.
Key numbers
- 0.33-0.60Spearman correlation between code added and tasks solved, on boards with 26+ models
- 0.88xcode the author's own GPT-5.6 Sol run added compared with its published run
As reported by the source.
English·Web page·opens parsimony.lasu.dev
What it reports
- Models are ranked by mean net units added on tasks they solved, lowest first; a second column includes failed attempts.
- Solve results come from SWE-bench Verified; tests are not rerun. All ranked models use the same agent (mini-SWE-agent).
- More code correlates with more solved tasks: Spearman 0.33-0.60 on boards with 26 or more models.
- A model whose fixes mostly delete code can rank first with a negative net; failed attempts that change nothing pull the mean towards zero.
- The author also ran some models in their own harnesses (pi, Codex, Claude Code) on DeepSWE Python tasks; these are shown faded and not ranked. One model's own run added 0.88x the code of its published run.
- Users can upload their own benchmark results as JSON to get the same chart.
How they got it
Static analysis of published patches: apply each patch, parse both versions of every changed file, exclude tests, docs and generated files, and count added minus removed syntax units. Nothing is executed.
Limits
- Python only.
- Size is not quality: units do not measure readability or design, as the page says.
- The saved copy did not contain the leaderboard values, so model rankings are not reported here.
Why it matters for companies
Agent-written code adds maintenance cost that standard benchmarks do not show. Engineering teams can track net code added per merged change, alongside pass rates, as a cheap signal of bloat. The positive correlation with solve rate is a caution against simply rewarding smaller diffs: the goal is the smallest change that works.
Summary based on the full text. Read the original for the full argument and data.Read the original on parsimony.lasu.dev