(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection
A · Strong evidence Microsoft Research and CMU authors, including Christian Bird and Emerson Murphy-Hill, build a bug-grounded review benchmark from 98 open-source and 2 proprietary repositories and test frontier models with cost and failure analysis.
Summary based on the full text·Original: about 45 minutes to read
In short
An arXiv paper (9 October 2026) by Chenyang Yang and colleagues at Microsoft and CMU. It asks whether AI can review code well enough for a "lights-off" software factory where agents write and review with little human oversight. Using real bugs traced back to the PRs that introduced them, the best agent now catches about 72% of bugs in its top five comments, up from 48% in February 2026. Agents still miss 20%+ of bugs, do worse on large and complex changes, and drop sharply on proprietary code that needs project context.
Key numbers
- 71.9%bugs caught within the top-5 comments by the best review agent (claude-opus-5 with copilot-cli)
- 47.7%same measure for opus-4.6, five months earlier
- up to 25.5%performance drop on proprietary industry repositories vs open-source ones
- 1,800review tasks in CRBugBench (1,000 open-source, 800 proprietary)
- 5.8%gain from exploration-nudging prompts at 1.10x cost, vs 4.7% for multi-agent at 5.55x cost
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- On open-source PRs, claude-opus-5-xhigh catches 79.6% of bugs humans found before merge and 64.2% of bugs found after merge, within top-5 comments.
- Recall@5 rose from 47.7% (opus-4.6) to 71.9% (opus-5) between February and July 2026.
- On proprietary repositories at one large company, performance drops up to 25.5%, partly because more bugs cannot be inferred from the diff alone.
- Logistic regression: agents miss more in larger, heterogeneous or bug-fixing PRs, on non-functional or less severe bugs, and where non-local reasoning is needed.
- Agents mostly read and search, rarely run code: opus-5 executes code in 9.7% of turns, gpt-5.6-sol in 6.9%. Half of opus-5 runs end by turn 35 vs turn 17 for gpt-5.6-sol. Agents trust misleading PR descriptions and comments.
- Compute scaling works best through prompts: exploration-nudging prompts +5.8% at 1.10x cost; multi-agent +4.7% at 5.55x; higher reasoning effort lifted one model from 36% to 57% at 23.22x cost.
How they got it
A pipeline mines human-verified bugs from addressed review comments and from bug-fix PRs traced to their introducing PR, then gives each agent a sandboxed repository at that commit with future commits removed. Agents write a ranked review; a panel of LLM judges matches findings to the ground-truth bug. 1,000 tasks span 98 GitHub repositories and 5 languages; 800 come from two internal repositories that are not in training data.
Limits
- Only bugs humans later found count as ground truth; valid agent findings outside that set are not credited, and false-positive cost is not the main metric.
- LLM judges decide matches.
- The proprietary portion comes from one company and covers four models.
- Microsoft authors test Microsoft's copilot-cli harness among others.
Why it matters for companies
AI review is now good enough to catch most bugs in typical open-source PRs, but not good enough to remove humans, least of all on internal code where context matters. Firms should scale human review effort with PR size and complexity, give review agents repository knowledge, and route models by PR type to control cost. Prompt changes give more per dollar than multi-agent setups. The authors recommend each firm build its own bug-grounded benchmark from its repositories and re-check as models change.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org