How we launched an AI reviewer on 4,000 PRs a day
Как мы запустили ИИ-ревьюера на 4000 PR в день — и какие баги он находит
A · Strong evidence First-hand production report from the head of Yandex Infrastructure's ML Laboratory with A/B-tested model comparisons, per-check costs and acceptance rates on about 4,000 pull requests a day.
Summary based on the full text·Original: about 8 minutes to read
In short
A Habr post (7 October 2026) by Viktor Ploshikhin, head of ML Laboratory at Yandex Infrastructure, on NeuroReview, an internal automated reviewer for pull requests. It runs two agents in sequence: a generator that looks for as many candidate defects as possible, and a validator that discards noise. Opus scored highest on the internal metric but cost about 83 times more per check, so Yandex runs a 27B open-weight Qwen model in production. The reviewer catches business-logic and language-level bugs that humans flag less often, and accepts that it cannot do architectural review.
Key numbers
- ~4,000pull requests a day reviewed by NeuroReview, about 40% of all PRs in Yandex's Arcadia monorepo
- $3.42 vs $0.041cost per check of the Opus configuration versus the Qwen configuration, about 83x
- ~30%share of NeuroReview comments that developers accept in online experiments
- 50% to 70%rise in the share of likes on NeuroReview comments over six months
As reported by the source.
Russian·Web page·opens habr.com
What it reports
- About 4,000 PRs a day pass through NeuroReview, roughly 40% of PRs created in Arcadia; use is voluntary and internal only.
- Generator: Qwen 3.6 27B on internal inference. Validator: MiniMax-M2-7. Baseline was MiniMax-M2-7 in both roles.
- Gains on Time Saved per PR (TSPR) relative to baseline: Gemini +37.22%, Qwen +62.77%; Gemini was also more expensive.
- Opus gave about 30% higher TSPR than Qwen, at $3.42 per check against $0.041. One check per PR across 4,000 PRs would cost about $13.7k a day before re-checks.
- Moving the generator from MiniMax-M2-7 to Qwen cut hardware needs: MiniMax needs a host with 4x more GPUs. The full stream now runs with about 25% spare capacity.
- First comment arrives on average 2-3 minutes after a PR is published; human reviewers usually join after several hours.
- 7.5% of human comments had a matching NeuroReview comment on the same line (LLM-judge matching). Assuming the 30% acceptance rate holds for unmatched agent comments, they add about 8% more accepted comments relative to manual review.
- Accepted NeuroReview comments: about 50% business logic, 16% language bugs. Accepted human comments: 31% business logic, 1.8% language bugs, 35% refactoring, 23% minor.
How they got it
Configurations were compared through A/B tests on live traffic, with each PR deterministically assigned to one group for all its iterations. TSPR adds estimated time saved for accepted suggestions and closed issues and subtracts a penalty for rejected ones; it is a proxy based on developer reactions, not measured working time. Human-versus-agent overlap was measured offline on single-line comments with an LLM judge.
Limits
- Model comparisons cover whole pipelines (model, prompt, tools, validator); component contributions were not isolated, so the Qwen result applies to this setup.
- TSPR is a proxy, and the 8% additional-yield figure rests on an assumed acceptance rate.
- Weak on very large PRs (hundreds of files) and on defects rooted outside the diff; an LGTM does not certify correctness.
Why it matters for companies
The cost table is the useful part: the most capable model was not worth 83 times the price once a narrow task and a validation step were in place. As more code is written by agents, review becomes the bottleneck, and the post shows a cheap independent check catching bugs in agent-written code too. Firms building review or quality gates can copy the generator-validator split and the habit of measuring acceptance rather than volume. Final responsibility for the code stays with a human reviewer.
Key passages in translation
Russian original
Среди протестированных конфигураций вариант с Opus показал самый высокий TSPR — примерно на 30% выше конфигурации с Qwen. При этом одна проверка стоила $3,42 против $0,041 — примерно в 83 раза дороже.
Among the configurations tested, the Opus variant showed the highest TSPR, about 30% above the Qwen configuration. But one check cost $3.42 against $0.041, roughly 83 times more.
Russian original
В такой системе генератору и не нужно быть лучшей универсальной моделью. Ему нужно хорошо решать узкую задачу — находить кандидатов на реальные дефекты.
In a system like this the generator does not need to be the best general-purpose model. It needs to handle a narrow task well: finding candidates for real defects.
Russian original
Чем больше кода пишут агенты, тем нужнее независимая проверка на выходе.
The more code agents write, the more an independent check at the output is needed.
Russian original
Его LGTM также не означает, что корректность кода доказана: агент лишь сообщает, что в текущем контексте не нашёл достаточно обоснованных проблем.
Its LGTM also does not mean the code is proven correct: the agent only reports that, in the current context, it found no sufficiently grounded problems.
Summary based on the full text. Read the original for the full argument and data.Read the original on habr.com