--- id: "lib-2026-10-07-yandex-ai-reviewer" type: "library" title: "How we launched an AI reviewer on 4,000 PRs a day" title_original: "Как мы запустили ИИ-ревьюера на 4000 PR в день — и какие баги он находит" lang: "RU" org: "Yandex Infrastructure (Habr)" source_type: "Practitioner post" date: "2026-10-07" week: "2026-W41" url: "https://nosov.ai/library/lib-2026-10-07-yandex-ai-reviewer/" original_url: "https://habr.com/ru/companies/yandex/articles/1089752/" tier: "A" tier_name: "Strong evidence" tier_reason: "First-hand production report from the head of Yandex Infrastructure's ML Laboratory with A/B-tested model comparisons, per-check costs and acceptance rates on about 4,000 pull requests a day." summary_basis: "full-text" original_minutes: "8" tags: ["ai technology", "business transformation", "code review agents", "model cost routing"] frames: ["verification-capacity"] --- # How we launched an AI reviewer on 4,000 PRs a day Original title: Как мы запустили ИИ-ревьюера на 4000 PR в день — и какие баги он находит Tier A (Strong evidence): First-hand production report from the head of Yandex Infrastructure's ML Laboratory with A/B-tested model comparisons, per-check costs and acceptance rates on about 4,000 pull requests a day. Summary based on the full text. ## In short A Habr post (7 October 2026) by Viktor Ploshikhin, head of ML Laboratory at Yandex Infrastructure, on NeuroReview, an internal automated reviewer for pull requests. It runs two agents in sequence: a generator that looks for as many candidate defects as possible, and a validator that discards noise. Opus scored highest on the internal metric but cost about 83 times more per check, so Yandex runs a 27B open-weight Qwen model in production. The reviewer catches business-logic and language-level bugs that humans flag less often, and accepts that it cannot do architectural review. ## Key numbers - ~4,000: pull requests a day reviewed by NeuroReview, about 40% of all PRs in Yandex's Arcadia monorepo - $3.42 vs $0.041: cost per check of the Opus configuration versus the Qwen configuration, about 83x - ~30%: share of NeuroReview comments that developers accept in online experiments - 50% to 70%: rise in the share of likes on NeuroReview comments over six months ## What it reports - About 4,000 PRs a day pass through NeuroReview, roughly 40% of PRs created in Arcadia; use is voluntary and internal only. - Generator: Qwen 3.6 27B on internal inference. Validator: MiniMax-M2-7. Baseline was MiniMax-M2-7 in both roles. - Gains on Time Saved per PR (TSPR) relative to baseline: Gemini +37.22%, Qwen +62.77%; Gemini was also more expensive. - Opus gave about 30% higher TSPR than Qwen, at $3.42 per check against $0.041. One check per PR across 4,000 PRs would cost about $13.7k a day before re-checks. - Moving the generator from MiniMax-M2-7 to Qwen cut hardware needs: MiniMax needs a host with 4x more GPUs. The full stream now runs with about 25% spare capacity. - First comment arrives on average 2-3 minutes after a PR is published; human reviewers usually join after several hours. - 7.5% of human comments had a matching NeuroReview comment on the same line (LLM-judge matching). Assuming the 30% acceptance rate holds for unmatched agent comments, they add about 8% more accepted comments relative to manual review. - Accepted NeuroReview comments: about 50% business logic, 16% language bugs. Accepted human comments: 31% business logic, 1.8% language bugs, 35% refactoring, 23% minor. ## How they got it Configurations were compared through A/B tests on live traffic, with each PR deterministically assigned to one group for all its iterations. TSPR adds estimated time saved for accepted suggestions and closed issues and subtracts a penalty for rejected ones; it is a proxy based on developer reactions, not measured working time. Human-versus-agent overlap was measured offline on single-line comments with an LLM judge. ## Limits - Model comparisons cover whole pipelines (model, prompt, tools, validator); component contributions were not isolated, so the Qwen result applies to this setup. - TSPR is a proxy, and the 8% additional-yield figure rests on an assumed acceptance rate. - Weak on very large PRs (hundreds of files) and on defects rooted outside the diff; an LGTM does not certify correctness. ## Why it matters for companies The cost table is the useful part: the most capable model was not worth 83 times the price once a narrow task and a validation step were in place. As more code is written by agents, review becomes the bottleneck, and the post shows a cheap independent check catching bugs in agent-written code too. Firms building review or quality gates can copy the generator-validator split and the habit of measuring acceptance rather than volume. Final responsibility for the code stays with a human reviewer. ## Key passages in translation > Среди протестированных конфигураций вариант с Opus показал самый высокий TSPR — примерно на 30% выше конфигурации с Qwen. При этом одна проверка стоила $3,42 против $0,041 — примерно в 83 раза дороже. Among the configurations tested, the Opus variant showed the highest TSPR, about 30% above the Qwen configuration. But one check cost $3.42 against $0.041, roughly 83 times more. > В такой системе генератору и не нужно быть лучшей универсальной моделью. Ему нужно хорошо решать узкую задачу — находить кандидатов на реальные дефекты. In a system like this the generator does not need to be the best general-purpose model. It needs to handle a narrow task well: finding candidates for real defects. > Чем больше кода пишут агенты, тем нужнее независимая проверка на выходе. The more code agents write, the more an independent check at the output is needed. > Его LGTM также не означает, что корректность кода доказана: агент лишь сообщает, что в текущем контексте не нашёл достаточно обоснованных проблем. Its LGTM also does not mean the code is proven correct: the agent only reports that, in the current context, it found no sufficiently grounded problems. Original: https://habr.com/ru/companies/yandex/articles/1089752/