What we saw after removing code review
コードレビューをなくして見えたもの
A · Strong evidence Engineer henteko reports seven months without human code review on a small Rails team, with defined before/after windows, PR filters and an honest count of where the replacement human check failed.
Summary based on the full text·Original: about 12 minutes to read
In short
A Zenn post (21 October 2026) by henteko, following up an essay from six months earlier that proposed replacing human code review with a team check of the "concept" given to the AI before implementation. The team built the workflow as a tool called Ito and ran it from March to October 2026. Throughput held and waiting time fell, with no rise in hotfixes or reverts. The human check that was meant to replace code review mostly did not work: the AI wrote the concepts, and reviewers approved them. The author redesigns the flow so that people record only the decisions that need a person.
Key numbers
- 97% to under 10%share of merged PRs with a human approval, before versus after April 2026
- 19.2 h to 16.1 hmedian time to merge, before versus after
- 0 of 40full-level concepts sent back in mandatory pre-implementation review
- 84%share of 68 reviewer comments on concepts that were plain approvals such as 'LGTM!'
- 2 minmedian time from concept creation to start of implementation
As reported by the source.
Japanese·Web page·opens zenn.dev
What it reports
- Concepts are GitHub issues with Why, Outcome, Boundaries and AI Findings, sized minor, quick or full. After implementation an AI writes a Gap Report comparing concept and code.
- 277 concepts in the period: 205 quick, 40 full, 32 minor. Gap Reports exist for 254 (92%).
- PRs with human approval fell from 97% to under 10% from April; exceptions were hotfixes and deliberate requests.
- Weekly merges 5.1 to 5.5; median merge time 19.2 h to 16.1 h; slowest quartile 2.8 days to 23.0 hours; share merged within 24 h 65% to 77%; share over 3 days 22% to 14%; median for PRs over 300 lines 2.8 days to 21.3 hours.
- The team shrank from five to four people; merges per person per week rose from 1.03 to 1.38 (34%).
- What failed: AI-generated research made up 64% of concept text; median 2 minutes from concept to implementation; 37% of Gap Reports were written within 30 minutes of the concept.
- Of 40 full concepts, none were sent back; 57 of 68 comments (84%) were bare approvals. The few useful comments raised context invisible in code, such as how users actually reach a feature.
- Post-implementation sharing failed: 145 of 245 quick and full concepts (59%) remain unreviewed.
- Next design: the AI drafts, then asks the person 0-4 open decisions and records the answers verbatim in a field the AI may not fill; level depends on whether such a decision exists; Gap Reports list only new decisions the AI made during implementation. AI-initiated improvement loops get one human-approved loop concept with metrics, guardrails and an expiry date.
How they got it
Repository data for human-created PRs merged to main, excluding bots, comparing 10 September 2025 to 9 March 2026 against 10 March to 8 October 2026; merge is close to release in this repository. Concept statistics cover all 277 Ito issues. Times are wall-clock, not business hours.
Limits
- Small team (five, then four) on one Rails product; the author notes the product made dropping review easier than it would be elsewhere.
- Defect evidence is limited to hotfix and revert counts; no measure of latent quality or incidents.
- The proposed redesign is untested.
Why it matters for companies
This is one of the few reports that measures whether a human control placed upstream of AI-written code actually does anything. It did not, because every field could be filled by the AI and reviewers had nothing of their own to add. Teams replacing code review with plan or prompt review should require inputs only the requester can supply, and set expiry dates on any autonomous loop. Without that, the author warns, human involvement drifts toward zero.
Key passages in translation
Japanese original
コードレビューの代わりに置いたはずの「人間による意図の確認」はほとんど機能しなかった
The "human check of intent" that we put in place of code review barely worked.
Japanese original
ワークフロー内のどこにも、本人しか持っていない情報を要求する箇所がなかった
Nowhere in the workflow did it ask for information that only the person themselves had.
Japanese original
何もしないと人間による関与はゼロに収束していくと思います。
If you do nothing, I think human involvement converges to zero.
Summary based on the full text. Read the original for the full argument and data.Read the original on zenn.dev
- Tags
- Coding agents