Static reasoning can't prove runtime behavior: a benchmark of what AI PR reviewers miss
B · Useful, limited A vendor benchmark by Heal, which sells the runtime-testing alternative to AI code review; the replay method is stated and open-source data is published, but the sample is small, matching uses an LLM judge, and bot findings were not checked for relevance.
Summary based on the full text·Original: about 12 minutes to read
In short
A benchmark page (October 2026) from Heal, a start-up that builds a sandbox around an application, drives it through its entry points the way users do, and files each defect with a failing test. It replayed the pull-request history of five products into fresh repositories with three AI code-review bots (CodeRabbit, Greptile, cubic) installed, then compared findings. Most runtime defects that reach users were missed by every bot; most bot findings were in files that do not affect runtime. Stacking several review bots added little, because they flag the same things.
Key numbers
- 70 %user-facing defects proven by Heal's runtime sandbox that no review bot found (141 of 202)
- 236pull requests replayed across five products with three AI review bots on default settings
- 654 of 1,003distinct review-bot flags that sat in files the runtime tool does not read (CI, migrations, analytics, tests)
- 24 % to 30 %share of Heal-proven defects on Product A found by one review bot alone against all three stacked
As reported by the source.
English·Web page·opens heal.dev
What it reports
- 141 of 202 user-facing defects that Heal proved on the replayed code were not found by any review bot; the share ranged from 58% to 100% by product.
- Many missed defects spanned files, repositories or PRs, such as a backend route rename that broke every event page in the frontend.
- Of 1,003 distinct bot flags, 654 were in files Heal skips by design, 202 were on code Heal tests but did not report (mostly race conditions), 100 were out of scope, and 47 were also found by Heal.
- Adding a second and third bot raised coverage of Heal-proven defects on Product A from 24% to 30%, and not at all on the other four products.
- Heal's runs produced 1,085 regression tests across the five products; 98-99% of bot claims came with a proposed fix.
How they got it
Five products, two private and anonymised, three open source (Formbricks, Sim, Infisical), 236 PRs replayed PR by PR. Heal ran before and after each sequence. A language-model judge, calibrated on a manually curated sample, decided whether a bot claim and a Heal defect named the same problem. An earlier attempt using live one-week windows was dropped as too hard to attribute. Bot comments and Heal defects for the open-source products are downloadable.
Limits
- The vendor designed and ran the benchmark for its own product.
- Bot flags were not checked for relevance, while Heal defects were proven, so the comparison is asymmetric (the vendor notes this may favour the bots on volume).
- Small dataset, stopped after "several thousand dollars" of review spend; default bot settings only.
- Product A contributes most of the defects and its whole early history was replayed.
Why it matters for companies
Teams adding more AI review bots to cope with agent-written code are buying overlap. Static review and runtime testing find different defects: review covers configuration, CI and style; running the product finds bugs that cross files and services. A verification budget for agent-written code should include tests that exercise the running product, not just more readers of the diff.
Summary based on the full text. Read the original for the full argument and data.Read the original on heal.dev