Your AI agent fixed the bug, but what else did it change?
A · Strong evidence Trail of Bits publishes its own regression rate on merged agent-assisted patches, a precision figure for its review process and a plugin comparison with a confidence interval, and open-sources the tooling.
Summary based on the full text·Original: about 8 minutes to read
In short
A 21 October 2026 post by Anish Naik and Marcelo Morales of Trail of Bits releasing two open-source agent plugins, patching-bugs and regression-hunt, built from its Patch the Planet work with OpenAI. Reviewing every upstream patch its engineers authored and maintainers merged, Trail of Bits confirmed 21 regressions in over 130 PRs; none broke a security protection. The plugins standardise how agents write patches and how they check for regressions, requiring a live proof of concept for every claim.
Key numbers
- 21confirmed regressions in over 130 merged upstream PRs (16 functional, one performance, four build/test)
- ~36%precision of the agent regression review: 21 confirmed among 58 candidates in one run
- 25%fewer false-positive findings from regression-hunt than a baseline Codex prompt
- 14% to 38%95% confidence interval on that false-positive reduction
As reported by the source.
English·Web page·opens blog.trailofbits.com
What it reports
- 21 regressions from over 130 merged PRs: 16 functional, one performance, four build/test.
- The review found them among 58 candidates in a single run (about 36% precision); different agent runs produce different candidate sets, adding manual sorting.
- regression-hunt traces the diff to real interfaces, reproduces each candidate on a live deployment, replays the PoC in a fresh-context agent, then an adjudication step must rule out false positives, incomplete fixes, latent and pre-existing bugs.
- Against a baseline Codex agent told to "review the PR for regressions", regression-hunt produced 25% fewer false positives (95% CI 14% to 38%), but left some known regressions as borderline for lack of evidence.
- patching-bugs uses a "three-arm proof": original code fails, patched code passes, full revert fails again. It aims to fix the violated security invariant, not just the instance, and must run regression-hunt.
- Design draws on thousands of Codex sessions by Trail of Bits engineers. Steps exchange strict schemas (claim, revision, check, raw evidence, open questions).
- Results improve with a project THREAT_MODEL.md and PATCHING_GUIDELINES.md (or AGENTS.md); GPAC, liboqs and validator.js have published threat models.
How they got it
Retrospective review of the firm's own merged patches with agent-generated, PoC-confirmed regression candidates checked by engineers, then a comparison of the plugin and a baseline prompt on the 21 known regressions.
Limits
- The evaluation set is small (21 regressions) and drawn from the same work that shaped the plugin.
- The false-positive gain trades off against recall: some real regressions were not confirmed.
- No evaluation yet of whether patching-bugs fixes bugs without regressions; that is listed as future work.
- Trail of Bits is a security consultancy with a partnership interest in the programme.
Why it matters for companies
This is one of few published regression rates for agent-assisted patching: roughly one in six merged patches broke something, though nothing security-critical. Teams letting agents write fixes should require reproduction before and after, a revert check, and a separate review step that must prove a regression, not just suspect one. Writing a threat model and patching guidelines into the repo is cheap and gives agents the project's rules.
Summary based on the full text. Read the original for the full argument and data.Read the original on blog.trailofbits.com
- Tags
- Coding agents, AI security