---
id: "lib-2026-10-21-your-ai-agent-fixed-the-bug-but-what-else-did-it"
type: "library"
title: "Your AI agent fixed the bug, but what else did it change?"
lang: "EN"
org: "Trail of Bits"
source_type: "Practitioner post"
date: "2026-10-21"
week: "2026-W43"
url: "https://nosov.ai/library/lib-2026-10-21-your-ai-agent-fixed-the-bug-but-what-else-did-it/"
original_url: "https://blog.trailofbits.com/2026/10/21/your-ai-agent-fixed-the-bug-but-what-else-did-it-change/"
tier: "A"
tier_name: "Strong evidence"
tier_reason: "Trail of Bits publishes its own regression rate on merged agent-assisted patches, a precision figure for its review process and a plugin comparison with a confidence interval, and open-sources the tooling."
summary_basis: "full-text"
original_minutes: "8"
tags: ["coding agents", "regressions", "patch verification", "ai security"]
frames: ["verification-capacity"]
---

# Your AI agent fixed the bug, but what else did it change?

Tier A (Strong evidence): Trail of Bits publishes its own regression rate on merged agent-assisted patches, a precision figure for its review process and a plugin comparison with a confidence interval, and open-sources the tooling.

Summary based on the full text.

## In short

A 21 October 2026 post by Anish Naik and Marcelo Morales of Trail of Bits releasing two open-source agent plugins, patching-bugs and regression-hunt, built from its Patch the Planet work with OpenAI. Reviewing every upstream patch its engineers authored and maintainers merged, Trail of Bits confirmed 21 regressions in over 130 PRs; none broke a security protection. The plugins standardise how agents write patches and how they check for regressions, requiring a live proof of concept for every claim.

## Key numbers

- 21: confirmed regressions in over 130 merged upstream PRs (16 functional, one performance, four build/test)
- ~36%: precision of the agent regression review: 21 confirmed among 58 candidates in one run
- 25%: fewer false-positive findings from regression-hunt than a baseline Codex prompt
- 14% to 38%: 95% confidence interval on that false-positive reduction

## What it reports

- 21 regressions from over 130 merged PRs: 16 functional, one performance, four build/test.
- The review found them among 58 candidates in a single run (about 36% precision); different agent runs produce different candidate sets, adding manual sorting.
- regression-hunt traces the diff to real interfaces, reproduces each candidate on a live deployment, replays the PoC in a fresh-context agent, then an adjudication step must rule out false positives, incomplete fixes, latent and pre-existing bugs.
- Against a baseline Codex agent told to "review the PR for regressions", regression-hunt produced 25% fewer false positives (95% CI 14% to 38%), but left some known regressions as borderline for lack of evidence.
- patching-bugs uses a "three-arm proof": original code fails, patched code passes, full revert fails again. It aims to fix the violated security invariant, not just the instance, and must run regression-hunt.
- Design draws on thousands of Codex sessions by Trail of Bits engineers. Steps exchange strict schemas (claim, revision, check, raw evidence, open questions).
- Results improve with a project THREAT_MODEL.md and PATCHING_GUIDELINES.md (or AGENTS.md); GPAC, liboqs and validator.js have published threat models.

## How they got it

Retrospective review of the firm's own merged patches with agent-generated, PoC-confirmed regression candidates checked by engineers, then a comparison of the plugin and a baseline prompt on the 21 known regressions.

## Limits

- The evaluation set is small (21 regressions) and drawn from the same work that shaped the plugin.
- The false-positive gain trades off against recall: some real regressions were not confirmed.
- No evaluation yet of whether patching-bugs fixes bugs without regressions; that is listed as future work.
- Trail of Bits is a security consultancy with a partnership interest in the programme.

## Why it matters for companies

This is one of few published regression rates for agent-assisted patching: roughly one in six merged patches broke something, though nothing security-critical. Teams letting agents write fixes should require reproduction before and after, a revert check, and a separate review step that must prove a regression, not just suspect one. Writing a threat model and patching guidelines into the repo is cheap and gives agents the project's rules.

Original: https://blog.trailofbits.com/2026/10/21/your-ai-agent-fixed-the-bug-but-what-else-did-it-change/