Catching Bugs Before They Land: JiTTesting's Human-on-the-Loop Approach at Full Scale
B · Useful, limited Meta engineering post, co-authored by testing researcher Mark Harman, on a production LLM test-generation system backed by an FSE 2026 paper; the design is well described, but the outcome numbers are self-reported and not isolated.
Summary based on the full text·Original: about 7 minutes to read
In short
A Meta Engineering post (29 October 2026) by Sreeja Thummala, Sophie Zeng, Mark Harman, Weini Liu and Eric Liu. Catching Just-in-Time Tests generate executable tests designed to fail on a code change, exposing behaviour the change introduced, instead of having a model review the code as a human would. An ensemble of LLM judges plus deterministic rules filters false positives, and engineers set criteria and review outcomes in aggregate rather than checking each output. The system now runs as part of Meta's infrastructure.
Key numbers
- 589engineers who engaged with Catching JiTTests, 64% of those contacted
- 60%engineer-reported true-positive rate of the findings
- 74serious privacy bugs caught at the diff stage in the first half of 2026
- 20%fall in privacy SEVs over the same period, to which the system contributed
As reported by the source.
English·Web page·opens engineering.fb.com
What it reports
- Runs overnight on spare capacity against the previous day's diffs flagged as risky by a model similar to Meta's Diff Risk Score.
- Two generation workflows: "dodgy diff" treats every change as a possible bug and generates tests that distinguish it from its parent; "intent-aware" infers what the change is meant to do, lists ways that could go wrong, builds mutants for each risk and generates tests to kill them.
- Failing tests ("weak catches") are ranked by an LLM-as-Judge ensemble and RubFake, a rule-based false-positive filter with deterministic verdicts.
- Engineers are contacted only for highly ranked catches; they confirm whether a behaviour change was intended and rarely read test code.
- 589 engineers engaged (64% of outreach) and report a 60% true-positive rate; 74 serious privacy bugs caught before landing in the first half of 2026, while privacy SEVs fell 20% in that period.
How they got it
Production deployment over roughly five months since the approach was first published, with engagement and outcome figures from Meta's own tracking. The detailed method is in the FSE 2026 paper "Just-in-Time Catching Test Generation at Meta", which this post summarises.
Limits
- The true-positive rate is engineer-reported.
- The 20% fall in privacy SEVs is not attributed to the system alone; the post says it "played a key role".
- No compute cost, model names or false-positive volumes are given.
- Results come from Meta's code base and tooling, which few firms share.
Why it matters for companies
As coding agents raise the volume of code changes, review capacity becomes the bottleneck. This is a documented alternative to LLM code review: generate tests that show a concrete behavioural difference, filter with a mix of model judges and fixed rules, and only then ask a person. The human-on-the-loop rule (people set criteria and audit aggregates, not every output) is a governance pattern that transfers beyond testing.
Summary based on the full text. Read the original for the full argument and data.Read the original on engineering.fb.com