Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines
A · Strong evidence Samuel Groß of Google Project Zero compares two production bug-finding systems on V8 using public bug-tracker queries, a rediscovery experiment with 25 attempts per bug, a memorisation check and worked case studies; one target and no cost data, but the method is laid out step by step.
Summary based on the full text·Original: about 25 minutes to read
In short
A Google Project Zero post (27 October 2026) by Samuel Groß comparing Fuzzilli, a JavaScript-engine fuzzer, with BigSleep, an LLM-based agent that reviews new commits and writes crashing test cases, on Chrome's V8 engine. Over six months BigSleep found slightly more actionable vulnerabilities, with about a quarter overlap in production. In a controlled rediscovery test the overlap was nearer 60%. Each system finds bugs the other misses, so the author recommends running both.
Key numbers
- 45 vs. 37actionable V8 vulnerabilities found by BigSleep and by Fuzzilli, April to September 2026
- 20 out of a total of 81actionable vulnerabilities found by both systems in production, about 25% overlap
- ~62%share of 112 historic Fuzzilli bugs that BigSleep rediscovered from the introducing commit
- ~70%rediscovery share after extra runs with different starting points and prompts
- 441 vs. 316raw bug reports filed by BigSleep and Fuzzilli over the six months
As reported by the source.
English·Web page·opens projectzero.google
What it reports
- April to September 2026: 441 raw reports from BigSleep, 316 from Fuzzilli. Restricted to vulnerabilities: 95 and 178. Restricted to actionable vulnerabilities (not duplicates or working-as-intended, with at least one fix commit): 45 and 37.
- Production overlap after several layers of deduplication: 20 of 81 actionable vulnerabilities, a slight majority found first by Fuzzilli.
- Experiment: from about 1,150 historic Fuzzilli bugs (early 2023 to mid-2026), about 600 vulnerabilities, 132 reproducible in a standard build, 112 after deduplication. Given only the likely introducing commit and 25 attempts per bug, BigSleep (Gemini 3.6 flash) rediscovered about 62%, found a different real bug for about 12% and nothing for about 26%.
- Extra runs from the vulnerable file or function, or with a prompt that also accepts assertion failures, raised rediscovery to about 70% and some bug found to about 84%.
- No statistically significant memorisation effect was found for bugs before the model's effective knowledge cutoff.
- Bugs that took Fuzzilli longer to find were also harder for the agent. Overlap was lower for JIT compiler bugs.
- Agent strengths: bugs needing step-by-step reasoning, specific edge-case values, newer language features or non-standard flags. Fuzzer strengths: bugs arising from interactions between components that are each correct, C++ compiler miscompilation, and not being distracted by shallow bugs.
How they got it
Public Chromium bug-tracker queries on each system's hotlist, with deduplication through linked fix commits and the Clusterfuzz database. The rediscovery experiment used the production BigSleep configuration on historic bugs, checking that each test case crashes the pre-fix build and not the post-fix one. Case studies walk through three bugs found by only one system.
Limits
- No cost per bug: neither system is tuned for it, and the author declines to compare.
- The BigSleep model was upgraded several times during the six months.
- One target (V8) and one fuzzer; the experiment covers only bugs reproducible without special flags and runs in one direction (agent rediscovering fuzzer bugs).
- Pinpointing the introducing commit by bisection may be wrong, which the author notes would handicap the agent.
Why it matters for companies
Security teams are asking whether LLM code review can replace fuzzing budgets. On one of the most heavily tested code bases available, the answer from Project Zero is no: the agent matches the fuzzer on volume but the two find different bugs, and the hardest bugs for one tend to be hard for the other. For vendors shipping C or C++ code, the practical step is to add agentic review on each commit while keeping fuzzers running, and to watch for the agent stopping at easy bugs. The absence of cost figures leaves the budget question open.
Summary based on the full text. Read the original for the full argument and data.Read the original on projectzero.google