How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0
B · Useful, limited Snyk researchers publish a carefully built benchmark with attacker-reachable ground truth, strict scoring and all run data public; the method is strong, but the authors sell one of the two engines compared, and 108 vulnerabilities cannot separate the leaders.
Summary based on the full text·Original: about 60 minutes to read
In short
A preprint (arXiv, 28 October 2026) by Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth and Brian Clark of Snyk. VulnBench 2.0 tests nine packaged security-scanning setups, from a classic static analyser to AI agents, on 108 hand-verified, attacker-reachable vulnerabilities in 20 open-source web apps. The static analyser and the best AI review find about the same number, but different ones; together they find almost everything any setup found. What an AI tool misses depends on its built-in review policy as much as on its model.
Key numbers
- 82.4% vs 79.3%recall of Snyk Code (89 of 108) versus Claude Code /security-review with Opus 5 at xhigh effort (85 of 108)
- 101 of 108vulnerabilities found by the two together, the ceiling of all seven comparable configurations combined
- 20.9% vs 74.6%precision against the reference set: Snyk Code (89 of 425 findings) versus Opus 5 xhigh (85 of 114)
- 9 vs 16 of 20vulnerabilities matched in three classes Claude Code's review policy restricts, with versus without that policy
- 80 vs 34vulnerabilities Codex Security with GPT-5.6 Sol located at file level versus at line level
As reported by the source.
English·Web page·opens arxiv.org
What it reports
- Snyk Code 82.4% recall, Claude Code /security-review with Opus 5 xhigh 79.3%. They disagree on 28 vulnerabilities, 16 to 12 (McNemar p = 0.57), too few to call a winner.
- The static analyser covered vulnerabilities no AI configuration found; AI configurations found cross-file and logic flaws it missed. Their union, 101 of 108, equals all seven comparable configurations combined; the best pair of AI configurations reached 93.
- AI configurations also reported at least seven real vulnerabilities absent from the reference set.
- Triage cost: together the two report 539 findings for 101 matches; 336 of Snyk Code's 425 findings matched nothing.
- Claude Code's review policy excludes or restricts three vulnerability classes; without it, the same model matched 16 of 20 in those classes instead of 9, at 0.63 times the cost.
- Codex Security with GPT-5.6 Sol cost 35.0 times as much as GPT-5.6 Luna; DeepSec's file selection kept a file from only 56 of 108 vulnerable flows. Coverage and cost were only loosely related.
How they got it
Twenty apps in Go, JavaScript/TypeScript, Python and Java; 62 of the vulnerabilities span more than one file. Each was curated as reachable from outside with an annotated source-to-sink flow and an exploit proof of concept verified by the authors, independent of any tool's output. A finding counts only if it locates both the source and the sink within two lines. Nine productised configurations were run under the same blind protocol; harness, fixtures and run data are public.
Limits
- Authors work for Snyk, whose product is one of the two leaders; the MCP-assisted configuration also uses Snyk.
- 108 vulnerabilities in 20 apps; small differences are not statistically meaningful.
- Precision is a lower bound because the reference set may omit real flaws.
- The policy-free Claude Code run is a single supplementary run on 19 apps.
Why it matters for companies
Security teams deciding whether AI review can replace static analysis get a clear answer from this data: in this test, neither alone was enough, and the combination covered nearly everything. Before relying on an AI security agent, check what its harness policy suppresses, because a built-in precision-first policy can hide whole vulnerability classes. Budget for triage too: the static tool was noisier, the AI tools more precise, and spending more on a larger model did not reliably buy coverage.
Summary based on the full text. Read the original for the full argument and data.Read the original on arxiv.org