Do LLM-based scanners actually find more real vulnerabilities than traditional SAST tools? We built the benchmark to answer it, and the answer is yes — by roughly 3× on recall-weighted F2. Every label, scanner output and scoring script is public so the result can be checked rather than taken on trust.
The paper describes v1.0 (26 repositories, 796 findings). The benchmark has since grown; the figures below are the current release and are generated from the same ground truth the scoring uses.
Static application security testing (SAST) tools remain essential for detecting vulnerabilities early in the development lifecycle, yet traditional analyzers suffer from high false-positive rates and shallow semantic understanding. A new wave of LLM-based security scanners promises deeper reasoning, but no open, reproducible benchmark exists to measure their real-world effectiveness.
We introduce RealVuln, the first fully open-source vulnerability-detection benchmark built from 26 real-world Python repositories containing 796 hand-labeled findings and 120 false-positive traps. We evaluate 15 scanners — 3 traditional SAST tools, 11 LLM-based scanners, and 1 hybrid — using the recall-weighted F2 score as the primary metric.
Our results show that the best LLM scanner (Claude Sonnet 4.6, F2 = 53.8) outperforms the best traditional SAST tool (Snyk, F2 = 18.2) by nearly 3×, with LLM-based tools achieving 2–4× higher F2 overall. Agentic multi-turn scanning adds approximately 46% F2 over single-turn inference for only ~6% additional cost.
All code, ground-truth data, scanner outputs, and scoring scripts are released under an open-source license. RealVuln is designed as a living benchmark — versioned, community-driven, and with a roadmap toward multi-language coverage.
On this benchmark, substantially — the best general-purpose LLM scanner scored roughly 3× the best rule-based tool on recall-weighted F2. The gap is largest on weaknesses that require following data flow, such as insecure deserialization and code injection, and narrowest on syntactic patterns. Current per-class figures are on the findings page.
A missed vulnerability is more costly than a false positive a reviewer discards, so the metric weights recall above precision. F2 weights recall four times precision; F3 nine times. Both are reported, alongside precision and raw counts. The definitions are on the methodology page.
On three fields: file path, CWE (matched against a list of acceptable CWEs, not just one), and line number within a ±10-line tolerance. Entries marked is_vulnerable: false are false-positive traps — flagging one is penalised, not rewarded.
That is the point of the design. Every repository is pinned by commit SHA, every scanner's raw output is committed, and the scoring scripts are public. Frozen snapshots of each release stay online, so a figure cited from an earlier version keeps resolving to the numbers that produced it.
The paper describes v1.0. Cite the frozen dataset revision alongside it so the corpus is unambiguous.
@misc{realvuln2026,
title = {RealVuln: An Open Benchmark for Evaluating Security
Scanners on Real-World Code},
author = {Raza, Faizan and Pellew, John},
year = {2026},
eprint = {2604.13764},
archivePrefix= {arXiv},
url = {https://realvuln.com/v/1.0.0/}
}
Dataset: load_dataset("Kolega-Dev/RealVuln", revision="v1.0.0") for the corpus in the paper, or Kolega-Dev/RealVuln-v2 for the current one. Source and ground truth: github.com/kolega-ai/Real-Vuln-Benchmark.