RealVuln
An open benchmark measuring how Rule-Based SAST, General-Purpose LLM, and Security-Specialized scanners perform on real-world vulnerable code.
Leaderboard
Open full dashboard →33 scanners on identical pinned commits across 140 repositories. Headline metric is F3 (strict): recall weighted nine times over precision, with unfinished repositories counted as misses. Switch between F2 and F3 to re-rank. Metric definitions →
| # | Scanner ▼ | F3 ▼ | Python F3 ▼ | TS/JS F3 ▼ | Missed Vulns ▼ | Cost /100k LoC ▼ | Cost/100 Vulns ▼ |
|---|
Amber repo counts mark runs that did not complete all repositories of the selected corpus; under strict scoring their unscored repos count as misses. The Overall tab ranks only scanners that covered every language in the corpus; a run that covered one language appears on that language's tab instead, so every ranking is like-for-like. Cost is the total USD spend for the scored run; rule-based tools are free or variably priced (—).
Precision vs. recall, selected leaderboard. Security-Specialized systems (violet) reach the high-recall right; General-Purpose LLMs cluster center with strong precision; Rule-Based tools occupy the low-recall left.
breadth-driven
high variance
syntactic only
Tier figures follow the selected leaderboard. On the Python leaderboard (66 repositories, where the most scanners have full coverage) a three-tier ordering — Security-Specialized > General-Purpose LLM > Rule-Based SAST — holds under both F2 and F3, though within-tier rankings shift with the metric and strict/standard mode. Read the analysis →
Documentation
Benchmark design & scoring
Target-type taxonomy, ground-truth labeling, the matching algorithm, and why the primary metric is recall-weighted F3.
→The corpus
140 hand-labeled Python and TypeScript/JavaScript repositories, the ground-truth schema, false-positive traps, and framework coverage.
→Results & analysis
The three-tier hierarchy, per-CWE detection, cost-efficiency, reliability, and the precision–recall trade-off.
→Living benchmark & contributing
Versioning, the v2 roadmap, the authorship research question, how to contribute, and how to cite.