03 · Findings
Results & analysis
Across 33 scanners on identical ground truth, a three-tier ordering emerges and holds under every metric. The within-tier rankings, and several second-order effects, are documented below. The full per-scanner table is on the dashboard.
3.1
A three-tier hierarchy
On the Python leaderboard (66 repositories, where the most scanners have full coverage) the best Security-Specialized scanner reaches F3 86.4 (strict), at recall 0.89. The best General-Purpose LLM (Daybreak Blue, F3 80.0) clears the best rule-based tool (Rowan, F3 23.1) by a wide margin. The ordering — Security-Specialized > General-Purpose LLM > Rule-Based SAST — holds under F1, F2, and F3.
Security-Specialized systems take the top of the table: Kolega DevSec Max V0.1.0 leads at F3 86.4 (strict), 0.89 recall. The best General-Purpose LLM trails by 6.4 F3 points, and the best rule-based tool by 63.3.
The standard/strict mode matters more than the F2/F3 choice for the leaderboard's middle. Strict scoring counts a scanner's unfinished repositories as misses, so models that completed all 66 Python repos rise relative to those that timed out, whose unscored repos count as misses.
Top, F3 strictKolega DevSec Max V0.1.0 — 86.4
Best LLM, F3 strictDaybreak Blue — 80.0
Best rule-basedRowan — 23.1
3.2
Where the advantage originates
The per-CWE breakdown shows LLM-based scanners dominate on classes that require semantic understanding of data flow; rule-based tools remain competitive only on weaknesses that reduce to syntactic patterns.
SQL injection · recall
LLM-based 96%
Rule-based 31%
Insecure deserialization · recall
LLM-based 100%
Rule-based 92%
Syntactic patterns
Rule-based tools stay competitive on weaknesses like hardcoded secrets, but their overall recall remains low even there.
3.3
Cost-efficiency
Cost is the total USD spend for the scored run. There is no clean correlation between price and detection: the cheapest capable model lands within a few points of the most expensive, and the priciest LLM does not lead its tier by much.
DeepSeek V4 Flash reaches F3 56.5 for $0.96 — within four points of the top LLM (GPT-5.5, 60.2) at barely 1.5% of its $66 cost. Minimax M2.7 runs the full benchmark for $1.11. Spend is not a proxy for detection.
3.4
Reliability & the precision–recall trade-off
Reliability
Bigger models do not always win
Models that fail to complete all 140 repositories drop under strict scoring, since unscored repositories count as misses — coverage, not just per-repo accuracy, matters. Several frontier models failed to return results for 15–27% of repositories. Selection should favour reliability-adjusted performance, not peak capability.
The trade-off
Conservatism carries a cost
Grok 4.20 Reasoning posts the highest precision in the benchmark (0.932) but recalls only 26% of vulnerabilities. SonarQube shows the opposite failure mode — reasonable precision (0.611), negligible recall (0.063). The recall-weighted F3 metric rewards coverage over conservatism.
Threats to validity. The target repositories are public, so some may appear in the training data of the evaluated LLMs; memorized vulnerability locations could inflate recall. Two mitigations apply: the 280 false-positive traps test discrimination rather than recall, and the LLM-generated corpus consists of repositories created after most model training cutoffs. The full set of limitations — two-language scope (Python and TypeScript/JavaScript), label subjectivity, LLM non-determinism, and default configurations — is documented in the paper.