Why this matters
A benchmark that only counts how many known vulnerabilities a scanner finds is measuring half of the accuracy problem. The other half — how many warnings a tool raises that aren't vulnerabilities at all — is where real-world deployments burn out, and it's the half almost everyone ignores. Our motivating numbers have not improved with repetition: a widely cited analysis put the false-positive rate for command-injection findings in Python/Flask at 99.5%, and NIST's SATE program found only 8–30% of tool warnings were security-relevant. If we want a security scanner benchmark that means anything, false positives have to be part of the score.
What we did
Release 2.0.0 (release_date 2026-06-29 in versions.json; the "Release RealVuln Benchmark v2.0" commit merged 2026-07-04) freezes the full v2 corpus we described earlier in this journal: 66 Python repositories — 26 human-authored and 40 LLM-generated — carrying 1,443 vulnerable findings and 280 false-positive traps. The release also ships the Apache-2.0 license text, a CONTRIBUTING.md, and a manifest that pins every repository commit and hashes the ground truth, so any scanner's score can be reproduced against the exact code it was run on.
The important new idea is the trap. A false-positive trap is a code pattern that looks suspicious to static analysis but is demonstrably safe — the canonical example being a login function that passes user input to an ORM's filter_by(), which auto-parameterizes the query. A scanner that flags one of these patterns is penalized as a false positive, exactly as if it had flagged a real bug that wasn't there. To our knowledge, no other open SAST benchmark does this: SAST false positives become a scored, first-class quantity rather than an anecdote.
Two housekeeping notes. The scanners count jumped to 39 as the field expanded, now spanning rule-based SAST, general-purpose LLMs, and security-specialized agents. And because the corpus itself changed, we report scores against 1.x and 2.x separately — a number from the old corpus doesn't compare to a number from the new one.
What we observed
The traps were, in a way, harder to build than the vulnerable findings. A vulnerable finding only has to be exploitable; a trap has to be safe in a way an independent reviewer can verify, because a single trap that isn't actually safe would quietly corrupt the precision number for every scanner in the benchmark. That's the reason the manifest pins commits and hashes the ground truth — so the safety of each trap can be disputed in public, with evidence, rather than trusted on our word.
The jump to 39 scanners was itself a finding. It happened mostly in the LLM vulnerability detection categories — general-purpose models and security-specialized agents — which grew fast enough between the 1.x and 2.x runs that keeping two separate scoreboards stopped being a nicety and became a necessity.
Why we think that happened
On the scanner count, our explanation is unglamorous: the field genuinely expanded. Rule-based SAST vendors added rules, LLM-based tools shipped new models, and agent-style scanners became a recognizable category of their own. We don't have a deeper theory, and we won't pretend to.
On why traps matter, we have a hypothesis we're willing to state but not yet defend as fact. We suspect that patterns designed to look dangerous to a static-analysis rule set are exactly where recall-focused tools overreach — a rule that fires on filter_by(user_input) can't always distinguish auto-parameterized input from raw string concatenation. If that's right, the traps measure something real about how SAST rules trade recall against precision. It's a hypothesis, not a conclusion, and the 2.0.0 data will either support it or not.
What's next
We'll publish scores for the 39 scanners against the frozen 2.0.0 corpus — reported separately from all 1.x numbers — and invite corrections to the traps and findings themselves, each of which is now a public, pinned commit.