Why this matters
A benchmark that shifts under its users is not a benchmark; it is a website with opinions. Until now, RealVuln was closer to the latter. This entry marks the point where our vulnerability scanner benchmark became something you can pin, cite, and re-run: a version number, a manifest hash, and a permanent URL for every number that version stands on.
What we did
On June 11 we froze release 1.0.0. The versions.json manifest records it with a release date of 2026-06-11: 26 repositories, 25 scanners, and 796 labeled findings. The git tag v1.0 points at the merge from 2026-06-12, which is the detail we care about most — it means the tag and the manifest agree on exactly which code produced which numbers, and the release is served from a permanent address, v1.0.0, rather than from whatever the homepage happens to say today.
The mechanism is deliberately simple. Each version is frozen by a manifest hash, so old results stay valid and can be compared side by side with newer ones. 1.0.0 preserves the pre-v2, 26-repository corpus as a citable snapshot: the F2 and F3 headline metrics in standard and strict modes, per-CWE and per-severity breakdowns, and the public dashboard. In the same window we added Claude Fable 5, our Claude Code harness, to the scanner roster (June 10–11). Around it, we reskinned the site into an editorial, multi-page design wired to the live data — those commits landed between June 3 and June 8 — and on June 11 we added robots.txt, a generated sitemap.xml, and Google Analytics (GA4). On June 12 we put vendor and website links on the leaderboard, so a reader can move from a result to the scanner being measured without guessing which tool a row refers to.
What we observed
The most interesting observation this cycle was not a scanner finding; it was how freezing a version changed the conversation. Numbers that used to float — "as of last week's run" — now have a fixed referent. We can say "in 1.0.0, scanner X scored Y" and mean something stable even after the corpus grows again.
We also noticed, while wiring the dashboard to live data, how much of the story lives in the tails rather than the headline. F3 (beta = 3, recall weighted nine times over precision) and F2 are the summary, but the per-CWE and per-severity breakdowns are where a security scanner benchmark earns its keep: rule-based static analysis tools, general-purpose LLMs, and security-specialized agents do not fail the same way, and a single number flattens those differences.
Why we think that happened
Version pinning matters for a reason specific to accuracy measurement. Because we report false positives explicitly — through false-positive traps rather than by assuming a tool is right — every number is a claim against a particular snapshot of code. If the snapshot moves, the claim becomes unreproducible, and reviewers end up arguing about which version of reality a paper was describing. Freezing the manifest hash is the cheapest way we know to make a claim about scanner accuracy checkable, which is the point of a benchmark like ours.
We are less sure about the site work. The editorial reskin, sitemap, and GA4 are about accessibility and citation, not measurement, and it is too early to say whether they help or just add moving parts. We are treating them as an experiment.
One thing we are comfortable stating plainly: we still decline to name a best vulnerability scanner, because the honest answer depends on how a reader weights missed findings against false alarms. Different SAST tools and agents optimize for different parts of that trade-off, and our job is to make the trade-off visible, not to pick a winner.
What's next
The next release on the roadmap is the v2 corpus expansion, which is exactly why 1.0.0 exists as a frozen point: when the repository set grows, the 26-repo snapshot stays citable and comparable. For now, the work is to watch how 1.0.0 is used and to fix whatever turns out to be ambiguous once other people start poking at it.