Why this matters
A vulnerability scanner benchmark is only as trustworthy as the labels it ships, and labels that are hand-copied between systems are labels that will eventually disagree. This entry closes the journal's first arc by freezing the benchmark at version 2.1.0 and describing what the current release actually contains.
What we did
On 2026-08-24 we cut RealVuln 2.1.0 — the current release in versions.json — at 66 repositories, 30 scanners, and 2,176 labeled findings: 1,896 vulnerable findings plus 280 false-positive traps. The change that matters most is mundane to describe but fundamental to the project: the Hugging Face dataset (Kolega-Dev/RealVuln) is now generated from the ground truth rather than hand-assembled. The commit that did it ("feat: generate the HuggingFace dataset instead of hand-assembling it", 2026-08-24) flattens and maps the ground-truth export fields, so the published dataset and the source of truth can no longer drift apart quietly.
The same window held a ground-truth correction that reclassified damn-vulnerable-flask-app-003 as a false-positive trap (2026-08-24).
What we observed
The manifest now reads as follows: 66 repos, 1,896 vulnerable findings and 280 false-positive traps (2,176 total), with 26 human-authored and 40 LLM-generated repositories across Django, FastAPI, Flask, aiohttp, and Tornado. We score rule-based SAST tools, general-purpose LLMs, and security-specialized agents on the same labels, reporting F3 (beta=3, recall weighted nine-to-one over precision) alongside F2 in standard and strict modes.
Three things stand out. First, the traps are about one finding in eight — a small fraction of the total, but disproportionate in what they reveal about a scanner's precision. Second, the dataset export now has a property it lacked before: it is a function of the ground truth, so a correction like damn-vulnerable-flask-app-003 propagates automatically instead of waiting for a manual re-export. Third, our own label changes are visible in the record, which we think is the honest way to run a security scanner benchmark — labels should change as we learn, and the changelog should show it.
Why we think that happened
The generated export is, candidly, a response to a failure mode we kept flirting with: every hand-assembled dataset release was another opportunity to ship stale or mismatched labels. Generating from the source of truth removes that class of error, and the field flattening was the prerequisite — our internal ground truth nests fields in ways that do not map cleanly onto a flat, single-table dataset.
The trap reclassification is worth being direct about. It means an earlier version of our labels did not survive closer scrutiny, and anyone who ran against the older export was benchmarking against a label we have since changed. We would rather surface that correction than bury it; if a benchmark's own history is not inspectable, its numbers are not reproducible.
What's next
Freezing 2.1.0 gives us a stable artifact to point at while we keep scoring new scanners against the same labels. The question we are most interested in next is coverage — 66 repositories are enough to expose false positives, but we do not yet know how well they bound recall across the frameworks we claim to cover.