What 3.0.0 is
RealVuln 3.0.0, cut on 2026-09-02 and frozen at realvuln.com/v/3.0.0/, is the first release of the benchmark to cover more than one language. It doubles the corpus to 140 repositories by adding 74 TypeScript/JavaScript applications to the unchanged 66-repository Python corpus, bringing the dataset to 4,138 labeled vulnerabilities (2,236 of them new), 280 false-positive traps, and 741,034 lines of code. Every figure in it can be regenerated from the repository with make dashboard.
This release also changed how we label and how we rank. Each change is a story of its own, so rather than one very long entry this release note is short and the details live in the four issues that follow it.
What changed
A third ground-truth state. 176 reviewed locations that a competent reviewer could call either way are now recorded as non_scoring, with a written reason, and excluded from scoring in both directions — never a hit, never a miss. Issue 14 explains why the two old options, label it or delete it, were both wrong.
A second language, and language tabs. The 74 TypeScript/JavaScript repositories — 24 community apps and 50 generated by coding agents across Express, Next.js, NestJS + Angular, Remix and Fastify + Vue — get their own leaderboard tab. The Overall tab ranks only scanners that have covered every repository in every language, so no scanner is ever scored on code it did not see. Issue 15 covers the corpus and the ranking rule.
What happens when the language changes. Eight of the nine scanners with full coverage lose precision on TypeScript, by 15 to 36 points, while the strongest hold their recall and the mid-tier does not. Issue 16 works through the numbers and finds the precision drop is mostly a property of the corpus.
Three new full-coverage scanners. GLM-5.3, GPT-6 Astra and Claude Sonnet 5 have now run on all 140 repositories and take fourth, fifth and sixth Overall. Issue 17 introduces them and the leaderboard's new effort and cost-per-line columns.
Where the leaderboard stands
The Python leaderboard is unchanged from 2.1.0 — same 28 scanners, same true- and false-positive counts, same F3 to the decimal — with one label moved: the Python results published as Kolega DevSec Max V0.0.1 now sit under V0.1.0, the same scanner whose Python path did not change. One correction to a previously published figure: GPT-5.6 Sol's Python score rose from F3 53.9 to 77.8 once we re-ran it at the same high reasoning-effort setting as its TypeScript run; the original had been run at an unrecorded default, and both halves of a comparison have to run at the same setting to be a comparison.
The Overall tab, strict F3 across all 140 repositories, reads: Kolega DevSec Max V0.1.0 84.4, Daybreak Blue 79.5, GPT-5.6 Sol 74.7, GLM-5.3 54.6, GPT-6 Astra 52.1, Claude Sonnet 5 43.3, DeepSeek V4 Flash 41.8, DeepSeek V4 Pro 38.0, Semgrep 10.0.
What's next
False-positive traps for the TypeScript/JavaScript corpus, the remaining Python-only scanners run on the new repositories, and Java as the next language — each arriving as its own tab, ranked separately until it has earned a place on Overall.