# RealVuln > An open, reproducible benchmark measuring how well security scanners find real vulnerabilities in real code. Version 2.1: 66 pinned Python repositories, 1,902 hand-labeled vulnerabilities and 280 false-positive traps, 28 scanners compared. Ranked by F3 (recall-weighted). Every ground-truth label, scanner output and scoring script is public so results can be reproduced independently. RealVuln is maintained by John Pellew and Faizan Raza of Kolega.Dev, whose scanner is itself one of the evaluated tools — hence the open audit trail. ## Core pages - [Benchmark overview](https://realvuln.com/): headline leaderboard and what the benchmark measures. - [Interactive dashboard](https://realvuln.com/dashboard.html): full leaderboard, precision-recall, cost-efficiency and per-category breakdowns. - [Methodology](https://realvuln.com/methodology.html): benchmark design, ground-truth labeling, finding-matching algorithm and F3 scoring. - [Dataset](https://realvuln.com/dataset.html): the corpus, ground-truth schema, false-positive traps and framework coverage. - [Findings](https://realvuln.com/findings.html): results and analysis, including the three-tier scanner hierarchy and per-CWE detection. - [Roadmap & contributing](https://realvuln.com/roadmap.html): versioning policy, how to contribute scanners or repositories, and how to cite. - [Paper](https://realvuln.com/paper.html): abstract, headline results, common questions and citation for arXiv:2604.13764. ## Per-scanner results - [Kolega DevSec Max V0.0.1 (Kolega DevSec Platform)](https://realvuln.com/scanners/kolega-devsec-max-v0.0.1.html): per-repository scores, detection breakdown and run conditions. - [Kolega DevSec Core V0.0.1 (Kolega DevSec Platform)](https://realvuln.com/scanners/kolega-devsec-core-v0.0.1.html): per-repository scores, detection breakdown and run conditions. - [Kolega Scan OSS - V2 - 3 Model (Kolega Scan OSS)](https://realvuln.com/scanners/kolega-claude-adaptation.html): per-repository scores, detection breakdown and run conditions. - [Kolega Scan OSS - V1 - 2 Model (Kolega Scan OSS)](https://realvuln.com/scanners/kolega-claude-adaptation-deepseek-only.html): per-repository scores, detection breakdown and run conditions. - [Anthropic Method (DeepSeek V4 Pro) (Bespoke Adaptation)](https://realvuln.com/scanners/kolega-original-claude-adaptation-deepseek-v4-pro.html): per-repository scores, detection breakdown and run conditions. - [Anthropic Method - Sonnet 4.6 (Claude Code)](https://realvuln.com/scanners/kolega-ca-cc-sonnet.html): per-repository scores, detection breakdown and run conditions. - [GPT-5.5 (agentic-v1)](https://realvuln.com/scanners/gpt-5.5-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [GPT-5.6 Luna (codex-cli)](https://realvuln.com/scanners/gpt-5.6-luna-codex-cli.html): per-repository scores, detection breakdown and run conditions. - [GPT-5.6 Terra (codex-cli)](https://realvuln.com/scanners/gpt-5.6-terra-codex-cli.html): per-repository scores, detection breakdown and run conditions. - [GPT-5.6 Sol (codex-cli)](https://realvuln.com/scanners/gpt-5.6-sol-codex-cli.html): per-repository scores, detection breakdown and run conditions. - [GLM-5.2 (agentic-v1)](https://realvuln.com/scanners/glm-5.2-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [GLM-5.1 (agentic-v1)](https://realvuln.com/scanners/glm-5.1-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [GLM-5 (agentic-v1)](https://realvuln.com/scanners/glm-5-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [DeepSeek V4 Flash (agentic-v1)](https://realvuln.com/scanners/deepseek-v4-flash-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [DeepSeek V4 Pro (agentic-v1)](https://realvuln.com/scanners/deepseek-v4-pro-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [Kimi K3 (agentic-v1)](https://realvuln.com/scanners/kimi-k3-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [Kimi K2.7 (agentic-v1)](https://realvuln.com/scanners/kimi-k2.7-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [Kimi K2.6 (agentic-v1)](https://realvuln.com/scanners/kimi-k2.6-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [Kimi K2.5 (agentic-v1)](https://realvuln.com/scanners/kimi-k2.5-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [Claude Opus 5 (agentic-v1)](https://realvuln.com/scanners/claude-opus-5-cc-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [Gemini 3.5 Flash (agentic-v1)](https://realvuln.com/scanners/gemini-3.5-flash-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [Minimax M2.7 (agentic-v1)](https://realvuln.com/scanners/minimax-m2.7-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [Qwen 3.6 35B (agentic-v1)](https://realvuln.com/scanners/qwen3.6-35b-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [Gemma 4 31B (agentic-v1)](https://realvuln.com/scanners/gemma4-31b-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [Ornith 1.0 35B (agentic-v1)](https://realvuln.com/scanners/ornith-q3-agentic-v1.html): per-repository scores, detection breakdown and run conditions. - [Rowan (0.2.0)](https://realvuln.com/scanners/rowan.html): per-repository scores, detection breakdown and run conditions. - [Semgrep (rule-based)](https://realvuln.com/scanners/semgrep.html): per-repository scores, detection breakdown and run conditions. - [SonarQube (community)](https://realvuln.com/scanners/sonarqube.html): per-repository scores, detection breakdown and run conditions. ## Journal - [RealVuln Journal](https://realvuln.com/journal.html): the narrative dev-log — every release, benchmark run, and methodology change, dated and in order. - [First LLM scanner runs: the agentic models enter the leaderboard](https://realvuln.com/journal/first-llm-scanner-runs.html) - [Frontier and community scanners: Claude Opus 5 and the first external contribution (Rowan)](https://realvuln.com/journal/frontier-and-community-scanners.html) - [Introducing the RealVuln benchmark framework](https://realvuln.com/journal/introducing-the-benchmark.html) - [Local models and GPT-5.6: can open-weight scanners compete?](https://realvuln.com/journal/local-models-and-gpt-5-6.html) - [Reproducible by design: prompt versioning and the manifest](https://realvuln.com/journal/reproducible-by-design.html) - [Scaling the roster: Opus 4.7, GLM-5.1, Kimi K2.6, GPT-5.5 and DeepSeek-V4](https://realvuln.com/journal/scaling-the-scanner-roster.html) - [The paper, and why we score F3 (recall weighted nine times over precision)](https://realvuln.com/journal/the-paper-and-the-f3-metric.html) - [The v2 corpus: adding 40 LLM-generated applications and measuring authorship](https://realvuln.com/journal/the-v2-corpus-llm-generated-code.html) - [RealVuln 1.0.0: freezing the first release and opening the public site](https://realvuln.com/journal/v1-0-0-release.html) - [RealVuln 2.0.0: the full 66-repo, false-positive-aware release](https://realvuln.com/journal/v2-0-0-release.html) - [RealVuln 2.1.0: the current release and the dataset export](https://realvuln.com/journal/v2-1-0-release.html) - [Why we built RealVuln: the false-positive problem no benchmark measured](https://realvuln.com/journal/why-we-built-realvuln.html) ## Optional - [Paper (PDF)](https://realvuln.com/assets/realvuln-paper.pdf): arXiv:2604.13764. - [Source & ground truth](https://github.com/kolega-ai/Real-Vuln-Benchmark): Apache 2.0. - [Archived releases](https://realvuln.com/versions.json): frozen, citable snapshots under /v//. Cite these for reproducibility; prefer the live pages above for current results.