RealVuln favors fewer findings with audit-grade label quality over scale. Studies have found 40%+ of entries in popular auto-labeled datasets mislabeled; every entry here was reviewed by hand and ships with its evidence.
hand-labeled findings — 697 real vulnerabilities and 120 false-positive traps across 18 CWE families.
human-authored, with high confidence. All projects predate widespread LLM code generation — relevant for mitigating data-contamination concerns when evaluating LLM scanners.
Each target repository carries a manifest pinned to a commit SHA. An is_vulnerable: false entry is a false-positive trap; acceptable_cwes absorbs reasonable CWE ambiguity.
{
"schema_version": "1.0",
"repo_id": "realvuln-pygoat",
"repo_url": "https://github.com/adeyosemanputra/pygoat",
"commit_sha": "a1b2c3…", // pinned: prevents ground-truth drift
"type": 1, "language": "python", "framework": "django",
"authorship": "human_authored",
"authorship_confidence": "high",
"authorship_evidence": "pre-LLM project, established 2018",
"findings": [
{
"id": "pygoat-014", "is_vulnerable": true,
"vulnerability_class": "sql_injection",
"primary_cwe": "CWE-89",
"acceptable_cwes": ["CWE-89", "CWE-564", "CWE-943"],
"file": "introduction/views.py",
"location": { "start_line": 42, "end_line": 48, "function": "sql_lab" },
"severity": "high",
"evidence": { "source": "manual_review", "cve_id": null,
"description": "SQL injection via unsanitized parameter" }
},
{
"id": "pygoat-fp-003", "is_vulnerable": false, // false-positive trap
"vulnerability_class": "sql_injection",
"primary_cwe": "CWE-89",
"evidence": { "source": "manual_review",
"description": "ORM filter() — auto-parameterized, safe" }
}
]
}
The ground-truth corpus holds 26 repositories · 817 findings (697 vulnerabilities, 120 traps), all scored on the current leaderboard. The full corpus is listed below.
| Repository | Framework | Vulns | FP traps |
|---|---|---|---|
| pygoat | django | 70 | 10 |
| vulnpy | custom | 78 | 16 |
| vulpy | flask | 54 | 6 |
| djangoat | django | 50 | 6 |
| damn-vulnerable-graphql-application | flask | 35 | 4 |
| dsvpwa | custom | 32 | 6 |
| owasp-web-playground | flask | 29 | 6 |
| extremely-vulnerable-flask-app | flask | 28 | 4 |
| flask-xss | flask | 28 | 5 |
| dsvw | custom | 27 | 4 |
| lets-be-bad-guys | django | 24 | 4 |
| threatbyte | flask | 24 | 5 |
| defdev-app | flask | 22 | 5 |
| dvblab | flask | 22 | 4 |
| dvpwa | aiohttp | 22 | 4 |
| vulnerable-python-apps | flask | 22 | 5 |
| python-app | flask | 20 | 4 |
| vulnerable-flask-app | flask | 20 | 4 |
| damn-vulnerable-flask-application | flask | 15 | 4 |
| vulnerable-api | flask | 14 | 3 |
| vulnerable-tornado-app | tornado | 14 | 3 |
| vampi | flask | 13 | 4 |
| insecure-web | flask | 9 | 2 |
| vfapi | fastapi | 9 | 2 |
| python-insecure-app | fastapi | 8 | 2 |
| intentionally-vulnerable-python-application | flask | 7 | 2 |
| pythonssti | fastapi | 2 | 1 |