realvuln v2.0
Dashboard Methodology Dataset Findings Roadmap GitHub ↗
02 · Dataset

The corpus

RealVuln favors fewer findings with audit-grade label quality over scale. Studies have found 40%+ of entries in popular auto-labeled datasets mislabeled; every entry here was reviewed by hand and ships with its evidence.

2.1

At a glance

Ground truth
2,182

hand-labeled findings — 1,903 real vulnerabilities and 279 false-positive traps across 18 CWE families.

Frameworks (66 repos)
Django23
FastAPI23
Flask15
custom3
aiohttp1
Tornado1
Provenance
39%

human-authored (26 repos); the remaining 40 are vibe-coded (LLM-generated). Authorship is labeled per repo, enabling the authorship-vs-detection analysis and data-contamination controls when evaluating LLM scanners.

2.2

Ground-truth schema

Each target repository carries a manifest pinned to a commit SHA. An is_vulnerable: false entry is a false-positive trap; acceptable_cwes absorbs reasonable CWE ambiguity.

ground-truth.json — example entries
{
  "schema_version": "1.0",
  "repo_id": "realvuln-pygoat",
  "repo_url": "https://github.com/adeyosemanputra/pygoat",
  "commit_sha": "a1b2c3…",          // pinned: prevents ground-truth drift
  "type": 1, "language": "python", "framework": "django",
  "authorship": "human_authored",
  "authorship_confidence": "high",
  "authorship_evidence": "pre-LLM project, established 2018",
  "findings": [
    {
      "id": "pygoat-014", "is_vulnerable": true,
      "vulnerability_class": "sql_injection",
      "primary_cwe": "CWE-89",
      "acceptable_cwes": ["CWE-89", "CWE-564", "CWE-943"],
      "file": "introduction/views.py",
      "location": { "start_line": 42, "end_line": 48, "function": "sql_lab" },
      "severity": "high",
      "evidence": { "source": "manual_review", "cve_id": null,
        "description": "SQL injection via unsanitized parameter" }
    },
    {
      "id": "pygoat-fp-003", "is_vulnerable": false,   // false-positive trap
      "vulnerability_class": "sql_injection",
      "primary_cwe": "CWE-89",
      "evidence": { "source": "manual_review",
        "description": "ORM filter() — auto-parameterized, safe" }
    }
  ]
}
2.3

Repositories

The ground-truth corpus holds 66 repositories · 2,182 findings (1,903 vulnerabilities, 279 traps), all scored on the current leaderboard. The full corpus is listed below.

RepositoryFrameworkVulnsFP traps
vulnpycustom8016
pygoatdjango7810
vulpyflask576
djangoatdjango526
vc-codex-seeded-v2-healthcare-clinic-djangodjango414
vc-codex-seeded-v2-hr-payroll-djangodjango394
vc-codex-seeded-v2-fintech-lending-fastapifastapi374
dvgaflask364
vc-codex-seeded-v2-education-lms-djangodjango354
vc-claude-code-seeded-v2-support-desk-fastapifastapi344
vc-codex-seeded-v2-crm-saas-djangodjango344
vc-claude-code-seeded-v2-logistics-dispatch-fastapifastapi334
vc-claude-code-seeded-v2-property-management-fastapifastapi334
vc-codex-seeded-v2-legal-case-djangodjango334
vc-kimi-code-seeded-v2-fintech-lending-fastapifastapi334
dsvpwacustom326
extremely-vulnerable-flask-appflask324
vc-claude-code-seeded-v2-education-lms-djangodjango324
vc-claude-code-seeded-v2-marketplace-commerce-fastapifastapi324
vc-claude-code-seeded-v2-legal-case-djangodjango314
vc-codex-seeded-v2-property-management-fastapifastapi314
vc-kimi-code-seeded-v2-logistics-dispatch-fastapifastapi314
flask-xssflask305
vc-codex-seeded-v2-logistics-dispatch-fastapifastapi304
vc-codex-seeded-v2-support-desk-fastapifastapi303
vc-kimi-code-seeded-v2-healthcare-clinic-djangodjango304
vc-kimi-code-seeded-v2-property-management-fastapifastapi304
vc-claude-code-seeded-v2-fintech-lending-fastapifastapi294
vc-claude-code-seeded-v2-healthcare-clinic-djangodjango294
vc-codex-high-seeded-v2-fintech-lending-fastapifastapi294
vc-codex-high-seeded-v2-logistics-dispatch-fastapifastapi294
vc-codex-seeded-v2-marketplace-commerce-fastapifastapi294
vc-kimi-code-seeded-v2-hr-payroll-djangodjango294
owasp-web-playgroundflask286
vc-claude-code-seeded-v2-crm-saas-djangodjango284
vc-codex-high-seeded-v2-support-desk-fastapifastapi284
vc-kimi-code-seeded-v2-education-lms-djangodjango284
vc-kimi-code-seeded-v2-support-desk-fastapifastapi284
dsvwcustom274
vc-claude-code-seeded-v2-hr-payroll-djangodjango274
vc-kimi-code-seeded-v2-crm-saas-djangodjango274
vc-kimi-code-seeded-v2-marketplace-commerce-fastapifastapi274
threatbyteflask265
vc-codex-high-seeded-v2-healthcare-clinic-djangodjango264
vc-codex-high-seeded-v2-property-management-fastapifastapi264
vc-kimi-code-seeded-v2-legal-case-djangodjango264
vc-codex-high-seeded-v2-crm-saas-djangodjango254
vc-codex-high-seeded-v2-education-lms-djangodjango254
vc-codex-high-seeded-v2-hr-payroll-djangodjango254
vc-codex-high-seeded-v2-legal-case-djangodjango254
vc-codex-high-seeded-v2-marketplace-commerce-fastapifastapi254
lets-be-bad-guysdjango244
dvpwaaiohttp234
dvblabflask224
vulnerable-python-appsflask225
python-appflask214
vulnerable-flask-appflask214
damn-vulnerable-flask-appflask154
vampiflask154
vulnerable-apiflask143
vulnerable-tornado-apptornado143
insecure-webflask92
vfapifastapi92
python-insecure-appfastapi82
intentionally-vulnerable-python-appflask72
python-sstifastapi21