You are viewing an archived release — RealVuln v3.1.0. View the latest benchmark →
realvuln v3.1
Dashboard Methodology Dataset Findings Roadmap Journal Paper GitHub ↗
02 · Dataset

The corpus

RealVuln favors fewer findings with audit-grade label quality over scale. Studies have found 40%+ of entries in popular auto-labeled datasets mislabeled; every entry here was reviewed by hand and ships with its evidence.

2.1

At a glance

Ground truth
4,418

hand-labeled findings — 4,138 real vulnerabilities and 280 false-positive traps across 18 CWE families.

Languages & frameworks (140 repos)
Python66
TypeScript53
JavaScript21
Express27
Django23
FastAPI23
Flask15
Next.js12
NestJS + Angular10
Remix10
Fastify + Vue10
React5
custom3
aiohttp1
Tornado1
Provenance
31%

human-authored (43 repos); the remaining 90 are vibe-coded (LLM-generated). Authorship is labeled per repo, enabling the authorship-vs-detection analysis and data-contamination controls when evaluating LLM scanners.

Lines of code are counted per repository's declared language: .py only for Python repos, .ts/.tsx/.js/.jsx/.mjs/.cjs/.vue for TypeScript/JavaScript ones. A Django or Flask app's server-rendered HTML templates and static assets are real frontend code but are not counted; a Next.js or Vue app's frontend components are .tsx/.vue files and are. That asymmetry, not a difference in how much frontend the two corpora have, is why TS/JS's 607,252 LOC looks disproportionate next to Python's 133,782 across a similar repo count.

2.2

Ground-truth schema

Each target repository carries a manifest pinned to a commit SHA. An is_vulnerable: false entry is a false-positive trap; acceptable_cwes absorbs reasonable CWE ambiguity.

ground-truth.json — example entries
{
  "schema_version": "1.0",
  "repo_id": "realvuln-pygoat",
  "repo_url": "https://github.com/adeyosemanputra/pygoat",
  "commit_sha": "a1b2c3…",          // pinned: prevents ground-truth drift
  "type": 1, "language": "python", "framework": "django",
  "authorship": "human_authored",
  "authorship_confidence": "high",
  "authorship_evidence": "pre-LLM project, established 2018",
  "findings": [
    {
      "id": "pygoat-014", "is_vulnerable": true,
      "vulnerability_class": "sql_injection",
      "primary_cwe": "CWE-89",
      "acceptable_cwes": ["CWE-89", "CWE-564", "CWE-943"],
      "file": "introduction/views.py",
      "location": { "start_line": 42, "end_line": 48, "function": "sql_lab" },
      "severity": "high",
      "evidence": { "source": "manual_review", "cve_id": null,
        "description": "SQL injection via unsanitized parameter" }
    },
    {
      "id": "pygoat-fp-003", "is_vulnerable": false,   // false-positive trap
      "vulnerability_class": "sql_injection",
      "primary_cwe": "CWE-89",
      "evidence": { "source": "manual_review",
        "description": "ORM filter() — auto-parameterized, safe" }
    }
  ]
}
2.3

Repositories

The ground-truth corpus holds 140 repositories · 4,418 findings (4,138 vulnerabilities, 280 traps, plus 176 non-scoring entries excluded from every metric), all scored on the current leaderboard. The full corpus is listed below.

RepositoryLanguageFrameworkVulnsFP traps
juice-shopTypeScriptexpress810
vulnpyPythoncustom8016
pygoatPythondjango7810
dvws-nodeJavaScriptexpress770
vulpyPythonflask576
djangoatPythondjango526
juice-shop-goofTypeScriptexpress510
vc-claude-code-seeded-v3-logistics-dispatch-fastify-vueTypeScriptfastify-vue510
vuln-node-express-swaggerJavaScriptexpress490
vc-claude-code-seeded-v3-crm-saas-nestjs-angularTypeScriptnestjs-angular490
vc-codex-high-seeded-v3-education-lms-nextjsTypeScriptnextjs480
vc-codex-high-seeded-v3-legal-case-remixTypeScriptremix470
vc-kimi-code-seeded-v3-education-lms-nextjsTypeScriptnextjs460
vc-codex-high-seeded-v3-logistics-dispatch-fastify-vueTypeScriptfastify-vue450
vc-codex-high-seeded-v3-marketplace-commerce-expressTypeScriptexpress440
vc-kimi-code-seeded-v3-logistics-dispatch-fastify-vueTypeScriptfastify-vue440
vc-claude-code-seeded-v3-healthcare-clinic-nextjsTypeScriptnextjs430
vc-kimi-code-seeded-v3-marketplace-commerce-expressTypeScriptexpress430
vc-codex-high-seeded-v3-healthcare-clinic-nextjsTypeScriptnextjs420
vc-codex-high-seeded-v3-hr-payroll-remixTypeScriptremix420
vc-codex-high-seeded-v3-property-management-fastify-vueTypeScriptfastify-vue420
vc-kimi-code-seeded-v3-fintech-lending-expressTypeScriptexpress420
vc-claude-code-seeded-v3-support-desk-nestjs-angularTypeScriptnestjs-angular410
vc-codex-seeded-v2-healthcare-clinic-djangoPythondjango414
vc-kimi-code-seeded-v3-legal-case-remixTypeScriptremix410
oss-oopssec-storeTypeScriptnextjs400
vc-claude-code-seeded-v3-fintech-lending-expressTypeScriptexpress400
vc-claude-code-seeded-v3-marketplace-commerce-expressTypeScriptexpress400
vc-kimi-code-seeded-v3-crm-saas-nestjs-angularTypeScriptnestjs-angular400
vc-codex-high-seeded-v3-fintech-lending-expressTypeScriptexpress390
vc-codex-seeded-v2-hr-payroll-djangoPythondjango394
vc-claude-code-seeded-v3-education-lms-nextjsTypeScriptnextjs380
vc-claude-code-seeded-v3-hr-payroll-remixTypeScriptremix370
vc-codex-high-seeded-v3-crm-saas-nestjs-angularTypeScriptnestjs-angular370
vc-codex-seeded-v2-fintech-lending-fastapiPythonfastapi374
dvgaPythonflask364
vc-kimi-code-seeded-v3-healthcare-clinic-nextjsTypeScriptnextjs360
vc-kimi-code-seeded-v3-hr-payroll-remixTypeScriptremix360
vc-codex-high-seeded-v3-support-desk-nestjs-angularTypeScriptnestjs-angular350
vc-codex-seeded-v2-education-lms-djangoPythondjango354
vc-claude-code-seeded-v2-support-desk-fastapiPythonfastapi344
vc-codex-seeded-v2-crm-saas-djangoPythondjango344
vc-claude-code-seeded-v2-logistics-dispatch-fastapiPythonfastapi334
vc-claude-code-seeded-v2-property-management-fastapiPythonfastapi334
vc-codex-seeded-v2-legal-case-djangoPythondjango334
vc-kimi-code-seeded-v2-fintech-lending-fastapiPythonfastapi334
dsvpwaPythoncustom326
extremely-vulnerable-flask-appPythonflask324
vc-claude-code-seeded-v2-education-lms-djangoPythondjango324
vc-claude-code-seeded-v2-marketplace-commerce-fastapiPythonfastapi324
vc-claude-code-seeded-v3-property-management-fastify-vueTypeScriptfastify-vue320
vc-kimi-code-seeded-v3-property-management-fastify-vueTypeScriptfastify-vue320
vc-kimi-code-seeded-v3-support-desk-nestjs-angularTypeScriptnestjs-angular320
vc-claude-code-seeded-v2-legal-case-djangoPythondjango314
vc-codex-seeded-v2-property-management-fastapiPythonfastapi314
vc-kimi-code-seeded-v2-logistics-dispatch-fastapiPythonfastapi314
flask-xssPythonflask305
vc-codex-seeded-v2-logistics-dispatch-fastapiPythonfastapi304
vc-codex-seeded-v2-support-desk-fastapiPythonfastapi303
vc-deepseek-v4-pro-seeded-v3-crm-saas-nestjs-angularTypeScriptnestjs-angular300
vc-deepseek-v4-pro-seeded-v3-logistics-dispatch-fastify-vueTypeScriptfastify-vue300
vc-kimi-code-seeded-v2-healthcare-clinic-djangoPythondjango304
vc-kimi-code-seeded-v2-property-management-fastapiPythonfastapi304
vulnnodeappJavaScriptexpress290
vc-claude-code-seeded-v2-fintech-lending-fastapiPythonfastapi294
vc-claude-code-seeded-v2-healthcare-clinic-djangoPythondjango294
vc-codex-high-seeded-v2-fintech-lending-fastapiPythonfastapi294
vc-codex-high-seeded-v2-logistics-dispatch-fastapiPythonfastapi294
vc-codex-seeded-v2-marketplace-commerce-fastapiPythonfastapi294
vc-kimi-code-seeded-v2-hr-payroll-djangoPythondjango294
nodegoatJavaScriptexpress280
owasp-web-playgroundPythonflask286
vc-claude-code-seeded-v2-crm-saas-djangoPythondjango284
vc-claude-code-seeded-v3-legal-case-remixTypeScriptremix280
vc-codex-high-seeded-v2-support-desk-fastapiPythonfastapi284
vc-deepseek-v4-flash-seeded-v3-education-lms-nextjsTypeScriptnextjs280
vc-deepseek-v4-flash-seeded-v3-logistics-dispatch-fastify-vueTypeScriptfastify-vue280
vc-kimi-code-seeded-v2-education-lms-djangoPythondjango284
vc-kimi-code-seeded-v2-support-desk-fastapiPythonfastapi284
dsvwPythoncustom274
vc-claude-code-seeded-v2-hr-payroll-djangoPythondjango274
vc-deepseek-v4-pro-seeded-v3-education-lms-nextjsTypeScriptnextjs270
vc-deepseek-v4-pro-seeded-v3-fintech-lending-expressTypeScriptexpress270
vc-kimi-code-seeded-v2-crm-saas-djangoPythondjango274
vc-kimi-code-seeded-v2-marketplace-commerce-fastapiPythonfastapi274
threatbytePythonflask265
vc-codex-high-seeded-v2-healthcare-clinic-djangoPythondjango264
vc-codex-high-seeded-v2-property-management-fastapiPythonfastapi264
vc-deepseek-v4-pro-seeded-v3-healthcare-clinic-nextjsTypeScriptnextjs260
vc-deepseek-v4-pro-seeded-v3-hr-payroll-remixTypeScriptremix260
vc-deepseek-v4-pro-seeded-v3-marketplace-commerce-expressTypeScriptexpress260
vc-kimi-code-seeded-v2-legal-case-djangoPythondjango264
vc-codex-high-seeded-v2-crm-saas-djangoPythondjango254
vc-codex-high-seeded-v2-education-lms-djangoPythondjango254
vc-codex-high-seeded-v2-hr-payroll-djangoPythondjango254
vc-codex-high-seeded-v2-legal-case-djangoPythondjango254
vc-codex-high-seeded-v2-marketplace-commerce-fastapiPythonfastapi254
vc-deepseek-v4-flash-seeded-v3-property-management-fastify-vueTypeScriptfastify-vue250
vc-deepseek-v4-pro-seeded-v3-legal-case-remixTypeScriptremix250
lets-be-bad-guysPythondjango244
vc-deepseek-v4-flash-seeded-v3-healthcare-clinic-nextjsTypeScriptnextjs240
vc-deepseek-v4-flash-seeded-v3-support-desk-nestjs-angularTypeScriptnestjs-angular240
dvpwaPythonaiohttp234
dvblabPythonflask224
ninjasworkoutJavaScriptexpress220
vulnerable-python-appsPythonflask225
vc-deepseek-v4-flash-seeded-v3-crm-saas-nestjs-angularTypeScriptnestjs-angular220
vc-deepseek-v4-flash-seeded-v3-fintech-lending-expressTypeScriptexpress220
vc-deepseek-v4-flash-seeded-v3-legal-case-remixTypeScriptremix220
python-appPythonflask214
vulnerable-flask-appPythonflask214
vulnerable-nodeJavaScriptexpress210
vc-deepseek-v4-flash-seeded-v3-marketplace-commerce-expressTypeScriptexpress210
ivnaJavaScriptexpress190
vulnerable-nodejsJavaScriptexpress190
vc-deepseek-v4-pro-seeded-v3-property-management-fastify-vueTypeScriptfastify-vue190
vc-deepseek-v4-pro-seeded-v3-support-desk-nestjs-angularTypeScriptnestjs-angular190
vc-deepseek-v4-flash-seeded-v3-hr-payroll-remixTypeScriptremix180
dvnaJavaScriptexpress170
nodejs-goofJavaScriptexpress170
vampiPythonflask154
vulnerable-rest-api-owasp-2023JavaScriptexpress150
damn-vulnerable-flask-appPythonflask145
vulnerable-apiPythonflask143
vulnerable-tornado-appPythontornado143
xvnaJavaScriptexpress120
insecure-webPythonflask92
vfapiPythonfastapi92
python-insecure-appPythonfastapi82
vulnerable-app-angularJavaScriptexpress80
intentionally-vulnerable-python-appPythonflask72
node-api-goatJavaScriptexpress70
react-securityJavaScriptreact70
vulnerable-react-fatihJavaScriptreact70
vuln-node-expressJavaScriptexpress50
python-sstiPythonfastapi21
nextjs-vulnerable-appJavaScriptnextjs10
react-test-benchJavaScriptreact10
reactvulnaJavaScriptreact10
simply-vulnerable-reactJavaScriptreact10