Scanner deep-dive
Claude Opus 5 by Anthropic ↗
General-Purpose LLM · agentic-v1 · scored on 66/66 repositories. Strict scoring (unfinished repos counted as misses).
Methodology note
Claude Code harness; post-hoc v2 scoring. Claude Opus 5 scanned all 66 pinned Python repositories through the Claude Code CLI using the generic agentic-v1 prompt (prompt hash sha256:de50937cb83f). This differs from the standard OpenCode delivery path used by the original agentic-v1 campaign. The raw findings were generated in a separate internal campaign against the same pinned Python commit set, then scored post-hoc against the frozen public v2 ground truth; ground truth was never supplied to the scanner. The public provenance manifest records all 66 commit SHAs, the full prompt/task hashes, and the exact CLI/model configuration. Internal dataset stamps are omitted from the public metrics. The 66 successful runs cost $90.68 in total.67.7
F3 (strict)
67.0
F2 (strict)
68.4%
Recall (strict)
62.0%
Precision
66/66
Repos scored
claude-opus-5
Model
$91
Total cost
208s
Avg latency
§
Per-repository breakdown
Each bar shows true positives, false positives, and misses on one repository; bar length is proportional to that repo's labeled vulnerabilities. Ranked by F2.
| Repository | TP | FP | FN | Recall % | F2 |
|---|---|---|---|---|---|
| intentionally-vulnerable-python-application | 7 | 3 | 0 | 100.0 | 92.1 |
| vfapi | 9 | 4 | 0 | 100.0 | 91.8 |
| dsvw | 26 | 8 | 1 | 96.3 | 91.5 |
| pythonssti | 2 | 1 | 0 | 100.0 | 90.9 |
| dsvpwa | 28 | 10 | 4 | 87.5 | 84.3 |
| vulnerable-tornado-app | 13 | 10 | 1 | 92.9 | 82.3 |
| vampi | 12 | 3 | 3 | 80.0 | 80.0 |
| vc-codex-high-seeded-v2-healthcare-clinic-django | 21 | 7 | 5 | 80.8 | 79.5 |
| vc-kimi-code-seeded-v2-property-management-fastapi | 24 | 8 | 6 | 80.0 | 78.9 |
| lets-be-bad-guys | 22 | 22 | 2 | 91.7 | 78.6 |
| insecure-web | 8 | 7 | 1 | 88.9 | 78.4 |
| vc-codex-high-seeded-v2-property-management-fastapi | 21 | 9 | 5 | 80.8 | 78.4 |
| dvblab | 17 | 6 | 5 | 77.3 | 76.6 |
| vc-codex-high-seeded-v2-hr-payroll-django | 19 | 6 | 6 | 76.0 | 76.0 |
| vc-codex-seeded-v2-logistics-dispatch-fastapi | 22 | 6 | 8 | 73.3 | 74.3 |
| vc-codex-high-seeded-v2-education-lms-django | 19 | 9 | 6 | 76.0 | 74.2 |
| vc-claude-code-seeded-v2-logistics-dispatch-fastapi | 24 | 6 | 9 | 72.7 | 74.1 |
| vc-codex-high-seeded-v2-marketplace-commerce-fastapi | 20 | 15 | 5 | 80.0 | 74.1 |
| vc-claude-code-seeded-v2-legal-case-django | 23 | 9 | 8 | 74.2 | 73.7 |
| vc-kimi-code-seeded-v2-marketplace-commerce-fastapi | 21 | 14 | 6 | 77.8 | 73.4 |
| vulnerable-api | 11 | 8 | 3 | 78.6 | 73.3 |
| python-app | 18 | 21 | 3 | 85.7 | 73.2 |
| vc-codex-high-seeded-v2-fintech-lending-fastapi | 21 | 7 | 8 | 72.4 | 72.9 |
| vc-codex-high-seeded-v2-support-desk-fastapi | 20 | 6 | 8 | 71.4 | 72.5 |
| vc-codex-high-seeded-v2-crm-saas-django | 19 | 13 | 6 | 76.0 | 72.0 |
| vc-codex-seeded-v2-marketplace-commerce-fastapi | 21 | 9 | 8 | 72.4 | 71.9 |
| vc-codex-seeded-v2-crm-saas-django | 26 | 19 | 8 | 76.5 | 71.8 |
| owasp-web-playground | 20 | 8 | 8 | 71.4 | 71.4 |
| vc-kimi-code-seeded-v2-support-desk-fastapi | 20 | 8 | 8 | 71.4 | 71.4 |
| threatbyte | 19 | 11 | 7 | 73.1 | 70.9 |
| vc-kimi-code-seeded-v2-logistics-dispatch-fastapi | 23 | 16 | 8 | 74.2 | 70.6 |
| vc-claude-code-seeded-v2-property-management-fastapi | 22 | 3 | 11 | 66.7 | 70.1 |
| vc-kimi-code-seeded-v2-healthcare-clinic-django | 21 | 9 | 9 | 70.0 | 70.0 |
| vc-kimi-code-seeded-v2-legal-case-django | 20 | 20 | 6 | 76.9 | 69.4 |
| vc-kimi-code-seeded-v2-crm-saas-django | 21 | 24 | 6 | 77.8 | 68.6 |
| vc-claude-code-seeded-v2-crm-saas-django | 19 | 8 | 9 | 67.9 | 68.3 |
| vc-claude-code-seeded-v2-support-desk-fastapi | 22 | 4 | 12 | 64.7 | 67.9 |
| vc-codex-high-seeded-v2-logistics-dispatch-fastapi | 21 | 18 | 8 | 72.4 | 67.7 |
| vc-codex-seeded-v2-education-lms-django | 24 | 18 | 11 | 68.6 | 65.9 |
| damn-vulnerable-graphql-application | 24 | 17 | 12 | 66.7 | 64.9 |
| vc-kimi-code-seeded-v2-education-lms-django | 21 | 29 | 7 | 75.0 | 64.8 |
| extremely-vulnerable-flask-app | 20 | 7 | 12 | 62.5 | 64.5 |
| vc-claude-code-seeded-v2-hr-payroll-django | 17 | 7 | 10 | 63.0 | 64.4 |
| vulnerable-flask-app | 14 | 11 | 7 | 66.7 | 64.2 |
| vc-claude-code-seeded-v2-fintech-lending-fastapi | 21 | 27 | 8 | 72.4 | 64.0 |
| vc-codex-seeded-v2-support-desk-fastapi | 19 | 10 | 11 | 63.3 | 63.8 |
| vc-claude-code-seeded-v2-marketplace-commerce-fastapi | 20 | 9 | 12 | 62.5 | 63.7 |
| vulnpy | 49 | 16 | 31 | 61.3 | 63.6 |
| vc-codex-seeded-v2-property-management-fastapi | 19 | 7 | 12 | 61.3 | 63.3 |
| vc-codex-seeded-v2-healthcare-clinic-django | 25 | 9 | 16 | 61.0 | 63.1 |
| vc-codex-seeded-v2-hr-payroll-django | 23 | 6 | 16 | 59.0 | 62.2 |
| vulpy | 35 | 19 | 22 | 61.4 | 62.1 |
| vc-codex-seeded-v2-legal-case-django | 20 | 9 | 13 | 60.6 | 62.1 |
| pygoat | 54 | 70 | 24 | 69.2 | 61.9 |
| vulnerable-python-apps | 14 | 11 | 8 | 63.6 | 61.9 |
| vc-kimi-code-seeded-v2-fintech-lending-fastapi | 22 | 24 | 11 | 66.7 | 61.8 |
| vc-claude-code-seeded-v2-healthcare-clinic-django | 18 | 14 | 11 | 62.1 | 60.8 |
| vc-claude-code-seeded-v2-education-lms-django | 18 | 4 | 14 | 56.2 | 60.0 |
| python-insecure-app | 5 | 5 | 3 | 62.5 | 59.5 |
| vc-kimi-code-seeded-v2-hr-payroll-django | 17 | 15 | 12 | 58.6 | 57.4 |
| flask-xss | 16 | 9 | 14 | 53.3 | 55.2 |
| dvpwa | 12 | 9 | 11 | 52.2 | 53.1 |
| djangoat | 27 | 24 | 25 | 51.9 | 52.1 |
| vc-codex-seeded-v2-fintech-lending-fastapi | 14 | 7 | 23 | 37.8 | 41.4 |
| damn-vulnerable-flask-application | 5 | 11 | 10 | 33.3 | 32.9 |
| vc-codex-high-seeded-v2-legal-case-django | 6 | 20 | 19 | 24.0 | 23.8 |
§
Detection by severity
| Severity | TP | FP | FN | Recall % |
|---|---|---|---|---|
| Critical | 145 | 2 | 10 | 93.5 |
| High | 524 | 2 | 228 | 69.7 |
| Medium | 579 | 2 | 331 | 63.6 |
| Low | 53 | 0 | 33 | 61.6 |
§
Detection by vulnerability class
| CWE family | TP | FP | FN | Recall % |
|---|---|---|---|---|
| Open Redirect | 40 | 0 | 0 | 100.0 |
| HTTP Header Injection | 2 | 0 | 0 | 100.0 |
| XPath Injection | 4 | 0 | 0 | 100.0 |
| Insecure Deserialization | 38 | 0 | 2 | 95.0 |
| SQL Injection | 74 | 0 | 4 | 94.9 |
| XML External Entities | 36 | 1 | 2 | 94.7 |
| Code Injection / RFI | 32 | 0 | 2 | 94.1 |
| Command / OS Injection | 47 | 0 | 3 | 94.0 |
| Path Traversal | 41 | 0 | 3 | 93.2 |
| Security Misconfiguration | 96 | 0 | 16 | 85.7 |
| Hardcoded Credentials | 52 | 1 | 16 | 76.5 |
| Cross-Site Scripting | 81 | 0 | 29 | 73.6 |
| Server-Side Request Forgery | 26 | 1 | 15 | 63.4 |
| Missing Authentication / Authorization | 51 | 0 | 32 | 61.4 |
| Other | 525 | 2 | 352 | 59.9 |
| Sensitive Data Exposure | 92 | 0 | 63 | 59.4 |
| Broken Access Control / IDOR | 45 | 1 | 38 | 54.2 |
| Denial of Service | 19 | 0 | 25 | 43.2 |
§
LLM operational metrics
37
Avg input tokens
16,295
Avg output tokens
704,090
Avg total tokens
208s
Avg latency / repo
0.0%
JSON repair rate
66
Total runs
§
Cost
$91
Total cost
$1.37
Cost / run
$0.068
Cost / 100 LOC
133,782
Python LOC scanned
66
Successful runs