Scanner deep-dive
GPT-5.6 Sol by OpenAI ↗
General-Purpose LLM · codex-cli · scored on 140/140 repositories. Strict scoring (unfinished repos counted as misses).
Methodology note
Full coverage; subscription-metered. Runs through the Codex CLI authenticated against a ChatGPT subscription session rather than a metered OpenAI API key, at model_reasoning_effort=high on both languages. The cost shown is computed from Codex's reported token usage at OpenAI's published API rate for this model ($4.00/$20.00 per 1M input/output tokens, short context) -- an accurate estimate of what the run would cost if billed, not money actually charged on a subscription session. The Python figure was re-run. Our Codex runner had no reasoning-effort control at all until the TS/JS work in this release, so the original Python campaign ran at whatever Codex's ambient default was (unrecorded) rather than high. Re-running all 66 Python repositories at high on 2026-09-05 moved F3 from 53.9 to 77.8 (recall 48.3% to 78.7%, precision 78.5% to 70.6%) — confirming the mismatch, not the corpus, explained most of the earlier Python/TS-JS gap. The run was interrupted twice by the Codex/ChatGPT subscription's usage limit and resumed after each reset; every repository is now scored at the same effort level. The 74 TypeScript / JS repositories were scanned once each on 2026-09-04 with the same TS/JS prompt as the DeepSeek runs (prompt hash sha256:45a1200d61e6); all 74 validated first time.
74.7
F3 (strict)
72.6
F2 (strict)
76.9%
Recall (strict)
59.3%
Precision
140/140
Repos scored
gpt-5.6-sol
Model
$560
Total cost
569s
Avg latency
§
Per-repository breakdown
Each bar shows true positives, false positives, and misses on one repository; bar length is proportional to that repo's labeled vulnerabilities. Ranked by F2.
| Repository | TP | FP | FN | Recall % | F2 |
|---|---|---|---|---|---|
| insecure-web | 9 | 0 | 0 | 100.0 | 100.0 |
| lets-be-bad-guys | 24 | 0 | 0 | 100.0 | 100.0 |
| nextjs-vulnerable-app | 1 | 0 | 0 | 100.0 | 100.0 |
| pythonssti | 2 | 0 | 0 | 100.0 | 100.0 |
| react-test-bench | 1 | 0 | 0 | 100.0 | 100.0 |
| vampi | 15 | 0 | 0 | 100.0 | 100.0 |
| vfapi | 9 | 0 | 0 | 100.0 | 100.0 |
| vuln-node-express | 5 | 0 | 0 | 100.0 | 100.0 |
| vc-claude-code-seeded-v2-education-lms-django | 32 | 0 | 0 | 100.0 | 100.0 |
| vc-claude-code-seeded-v2-property-management-fastapi | 33 | 0 | 0 | 100.0 | 100.0 |
| vc-claude-code-seeded-v3-marketplace-commerce-express | 40 | 0 | 0 | 100.0 | 100.0 |
| vc-codex-high-seeded-v2-crm-saas-django | 25 | 0 | 0 | 100.0 | 100.0 |
| vc-codex-high-seeded-v3-support-desk-nestjs-angular | 35 | 0 | 0 | 100.0 | 100.0 |
| vc-deepseek-v4-pro-seeded-v3-crm-saas-nestjs-angular | 30 | 0 | 0 | 100.0 | 100.0 |
| vc-kimi-code-seeded-v2-healthcare-clinic-django | 30 | 0 | 0 | 100.0 | 100.0 |
| vc-codex-seeded-v2-healthcare-clinic-django | 41 | 1 | 0 | 100.0 | 99.5 |
| vc-codex-high-seeded-v2-marketplace-commerce-fastapi | 25 | 1 | 0 | 100.0 | 99.2 |
| vc-codex-high-seeded-v2-fintech-lending-fastapi | 29 | 2 | 0 | 100.0 | 98.6 |
| vc-codex-seeded-v2-hr-payroll-django | 39 | 3 | 0 | 100.0 | 98.5 |
| vc-claude-code-seeded-v3-hr-payroll-remix | 36 | 1 | 1 | 97.3 | 97.3 |
| vc-claude-code-seeded-v2-crm-saas-django | 28 | 4 | 0 | 100.0 | 97.2 |
| vc-codex-high-seeded-v3-fintech-lending-express | 39 | 6 | 0 | 100.0 | 97.0 |
| vc-claude-code-seeded-v2-legal-case-django | 31 | 5 | 0 | 100.0 | 96.9 |
| vc-codex-seeded-v2-property-management-fastapi | 31 | 6 | 0 | 100.0 | 96.3 |
| vc-codex-seeded-v2-legal-case-django | 33 | 7 | 0 | 100.0 | 95.9 |
| vc-claude-code-seeded-v2-marketplace-commerce-fastapi | 32 | 8 | 0 | 100.0 | 95.2 |
| vc-codex-high-seeded-v2-support-desk-fastapi | 28 | 7 | 0 | 100.0 | 95.2 |
| vc-codex-seeded-v2-logistics-dispatch-fastapi | 30 | 8 | 0 | 100.0 | 94.9 |
| vc-kimi-code-seeded-v3-healthcare-clinic-nextjs | 34 | 2 | 2 | 94.4 | 94.4 |
| vc-codex-high-seeded-v3-healthcare-clinic-nextjs | 42 | 14 | 0 | 100.0 | 93.8 |
| vc-claude-code-seeded-v3-support-desk-nestjs-angular | 41 | 14 | 0 | 100.0 | 93.6 |
| vc-claude-code-seeded-v2-healthcare-clinic-django | 29 | 10 | 0 | 100.0 | 93.5 |
| vuln-node-express-swagger | 48 | 13 | 1 | 98.0 | 93.4 |
| vc-codex-seeded-v2-education-lms-django | 34 | 9 | 1 | 97.1 | 92.9 |
| vc-codex-high-seeded-v3-marketplace-commerce-express | 44 | 17 | 0 | 100.0 | 92.8 |
| vc-kimi-code-seeded-v2-support-desk-fastapi | 28 | 12 | 0 | 100.0 | 92.1 |
| vc-codex-high-seeded-v2-logistics-dispatch-fastapi | 29 | 13 | 0 | 100.0 | 91.8 |
| vc-kimi-code-seeded-v2-hr-payroll-django | 29 | 15 | 0 | 100.0 | 90.6 |
| extremely-vulnerable-flask-app | 30 | 8 | 2 | 93.8 | 90.4 |
| vc-deepseek-v4-pro-seeded-v3-legal-case-remix | 24 | 11 | 1 | 96.0 | 88.9 |
| vc-codex-high-seeded-v3-property-management-fastify-vue | 41 | 22 | 1 | 97.6 | 88.7 |
| vc-codex-high-seeded-v3-crm-saas-nestjs-angular | 35 | 15 | 2 | 94.6 | 88.4 |
| vc-kimi-code-seeded-v2-crm-saas-django | 27 | 18 | 0 | 100.0 | 88.2 |
| vc-kimi-code-seeded-v3-education-lms-nextjs | 46 | 33 | 0 | 100.0 | 87.5 |
| vc-kimi-code-seeded-v3-logistics-dispatch-fastify-vue | 43 | 27 | 1 | 97.7 | 87.4 |
| vc-claude-code-seeded-v3-property-management-fastify-vue | 31 | 19 | 1 | 96.9 | 87.1 |
| dvws-node | 68 | 17 | 9 | 88.3 | 86.5 |
| python-insecure-app | 7 | 2 | 1 | 87.5 | 85.4 |
| vulnerable-app-angular | 8 | 7 | 0 | 100.0 | 85.1 |
| ivna | 18 | 12 | 1 | 94.7 | 84.9 |
| vc-kimi-code-seeded-v2-legal-case-django | 25 | 19 | 1 | 96.2 | 84.5 |
| vc-kimi-code-seeded-v3-property-management-fastify-vue | 31 | 26 | 1 | 96.9 | 83.8 |
| vc-claude-code-seeded-v3-crm-saas-nestjs-angular | 42 | 14 | 7 | 85.7 | 83.3 |
| vc-deepseek-v4-flash-seeded-v3-hr-payroll-remix | 18 | 18 | 0 | 100.0 | 83.3 |
| juice-shop-goof | 45 | 22 | 6 | 88.2 | 83.0 |
| dvna | 15 | 8 | 2 | 88.2 | 82.4 |
| vc-deepseek-v4-flash-seeded-v3-crm-saas-nestjs-angular | 20 | 15 | 2 | 90.9 | 81.3 |
| vc-kimi-code-seeded-v3-support-desk-nestjs-angular | 31 | 33 | 1 | 96.9 | 80.7 |
| dvblab | 19 | 11 | 3 | 86.4 | 80.5 |
| vc-codex-seeded-v2-fintech-lending-fastapi | 29 | 4 | 8 | 78.4 | 80.1 |
| vulnnodeapp | 25 | 16 | 4 | 86.2 | 79.6 |
| damn-vulnerable-graphql-application | 29 | 10 | 7 | 80.6 | 79.2 |
| vc-deepseek-v4-flash-seeded-v3-marketplace-commerce-express | 19 | 17 | 2 | 90.5 | 79.2 |
| threatbyte | 21 | 8 | 5 | 80.8 | 78.9 |
| vc-deepseek-v4-flash-seeded-v3-logistics-dispatch-fastify-vue | 27 | 32 | 1 | 96.4 | 78.9 |
| vc-claude-code-seeded-v3-education-lms-nextjs | 33 | 25 | 5 | 86.8 | 78.6 |
| vc-codex-high-seeded-v3-hr-payroll-remix | 37 | 32 | 5 | 88.1 | 78.1 |
| dsvw | 20 | 2 | 7 | 74.1 | 76.9 |
| react-security | 7 | 11 | 0 | 100.0 | 76.1 |
| vc-codex-high-seeded-v2-hr-payroll-django | 20 | 12 | 5 | 80.0 | 75.8 |
| python-app | 16 | 6 | 5 | 76.2 | 75.5 |
| vc-claude-code-seeded-v2-support-desk-fastapi | 25 | 9 | 9 | 73.5 | 73.5 |
| vc-claude-code-seeded-v2-logistics-dispatch-fastapi | 25 | 15 | 8 | 75.8 | 72.7 |
| vc-kimi-code-seeded-v2-logistics-dispatch-fastapi | 26 | 29 | 5 | 83.9 | 72.6 |
| vc-kimi-code-seeded-v2-property-management-fastapi | 26 | 33 | 4 | 86.7 | 72.6 |
| vc-claude-code-seeded-v2-fintech-lending-fastapi | 22 | 15 | 7 | 75.9 | 71.9 |
| intentionally-vulnerable-python-application | 5 | 2 | 2 | 71.4 | 71.4 |
| nodegoat | 21 | 14 | 7 | 75.0 | 71.4 |
| vc-deepseek-v4-flash-seeded-v3-property-management-fastify-vue | 23 | 38 | 2 | 92.0 | 71.4 |
| vulnpy | 55 | 12 | 25 | 68.8 | 71.1 |
| vc-codex-high-seeded-v2-property-management-fastapi | 20 | 17 | 6 | 76.9 | 70.9 |
| vc-codex-seeded-v2-marketplace-commerce-fastapi | 21 | 11 | 8 | 72.4 | 70.9 |
| dsvpwa | 23 | 12 | 9 | 71.9 | 70.6 |
| node-api-goat | 5 | 3 | 2 | 71.4 | 69.4 |
| xvna | 10 | 14 | 2 | 83.3 | 69.4 |
| vc-kimi-code-seeded-v2-education-lms-django | 22 | 25 | 6 | 78.6 | 69.2 |
| vc-deepseek-v4-pro-seeded-v3-healthcare-clinic-nextjs | 21 | 28 | 5 | 80.8 | 68.6 |
| vc-kimi-code-seeded-v2-marketplace-commerce-fastapi | 21 | 24 | 6 | 77.8 | 68.6 |
| vulnerable-tornado-app | 10 | 8 | 4 | 71.4 | 67.6 |
| owasp-web-playground | 18 | 7 | 10 | 64.3 | 65.7 |
| djangoat | 35 | 25 | 17 | 67.3 | 65.3 |
| vc-codex-high-seeded-v2-education-lms-django | 16 | 7 | 9 | 64.0 | 65.0 |
| vc-codex-seeded-v2-support-desk-fastapi | 19 | 8 | 11 | 63.3 | 64.6 |
| vc-claude-code-seeded-v3-legal-case-remix | 20 | 23 | 8 | 71.4 | 64.5 |
| nodejs-goof | 12 | 20 | 5 | 70.6 | 60.0 |
| vc-claude-code-seeded-v3-logistics-dispatch-fastify-vue | 32 | 31 | 19 | 62.7 | 59.9 |
| vc-codex-seeded-v2-crm-saas-django | 20 | 12 | 14 | 58.8 | 59.5 |
| vulnerable-flask-app | 12 | 6 | 9 | 57.1 | 58.8 |
| vulnerable-nodejs | 12 | 14 | 7 | 63.2 | 58.8 |
| vc-kimi-code-seeded-v3-fintech-lending-express | 26 | 28 | 16 | 61.9 | 58.6 |
| vc-kimi-code-seeded-v2-fintech-lending-fastapi | 19 | 13 | 14 | 57.6 | 57.9 |
| vulnerable-node | 13 | 16 | 8 | 61.9 | 57.5 |
| vc-deepseek-v4-flash-seeded-v3-education-lms-nextjs | 17 | 19 | 11 | 60.7 | 57.4 |
| vc-kimi-code-seeded-v3-marketplace-commerce-express | 26 | 29 | 17 | 60.5 | 57.3 |
| vc-deepseek-v4-flash-seeded-v3-legal-case-remix | 14 | 22 | 8 | 63.6 | 56.5 |
| pygoat | 43 | 28 | 35 | 55.1 | 56.1 |
| vc-deepseek-v4-flash-seeded-v3-fintech-lending-express | 15 | 31 | 7 | 68.2 | 56.0 |
| vc-kimi-code-seeded-v3-legal-case-remix | 23 | 25 | 18 | 56.1 | 54.2 |
| juice-shop | 45 | 47 | 36 | 55.6 | 54.1 |
| vc-codex-high-seeded-v3-logistics-dispatch-fastify-vue | 25 | 29 | 20 | 55.6 | 53.4 |
| vc-deepseek-v4-flash-seeded-v3-healthcare-clinic-nextjs | 15 | 31 | 9 | 62.5 | 52.8 |
| vulnerable-rest-api-owasp-2023 | 10 | 25 | 5 | 66.7 | 52.6 |
| dvpwa | 11 | 5 | 12 | 47.8 | 50.9 |
| vc-claude-code-seeded-v3-fintech-lending-express | 22 | 36 | 18 | 55.0 | 50.5 |
| damn-vulnerable-flask-application | 7 | 7 | 7 | 50.0 | 50.0 |
| flask-xss | 14 | 6 | 16 | 46.7 | 50.0 |
| vc-codex-high-seeded-v2-healthcare-clinic-django | 13 | 16 | 13 | 50.0 | 48.9 |
| vc-codex-high-seeded-v3-legal-case-remix | 23 | 25 | 24 | 48.9 | 48.7 |
| vc-kimi-code-seeded-v3-hr-payroll-remix | 19 | 35 | 17 | 52.8 | 48.0 |
| vc-claude-code-seeded-v3-healthcare-clinic-nextjs | 22 | 37 | 21 | 51.2 | 47.6 |
| vulnerable-api | 6 | 2 | 8 | 42.9 | 46.9 |
| vc-deepseek-v4-pro-seeded-v3-support-desk-nestjs-angular | 12 | 42 | 7 | 63.2 | 46.2 |
| reactvulna | 1 | 6 | 0 | 100.0 | 45.5 |
| vc-codex-high-seeded-v3-education-lms-nextjs | 22 | 28 | 26 | 45.8 | 45.5 |
| vc-deepseek-v4-pro-seeded-v3-logistics-dispatch-fastify-vue | 17 | 55 | 13 | 56.7 | 44.3 |
| vulnerable-react-fatih | 4 | 14 | 3 | 57.1 | 43.5 |
| vc-kimi-code-seeded-v3-crm-saas-nestjs-angular | 19 | 44 | 21 | 47.5 | 42.6 |
| vc-deepseek-v4-pro-seeded-v3-education-lms-nextjs | 13 | 34 | 14 | 48.1 | 41.9 |
| vc-deepseek-v4-pro-seeded-v3-property-management-fastify-vue | 10 | 37 | 9 | 52.6 | 40.7 |
| oss-oopssec-store | 17 | 34 | 23 | 42.5 | 40.3 |
| vc-claude-code-seeded-v2-hr-payroll-django | 10 | 6 | 17 | 37.0 | 40.3 |
| vc-deepseek-v4-flash-seeded-v3-support-desk-nestjs-angular | 9 | 20 | 15 | 37.5 | 36.0 |
| vulpy | 19 | 18 | 38 | 33.3 | 35.8 |
| vc-codex-high-seeded-v2-legal-case-django | 9 | 25 | 16 | 36.0 | 33.6 |
| vulnerable-python-apps | 7 | 10 | 15 | 31.8 | 33.3 |
| vc-deepseek-v4-pro-seeded-v3-hr-payroll-remix | 9 | 27 | 17 | 34.6 | 32.1 |
| vc-deepseek-v4-pro-seeded-v3-marketplace-commerce-express | 8 | 28 | 18 | 30.8 | 28.6 |
| vc-deepseek-v4-pro-seeded-v3-fintech-lending-express | 8 | 27 | 19 | 29.6 | 28.0 |
| simply-vulnerable-react | 1 | 15 | 0 | 100.0 | 25.0 |
| ninjasworkout | 5 | 28 | 17 | 22.7 | 20.7 |
§
Detection by severity
| Severity | TP | FP | FN | Recall % |
|---|---|---|---|---|
| Critical | 145 | 0 | 13 | 91.8 |
| High | 2196 | 1 | 652 | 77.1 |
| Medium | 737 | 2 | 229 | 76.3 |
| Low | 105 | 0 | 61 | 63.3 |
§
Detection by vulnerability class
| CWE family | TP | FP | FN | Recall % |
|---|---|---|---|---|
| XPath Injection | 17 | 0 | 0 | 100.0 |
| Open Redirect | 85 | 0 | 3 | 96.6 |
| Insecure Deserialization | 45 | 0 | 3 | 93.8 |
| Path Traversal | 96 | 0 | 8 | 92.3 |
| XML External Entities | 59 | 0 | 5 | 92.2 |
| Command / OS Injection | 124 | 0 | 12 | 91.2 |
| SQL Injection | 135 | 0 | 19 | 87.7 |
| Code Injection / RFI | 70 | 0 | 10 | 87.5 |
| Cross-Site Scripting | 165 | 0 | 41 | 80.1 |
| Server-Side Request Forgery | 74 | 1 | 19 | 79.6 |
| Security Misconfiguration | 161 | 0 | 43 | 78.9 |
| Broken Access Control / IDOR | 183 | 1 | 49 | 78.9 |
| Denial of Service | 68 | 0 | 19 | 78.2 |
| Missing Authentication / Authorization | 128 | 0 | 45 | 74.0 |
| Other | 1386 | 0 | 499 | 73.5 |
| Sensitive Data Exposure | 248 | 0 | 101 | 71.1 |
| Hardcoded Credentials | 138 | 1 | 78 | 63.9 |
| HTTP Header Injection | 1 | 0 | 1 | 50.0 |
§
LLM operational metrics
2,539,446
Avg input tokens
26,597
Avg output tokens
2,580,604
Avg total tokens
569s
Avg latency / repo
0.0%
JSON repair rate
140
Total runs
§
Cost
$560
Total cost
$4.00
Cost / run
$0.076
Cost / 100 LOC
741,034
Lines of code scanned
140
Successful runs