Scanner deep-dive
GPT-6 Astra by OpenAI ↗
General-Purpose LLM · codex-cli · scored on 140/140 repositories. Strict scoring (unfinished repos counted as misses).
Methodology note
Full coverage (140/140). GPT-6 Astra, OpenAI's newest flagship ("our most capable model for complex, demanding work"), run through the Codex CLI at high reasoning effort, the same setting used for Sol and Daybreak Blue. F3 49.4 on Python (66/66), 54.3 on TypeScript / JS (74/74), Overall 52.1 — well below GPT-5.6 Sol (74.7) and Daybreak Blue (79.5) at the identical effort level and prompt. OpenAI's newest and most expensive model in this benchmark (2.5× Sol's price) is not its strongest: it lands fifth, behind two models built on an older, cheaper base and behind GLM-5.3 (54.6). The TypeScript / JS run was interrupted twice by the Codex/ChatGPT subscription's usage limit and once by one repository's validation failure; every repository is now scored at the same effort as Python.52.1
F3 (strict)
51.2
F2 (strict)
53.1%
Recall (strict)
44.7%
Precision
140/140
Repos scored
gpt-6-astra
Model
$1,041
Total cost
575s
Avg latency
§
Per-repository breakdown
Each bar shows true positives, false positives, and misses on one repository; bar length is proportional to that repo's labeled vulnerabilities. Ranked by F2.
| Repository | TP | FP | FN | Recall % | F2 |
|---|---|---|---|---|---|
| pythonssti | 2 | 0 | 0 | 100.0 | 100.0 |
| react-test-bench | 1 | 0 | 0 | 100.0 | 100.0 |
| vfapi | 8 | 2 | 1 | 88.9 | 87.0 |
| insecure-web | 8 | 3 | 1 | 88.9 | 85.1 |
| nextjs-vulnerable-app | 1 | 1 | 0 | 100.0 | 83.3 |
| dsvw | 21 | 5 | 6 | 77.8 | 78.4 |
| vampi | 11 | 4 | 4 | 73.3 | 73.3 |
| vc-claude-code-seeded-v3-legal-case-remix | 24 | 30 | 4 | 85.7 | 72.3 |
| react-security | 6 | 8 | 1 | 85.7 | 71.4 |
| dsvpwa | 24 | 19 | 8 | 75.0 | 70.2 |
| intentionally-vulnerable-python-application | 5 | 4 | 2 | 71.4 | 67.6 |
| damn-vulnerable-graphql-application | 24 | 14 | 12 | 66.7 | 65.9 |
| dvna | 12 | 11 | 5 | 70.6 | 65.9 |
| vulnerable-node | 15 | 15 | 6 | 71.4 | 65.8 |
| vc-claude-code-seeded-v3-logistics-dispatch-fastify-vue | 35 | 31 | 16 | 68.6 | 64.8 |
| vc-codex-high-seeded-v3-support-desk-nestjs-angular | 25 | 28 | 10 | 71.4 | 64.8 |
| vc-codex-high-seeded-v3-marketplace-commerce-express | 30 | 27 | 14 | 68.2 | 64.4 |
| vc-claude-code-seeded-v3-fintech-lending-express | 28 | 31 | 12 | 70.0 | 63.9 |
| vulnerable-app-angular | 6 | 9 | 2 | 75.0 | 63.8 |
| vc-claude-code-seeded-v3-support-desk-nestjs-angular | 29 | 35 | 12 | 70.7 | 63.6 |
| vc-claude-code-seeded-v3-hr-payroll-remix | 25 | 24 | 12 | 67.6 | 63.5 |
| python-app | 13 | 6 | 8 | 61.9 | 63.1 |
| node-api-goat | 5 | 7 | 2 | 71.4 | 62.5 |
| vc-codex-high-seeded-v3-legal-case-remix | 30 | 22 | 17 | 63.8 | 62.5 |
| vc-deepseek-v4-flash-seeded-v3-education-lms-nextjs | 20 | 29 | 8 | 71.4 | 62.1 |
| vc-kimi-code-seeded-v2-support-desk-fastapi | 18 | 15 | 10 | 64.3 | 62.1 |
| vc-claude-code-seeded-v3-property-management-fastify-vue | 23 | 35 | 9 | 71.9 | 61.8 |
| vc-codex-high-seeded-v3-healthcare-clinic-nextjs | 28 | 34 | 14 | 66.7 | 60.9 |
| vulnerable-tornado-app | 9 | 9 | 5 | 64.3 | 60.8 |
| vc-claude-code-seeded-v2-fintech-lending-fastapi | 17 | 7 | 12 | 58.6 | 60.7 |
| vc-codex-high-seeded-v3-crm-saas-nestjs-angular | 24 | 26 | 13 | 64.9 | 60.6 |
| vc-codex-high-seeded-v3-hr-payroll-remix | 27 | 29 | 15 | 64.3 | 60.3 |
| vc-kimi-code-seeded-v2-property-management-fastapi | 19 | 19 | 11 | 63.3 | 60.1 |
| extremely-vulnerable-flask-app | 18 | 4 | 14 | 56.2 | 60.0 |
| vc-deepseek-v4-pro-seeded-v3-healthcare-clinic-nextjs | 18 | 28 | 8 | 69.2 | 60.0 |
| vc-deepseek-v4-flash-seeded-v3-crm-saas-nestjs-angular | 16 | 31 | 6 | 72.7 | 59.3 |
| vc-claude-code-seeded-v3-education-lms-nextjs | 25 | 34 | 13 | 65.8 | 59.2 |
| nodejs-goof | 11 | 14 | 6 | 64.7 | 59.1 |
| vc-codex-high-seeded-v3-education-lms-nextjs | 29 | 25 | 19 | 60.4 | 58.9 |
| lets-be-bad-guys | 13 | 2 | 11 | 54.2 | 58.6 |
| vc-kimi-code-seeded-v2-marketplace-commerce-fastapi | 16 | 14 | 11 | 59.3 | 58.0 |
| vc-claude-code-seeded-v3-healthcare-clinic-nextjs | 27 | 37 | 16 | 62.8 | 57.2 |
| dvblab | 12 | 5 | 10 | 54.5 | 57.1 |
| ivna | 11 | 10 | 8 | 57.9 | 56.7 |
| vc-codex-seeded-v2-logistics-dispatch-fastapi | 16 | 5 | 14 | 53.3 | 56.7 |
| vc-codex-high-seeded-v3-fintech-lending-express | 23 | 25 | 16 | 59.0 | 56.4 |
| vulnerable-nodejs | 11 | 11 | 8 | 57.9 | 56.1 |
| vc-kimi-code-seeded-v3-legal-case-remix | 25 | 34 | 16 | 61.0 | 56.1 |
| owasp-web-playground | 15 | 7 | 13 | 53.6 | 56.0 |
| threatbyte | 14 | 7 | 12 | 53.8 | 56.0 |
| vc-codex-seeded-v2-property-management-fastapi | 17 | 12 | 14 | 54.8 | 55.6 |
| vc-codex-high-seeded-v3-property-management-fastify-vue | 24 | 25 | 18 | 57.1 | 55.3 |
| vc-kimi-code-seeded-v3-logistics-dispatch-fastify-vue | 27 | 41 | 17 | 61.4 | 55.3 |
| nodegoat | 16 | 18 | 12 | 57.1 | 54.8 |
| vulpy | 31 | 24 | 26 | 54.4 | 54.8 |
| vc-kimi-code-seeded-v2-legal-case-django | 16 | 26 | 10 | 61.5 | 54.8 |
| vc-kimi-code-seeded-v3-fintech-lending-express | 24 | 27 | 18 | 57.1 | 54.8 |
| vc-deepseek-v4-flash-seeded-v3-fintech-lending-express | 14 | 26 | 8 | 63.6 | 54.7 |
| vc-deepseek-v4-pro-seeded-v3-support-desk-nestjs-angular | 14 | 39 | 5 | 73.7 | 54.3 |
| vc-codex-high-seeded-v2-hr-payroll-django | 13 | 7 | 12 | 52.0 | 54.2 |
| python-insecure-app | 4 | 1 | 4 | 50.0 | 54.1 |
| vc-kimi-code-seeded-v2-crm-saas-django | 16 | 24 | 11 | 59.3 | 54.1 |
| vc-claude-code-seeded-v2-property-management-fastapi | 16 | 2 | 17 | 48.5 | 53.3 |
| vc-kimi-code-seeded-v3-education-lms-nextjs | 29 | 59 | 17 | 63.0 | 53.3 |
| vc-claude-code-seeded-v3-marketplace-commerce-express | 22 | 27 | 18 | 55.0 | 52.6 |
| vc-claude-code-seeded-v3-crm-saas-nestjs-angular | 26 | 26 | 23 | 53.1 | 52.4 |
| vulnerable-api | 7 | 4 | 7 | 50.0 | 52.2 |
| pygoat | 40 | 33 | 38 | 51.3 | 51.9 |
| vc-claude-code-seeded-v2-crm-saas-django | 14 | 9 | 14 | 50.0 | 51.9 |
| vc-deepseek-v4-pro-seeded-v3-legal-case-remix | 14 | 21 | 11 | 56.0 | 51.9 |
| vc-kimi-code-seeded-v3-support-desk-nestjs-angular | 18 | 28 | 14 | 56.2 | 51.7 |
| vulnerable-rest-api-owasp-2023 | 10 | 27 | 5 | 66.7 | 51.5 |
| vc-deepseek-v4-flash-seeded-v3-legal-case-remix | 14 | 34 | 8 | 63.6 | 51.5 |
| vc-kimi-code-seeded-v3-crm-saas-nestjs-angular | 24 | 50 | 16 | 60.0 | 51.3 |
| oss-oopssec-store | 23 | 43 | 17 | 57.5 | 50.9 |
| vc-deepseek-v4-pro-seeded-v3-property-management-fastify-vue | 13 | 39 | 6 | 68.4 | 50.8 |
| vc-claude-code-seeded-v2-support-desk-fastapi | 16 | 6 | 18 | 47.1 | 50.6 |
| vc-kimi-code-seeded-v2-logistics-dispatch-fastapi | 17 | 27 | 14 | 54.8 | 50.6 |
| vc-kimi-code-seeded-v3-hr-payroll-remix | 21 | 43 | 15 | 58.3 | 50.5 |
| vc-codex-high-seeded-v3-logistics-dispatch-fastify-vue | 24 | 34 | 21 | 53.3 | 50.4 |
| vulnpy | 36 | 6 | 44 | 45.0 | 49.7 |
| vc-codex-seeded-v2-crm-saas-django | 16 | 9 | 18 | 47.1 | 49.7 |
| vc-codex-seeded-v2-marketplace-commerce-fastapi | 14 | 11 | 15 | 48.3 | 49.6 |
| vc-codex-seeded-v2-support-desk-fastapi | 14 | 7 | 16 | 46.7 | 49.6 |
| dvpwa | 11 | 8 | 12 | 47.8 | 49.5 |
| vc-kimi-code-seeded-v2-fintech-lending-fastapi | 16 | 14 | 17 | 48.5 | 49.4 |
| vc-claude-code-seeded-v2-legal-case-django | 15 | 13 | 16 | 48.4 | 49.3 |
| vc-kimi-code-seeded-v3-property-management-fastify-vue | 20 | 56 | 12 | 62.5 | 49.0 |
| vc-claude-code-seeded-v2-hr-payroll-django | 12 | 3 | 15 | 44.4 | 48.8 |
| vc-claude-code-seeded-v2-logistics-dispatch-fastapi | 15 | 7 | 18 | 45.5 | 48.7 |
| vc-codex-high-seeded-v2-education-lms-django | 12 | 12 | 13 | 48.0 | 48.4 |
| vc-codex-seeded-v2-legal-case-django | 15 | 8 | 18 | 45.5 | 48.4 |
| vulnnodeapp | 14 | 15 | 15 | 48.3 | 48.3 |
| vc-deepseek-v4-pro-seeded-v3-education-lms-nextjs | 14 | 24 | 13 | 51.8 | 47.9 |
| vc-codex-seeded-v2-hr-payroll-django | 17 | 5 | 22 | 43.6 | 47.8 |
| vc-kimi-code-seeded-v2-healthcare-clinic-django | 15 | 22 | 15 | 50.0 | 47.8 |
| vc-kimi-code-seeded-v3-healthcare-clinic-nextjs | 18 | 28 | 18 | 50.0 | 47.4 |
| vc-deepseek-v4-pro-seeded-v3-marketplace-commerce-express | 13 | 22 | 13 | 50.0 | 46.8 |
| vc-codex-high-seeded-v2-property-management-fastapi | 12 | 13 | 14 | 46.2 | 46.5 |
| vc-claude-code-seeded-v2-education-lms-django | 14 | 10 | 18 | 43.8 | 46.1 |
| vc-codex-seeded-v2-healthcare-clinic-django | 17 | 5 | 24 | 41.5 | 45.7 |
| vc-deepseek-v4-flash-seeded-v3-marketplace-commerce-express | 11 | 26 | 10 | 52.4 | 45.5 |
| vc-deepseek-v4-pro-seeded-v3-fintech-lending-express | 14 | 32 | 13 | 51.8 | 45.5 |
| vc-kimi-code-seeded-v2-hr-payroll-django | 14 | 24 | 15 | 48.3 | 45.5 |
| vc-deepseek-v4-pro-seeded-v3-logistics-dispatch-fastify-vue | 15 | 31 | 15 | 50.0 | 45.2 |
| vulnerable-flask-app | 9 | 7 | 12 | 42.9 | 45.0 |
| vc-codex-high-seeded-v2-support-desk-fastapi | 12 | 10 | 16 | 42.9 | 44.8 |
| djangoat | 23 | 27 | 29 | 44.2 | 44.6 |
| vc-claude-code-seeded-v2-marketplace-commerce-fastapi | 13 | 6 | 19 | 40.6 | 44.2 |
| vc-deepseek-v4-flash-seeded-v3-hr-payroll-remix | 10 | 31 | 8 | 55.6 | 44.2 |
| vuln-node-express-swagger | 21 | 21 | 28 | 42.9 | 44.1 |
| vc-deepseek-v4-flash-seeded-v3-property-management-fastify-vue | 14 | 45 | 11 | 56.0 | 44.0 |
| vuln-node-express | 2 | 1 | 3 | 40.0 | 43.5 |
| vc-codex-seeded-v2-education-lms-django | 14 | 7 | 21 | 40.0 | 43.5 |
| vc-codex-high-seeded-v2-healthcare-clinic-django | 11 | 12 | 15 | 42.3 | 43.3 |
| vc-kimi-code-seeded-v2-education-lms-django | 12 | 18 | 16 | 42.9 | 42.3 |
| vc-codex-high-seeded-v2-crm-saas-django | 10 | 9 | 15 | 40.0 | 42.0 |
| vc-deepseek-v4-flash-seeded-v3-logistics-dispatch-fastify-vue | 13 | 31 | 15 | 46.4 | 41.7 |
| xvna | 6 | 19 | 6 | 50.0 | 41.1 |
| vc-kimi-code-seeded-v3-marketplace-commerce-express | 19 | 40 | 24 | 44.2 | 41.1 |
| vc-deepseek-v4-flash-seeded-v3-support-desk-nestjs-angular | 10 | 16 | 14 | 41.7 | 41.0 |
| vc-deepseek-v4-flash-seeded-v3-healthcare-clinic-nextjs | 10 | 29 | 14 | 41.7 | 37.0 |
| vc-codex-high-seeded-v2-marketplace-commerce-fastapi | 9 | 13 | 16 | 36.0 | 36.9 |
| vc-deepseek-v4-pro-seeded-v3-crm-saas-nestjs-angular | 12 | 31 | 18 | 40.0 | 36.8 |
| vc-codex-seeded-v2-fintech-lending-fastapi | 12 | 6 | 25 | 32.4 | 36.1 |
| dvws-node | 26 | 34 | 51 | 33.8 | 35.3 |
| ninjasworkout | 8 | 18 | 14 | 36.4 | 35.1 |
| vulnerable-react-fatih | 3 | 14 | 4 | 42.9 | 33.3 |
| vc-codex-high-seeded-v2-logistics-dispatch-fastapi | 9 | 10 | 20 | 31.0 | 33.3 |
| juice-shop-goof | 19 | 64 | 32 | 37.2 | 33.1 |
| vc-claude-code-seeded-v2-healthcare-clinic-django | 9 | 18 | 20 | 31.0 | 31.5 |
| flask-xss | 8 | 5 | 22 | 26.7 | 30.1 |
| damn-vulnerable-flask-application | 4 | 8 | 10 | 28.6 | 29.4 |
| juice-shop | 24 | 61 | 57 | 29.6 | 29.3 |
| vc-deepseek-v4-pro-seeded-v3-hr-payroll-remix | 8 | 27 | 18 | 30.8 | 28.8 |
| simply-vulnerable-react | 1 | 14 | 0 | 100.0 | 26.3 |
| vc-codex-high-seeded-v2-fintech-lending-fastapi | 7 | 12 | 22 | 24.1 | 25.9 |
| vulnerable-python-apps | 4 | 8 | 18 | 18.2 | 20.0 |
| vc-codex-high-seeded-v2-legal-case-django | 4 | 13 | 21 | 16.0 | 17.1 |
| reactvulna | 0 | 5 | 1 | 0.0 | 0.0 |
§
Detection by severity
| Severity | TP | FP | FN | Recall % |
|---|---|---|---|---|
| Critical | 135 | 0 | 23 | 85.4 |
| High | 1654 | 1 | 1194 | 58.1 |
| Medium | 367 | 3 | 599 | 38.0 |
| Low | 41 | 0 | 125 | 24.7 |
§
Detection by vulnerability class
| CWE family | TP | FP | FN | Recall % |
|---|---|---|---|---|
| Command / OS Injection | 126 | 0 | 10 | 92.6 |
| Insecure Deserialization | 43 | 0 | 5 | 89.6 |
| XML External Entities | 55 | 0 | 9 | 85.9 |
| Open Redirect | 73 | 0 | 15 | 83.0 |
| Path Traversal | 86 | 1 | 18 | 82.7 |
| Server-Side Request Forgery | 75 | 2 | 18 | 80.6 |
| SQL Injection | 116 | 0 | 38 | 75.3 |
| Code Injection / RFI | 60 | 0 | 20 | 75.0 |
| XPath Injection | 12 | 0 | 5 | 70.6 |
| Cross-Site Scripting | 139 | 0 | 67 | 67.5 |
| Broken Access Control / IDOR | 143 | 0 | 89 | 61.6 |
| Hardcoded Credentials | 111 | 1 | 105 | 51.4 |
| HTTP Header Injection | 1 | 0 | 1 | 50.0 |
| Missing Authentication / Authorization | 80 | 0 | 93 | 46.2 |
| Other | 841 | 0 | 1044 | 44.6 |
| Security Misconfiguration | 78 | 0 | 126 | 38.2 |
| Sensitive Data Exposure | 131 | 0 | 218 | 37.5 |
| Denial of Service | 27 | 0 | 60 | 31.0 |
§
LLM operational metrics
1,854,218
Avg input tokens
13,533
Avg output tokens
1,871,335
Avg total tokens
575s
Avg latency / repo
0.0%
JSON repair rate
140
Total runs
§
Cost
$1,041
Total cost
$7.43
Cost / run
$0.140
Cost / 100 LOC
741,034
Lines of code scanned
140
Successful runs