Scanner deep-dive
GLM-5.3 by Z.ai ↗
General-Purpose LLM · agentic-v1 · scored on 136/140 repositories. Strict scoring (unfinished repos counted as misses).
Methodology note
136/140; strict scoring. GLM-5.3 through the standard agentic-v1 harness on Z.ai's API. All 66 Python repositories at F3 63.3; 70 of 74 TypeScript / JS. The four unfinished TS/JS repositories count as misses, giving F3 47.4 on TS/JS and 54.6 Overall (51.3 and 56.9 scored on completed repositories only) — fourth Overall, the best open-weight result in the benchmark. The retained raw output shows the four failures were the harness, not the model: in three the headless agent auto-rejected a permission prompt (reading a .env file, or a scratch write to /tmp) and ended the session before any findings were written; in one the model hit its output-length limit mid-answer. That permission behaviour is fixed as of 3.1.0 (see the DeepSeek V4.1 Flash note); the four repositories will be re-run under the fixed harness in a future release. Total spend $105.73, including one retry pass.54.6
F3 (strict)
53.3
F2 (strict)
55.9%
Recall (strict)
44.9%
Precision
136/140
Repos scored
glm-5.3
Model
$106
Total cost
519s
Avg latency
§
Per-repository breakdown
Each bar shows true positives, false positives, and misses on one repository; bar length is proportional to that repo's labeled vulnerabilities. Ranked by F2.
| Repository | TP | FP | FN | Recall % | F2 |
|---|---|---|---|---|---|
| vuln-node-express | 5 | 1 | 0 | 100.0 | 96.2 |
| vfapi | 9 | 2 | 0 | 100.0 | 95.7 |
| pythonssti | 2 | 1 | 0 | 100.0 | 90.9 |
| dsvw | 23 | 5 | 4 | 85.2 | 84.6 |
| react-test-bench | 1 | 1 | 0 | 100.0 | 83.3 |
| dvblab | 18 | 9 | 4 | 81.8 | 78.3 |
| dsvpwa | 26 | 13 | 6 | 81.2 | 77.8 |
| vc-codex-high-seeded-v2-crm-saas-django | 19 | 4 | 6 | 76.0 | 77.2 |
| owasp-web-playground | 25 | 27 | 3 | 89.3 | 76.2 |
| node-api-goat | 6 | 6 | 1 | 85.7 | 75.0 |
| python-app | 16 | 7 | 5 | 76.2 | 74.8 |
| vc-kimi-code-seeded-v2-marketplace-commerce-fastapi | 22 | 17 | 5 | 81.5 | 74.8 |
| vampi | 12 | 10 | 3 | 80.0 | 73.2 |
| vc-codex-high-seeded-v2-hr-payroll-django | 18 | 6 | 7 | 72.0 | 72.6 |
| vulnerable-tornado-app | 10 | 4 | 4 | 71.4 | 71.4 |
| vc-codex-high-seeded-v2-marketplace-commerce-fastapi | 18 | 8 | 7 | 72.0 | 71.4 |
| vc-codex-high-seeded-v2-property-management-fastapi | 20 | 18 | 6 | 76.9 | 70.4 |
| insecure-web | 7 | 7 | 2 | 77.8 | 70.0 |
| lets-be-bad-guys | 17 | 9 | 7 | 70.8 | 69.7 |
| intentionally-vulnerable-python-application | 5 | 3 | 2 | 71.4 | 69.4 |
| vulnerable-api | 10 | 6 | 4 | 71.4 | 69.4 |
| vc-codex-seeded-v2-logistics-dispatch-fastapi | 21 | 11 | 9 | 70.0 | 69.1 |
| vc-kimi-code-seeded-v2-property-management-fastapi | 23 | 24 | 7 | 76.7 | 68.9 |
| dvna | 12 | 8 | 5 | 70.6 | 68.2 |
| vc-claude-code-seeded-v2-property-management-fastapi | 21 | 2 | 12 | 63.6 | 67.7 |
| vulnerable-flask-app | 15 | 12 | 6 | 71.4 | 67.6 |
| vc-codex-high-seeded-v2-support-desk-fastapi | 18 | 4 | 10 | 64.3 | 67.2 |
| vulnerable-node | 15 | 13 | 6 | 71.4 | 67.0 |
| vc-codex-high-seeded-v2-healthcare-clinic-django | 18 | 13 | 8 | 69.2 | 66.7 |
| vc-kimi-code-seeded-v2-education-lms-django | 20 | 18 | 8 | 71.4 | 66.7 |
| vc-kimi-code-seeded-v2-support-desk-fastapi | 19 | 12 | 9 | 67.9 | 66.4 |
| threatbyte | 17 | 8 | 9 | 65.4 | 65.9 |
| vc-kimi-code-seeded-v2-logistics-dispatch-fastapi | 25 | 41 | 6 | 80.7 | 65.8 |
| vc-codex-high-seeded-v2-education-lms-django | 18 | 19 | 7 | 72.0 | 65.7 |
| vulnpy | 52 | 25 | 28 | 65.0 | 65.5 |
| vc-kimi-code-seeded-v2-healthcare-clinic-django | 21 | 21 | 9 | 70.0 | 64.8 |
| vc-kimi-code-seeded-v2-legal-case-django | 18 | 17 | 8 | 69.2 | 64.7 |
| vc-claude-code-seeded-v3-fintech-lending-express | 28 | 29 | 12 | 70.0 | 64.5 |
| vc-codex-seeded-v2-legal-case-django | 21 | 11 | 12 | 63.6 | 64.0 |
| vc-codex-high-seeded-v2-logistics-dispatch-fastapi | 19 | 15 | 10 | 65.5 | 63.3 |
| damn-vulnerable-graphql-application | 25 | 30 | 11 | 69.4 | 62.8 |
| vc-kimi-code-seeded-v2-hr-payroll-django | 19 | 17 | 10 | 65.5 | 62.5 |
| vc-kimi-code-seeded-v2-fintech-lending-fastapi | 21 | 16 | 12 | 63.6 | 62.1 |
| vc-claude-code-seeded-v2-fintech-lending-fastapi | 18 | 12 | 11 | 62.1 | 61.6 |
| vc-claude-code-seeded-v2-education-lms-django | 19 | 8 | 13 | 59.4 | 61.3 |
| vc-claude-code-seeded-v2-marketplace-commerce-fastapi | 19 | 8 | 13 | 59.4 | 61.3 |
| vc-kimi-code-seeded-v2-crm-saas-django | 17 | 14 | 10 | 63.0 | 61.2 |
| vc-deepseek-v4-flash-seeded-v3-legal-case-remix | 15 | 20 | 7 | 68.2 | 61.0 |
| extremely-vulnerable-flask-app | 19 | 9 | 13 | 59.4 | 60.9 |
| vc-claude-code-seeded-v2-support-desk-fastapi | 20 | 9 | 14 | 58.8 | 60.6 |
| vc-claude-code-seeded-v2-logistics-dispatch-fastapi | 19 | 6 | 14 | 57.6 | 60.5 |
| vc-codex-seeded-v2-hr-payroll-django | 23 | 11 | 16 | 59.0 | 60.5 |
| vc-claude-code-seeded-v2-crm-saas-django | 17 | 12 | 11 | 60.7 | 60.3 |
| vc-codex-seeded-v2-crm-saas-django | 21 | 17 | 13 | 61.8 | 60.3 |
| vc-codex-seeded-v2-education-lms-django | 21 | 14 | 14 | 60.0 | 60.0 |
| vulnerable-python-apps | 13 | 8 | 9 | 59.1 | 59.6 |
| vc-codex-high-seeded-v2-fintech-lending-fastapi | 19 | 25 | 10 | 65.5 | 59.4 |
| vc-codex-seeded-v2-property-management-fastapi | 18 | 10 | 13 | 58.1 | 59.2 |
| vc-kimi-code-seeded-v3-fintech-lending-express | 27 | 33 | 15 | 64.3 | 59.2 |
| vc-codex-seeded-v2-support-desk-fastapi | 17 | 7 | 13 | 56.7 | 59.0 |
| dvpwa | 13 | 6 | 10 | 56.5 | 58.6 |
| pygoat | 45 | 27 | 33 | 57.7 | 58.6 |
| vc-claude-code-seeded-v2-legal-case-django | 17 | 5 | 14 | 54.8 | 58.2 |
| vc-claude-code-seeded-v3-property-management-fastify-vue | 22 | 39 | 10 | 68.8 | 58.2 |
| vc-codex-high-seeded-v3-support-desk-nestjs-angular | 23 | 36 | 12 | 65.7 | 57.8 |
| vc-codex-high-seeded-v3-legal-case-remix | 27 | 20 | 20 | 57.5 | 57.4 |
| vc-codex-seeded-v2-healthcare-clinic-django | 23 | 15 | 18 | 56.1 | 56.9 |
| vc-claude-code-seeded-v3-support-desk-nestjs-angular | 26 | 39 | 15 | 63.4 | 56.8 |
| vc-kimi-code-seeded-v3-legal-case-remix | 27 | 48 | 14 | 65.8 | 56.5 |
| vc-codex-high-seeded-v3-education-lms-nextjs | 31 | 52 | 17 | 64.6 | 56.4 |
| nodejs-goof | 11 | 19 | 6 | 64.7 | 56.1 |
| vc-claude-code-seeded-v2-healthcare-clinic-django | 16 | 12 | 13 | 55.2 | 55.6 |
| djangoat | 28 | 17 | 24 | 53.8 | 55.3 |
| vc-claude-code-seeded-v3-logistics-dispatch-fastify-vue | 29 | 29 | 22 | 56.9 | 55.3 |
| vc-claude-code-seeded-v2-hr-payroll-django | 14 | 5 | 13 | 51.8 | 55.1 |
| vc-codex-high-seeded-v3-healthcare-clinic-nextjs | 25 | 35 | 17 | 59.5 | 54.8 |
| ivna | 11 | 14 | 8 | 57.9 | 54.5 |
| vc-claude-code-seeded-v3-legal-case-remix | 17 | 27 | 11 | 60.7 | 54.5 |
| vc-claude-code-seeded-v3-education-lms-nextjs | 24 | 45 | 14 | 63.2 | 54.3 |
| vc-deepseek-v4-flash-seeded-v3-education-lms-nextjs | 16 | 20 | 12 | 57.1 | 54.1 |
| vc-kimi-code-seeded-v3-logistics-dispatch-fastify-vue | 26 | 39 | 18 | 59.1 | 53.9 |
| vc-codex-seeded-v2-marketplace-commerce-fastapi | 16 | 17 | 13 | 55.2 | 53.7 |
| vc-claude-code-seeded-v3-hr-payroll-remix | 21 | 28 | 16 | 56.8 | 53.3 |
| vc-codex-high-seeded-v3-hr-payroll-remix | 24 | 33 | 18 | 57.1 | 53.3 |
| vc-deepseek-v4-flash-seeded-v3-healthcare-clinic-nextjs | 14 | 22 | 10 | 58.3 | 53.0 |
| vc-kimi-code-seeded-v3-hr-payroll-remix | 20 | 25 | 16 | 55.6 | 52.9 |
| vulpy | 29 | 22 | 28 | 50.9 | 52.0 |
| vc-claude-code-seeded-v3-marketplace-commerce-express | 21 | 21 | 19 | 52.5 | 52.0 |
| flask-xss | 14 | 3 | 16 | 46.7 | 51.1 |
| vulnerable-app-angular | 5 | 12 | 3 | 62.5 | 51.0 |
| vc-kimi-code-seeded-v3-property-management-fastify-vue | 19 | 40 | 13 | 59.4 | 50.8 |
| vc-codex-high-seeded-v3-property-management-fastify-vue | 22 | 27 | 20 | 52.4 | 50.7 |
| vc-codex-high-seeded-v3-crm-saas-nestjs-angular | 19 | 21 | 18 | 51.3 | 50.5 |
| nextjs-vulnerable-app | 1 | 5 | 0 | 100.0 | 50.0 |
| vc-deepseek-v4-flash-seeded-v3-property-management-fastify-vue | 15 | 35 | 10 | 60.0 | 50.0 |
| vc-claude-code-seeded-v3-crm-saas-nestjs-angular | 24 | 22 | 25 | 49.0 | 49.6 |
| vc-kimi-code-seeded-v3-marketplace-commerce-express | 23 | 39 | 20 | 53.5 | 49.1 |
| python-insecure-app | 4 | 5 | 4 | 50.0 | 48.8 |
| vulnerable-rest-api-owasp-2023 | 12 | 51 | 3 | 80.0 | 48.8 |
| vc-codex-high-seeded-v3-fintech-lending-express | 20 | 30 | 19 | 51.3 | 48.5 |
| ninjasworkout | 13 | 35 | 9 | 59.1 | 47.8 |
| vc-codex-high-seeded-v3-logistics-dispatch-fastify-vue | 22 | 28 | 23 | 48.9 | 47.8 |
| vc-deepseek-v4-flash-seeded-v3-support-desk-nestjs-angular | 13 | 27 | 11 | 54.2 | 47.8 |
| vc-deepseek-v4-pro-seeded-v3-property-management-fastify-vue | 13 | 48 | 6 | 68.4 | 47.4 |
| vc-kimi-code-seeded-v3-healthcare-clinic-nextjs | 19 | 38 | 17 | 52.8 | 47.3 |
| vc-deepseek-v4-pro-seeded-v3-logistics-dispatch-fastify-vue | 15 | 25 | 15 | 50.0 | 46.9 |
| vc-deepseek-v4-pro-seeded-v3-marketplace-commerce-express | 12 | 13 | 14 | 46.2 | 46.5 |
| vuln-node-express-swagger | 24 | 39 | 25 | 49.0 | 46.3 |
| nodegoat | 13 | 16 | 15 | 46.4 | 46.1 |
| vulnnodeapp | 14 | 22 | 15 | 48.3 | 46.1 |
| oss-oopssec-store | 19 | 32 | 21 | 47.5 | 45.0 |
| vc-codex-seeded-v2-fintech-lending-fastapi | 16 | 14 | 21 | 43.2 | 44.9 |
| vulnerable-nodejs | 9 | 16 | 10 | 47.4 | 44.6 |
| vc-kimi-code-seeded-v3-support-desk-nestjs-angular | 17 | 48 | 15 | 53.1 | 44.0 |
| vc-deepseek-v4-pro-seeded-v3-education-lms-nextjs | 13 | 27 | 14 | 48.1 | 43.9 |
| vc-deepseek-v4-flash-seeded-v3-crm-saas-nestjs-angular | 14 | 59 | 8 | 63.6 | 43.5 |
| vc-deepseek-v4-flash-seeded-v3-hr-payroll-remix | 9 | 23 | 9 | 50.0 | 43.3 |
| vc-kimi-code-seeded-v3-crm-saas-nestjs-angular | 19 | 45 | 21 | 47.5 | 42.4 |
| xvna | 7 | 29 | 5 | 58.3 | 41.7 |
| vc-deepseek-v4-flash-seeded-v3-fintech-lending-express | 11 | 34 | 11 | 50.0 | 41.4 |
| vc-deepseek-v4-pro-seeded-v3-healthcare-clinic-nextjs | 13 | 40 | 13 | 50.0 | 41.4 |
| vc-deepseek-v4-pro-seeded-v3-legal-case-remix | 12 | 33 | 13 | 48.0 | 41.4 |
| vc-claude-code-seeded-v3-healthcare-clinic-nextjs | 19 | 40 | 24 | 44.2 | 41.1 |
| vc-deepseek-v4-flash-seeded-v3-marketplace-commerce-express | 10 | 30 | 11 | 47.6 | 40.3 |
| react-security | 4 | 18 | 3 | 57.1 | 40.0 |
| vc-deepseek-v4-flash-seeded-v3-logistics-dispatch-fastify-vue | 11 | 19 | 17 | 39.3 | 38.7 |
| vc-deepseek-v4-pro-seeded-v3-support-desk-nestjs-angular | 9 | 33 | 10 | 47.4 | 38.1 |
| juice-shop-goof | 18 | 45 | 33 | 35.3 | 33.7 |
| vc-deepseek-v4-pro-seeded-v3-hr-payroll-remix | 10 | 35 | 16 | 38.5 | 33.6 |
| juice-shop | 23 | 40 | 58 | 28.4 | 29.7 |
| vc-deepseek-v4-pro-seeded-v3-crm-saas-nestjs-angular | 10 | 41 | 20 | 33.3 | 29.2 |
| damn-vulnerable-flask-application | 4 | 9 | 10 | 28.6 | 29.0 |
| vc-codex-high-seeded-v2-legal-case-django | 6 | 21 | 19 | 24.0 | 23.6 |
| simply-vulnerable-react | 1 | 17 | 0 | 100.0 | 22.7 |
| vulnerable-react-fatih | 2 | 32 | 5 | 28.6 | 16.1 |
| reactvulna | 0 | 11 | 1 | 0.0 | 0.0 |
§
Detection by severity
| Severity | TP | FP | FN | Recall % |
|---|---|---|---|---|
| Critical | 146 | 2 | 10 | 93.6 |
| High | 1581 | 0 | 1101 | 58.9 |
| Medium | 531 | 6 | 431 | 55.2 |
| Low | 57 | 0 | 87 | 39.6 |
§
Detection by vulnerability class
| CWE family | TP | FP | FN | Recall % |
|---|---|---|---|---|
| Insecure Deserialization | 47 | 0 | 0 | 100.0 |
| HTTP Header Injection | 2 | 0 | 0 | 100.0 |
| XPath Injection | 13 | 0 | 0 | 100.0 |
| Command / OS Injection | 124 | 0 | 5 | 96.1 |
| XML External Entities | 56 | 0 | 4 | 93.3 |
| Path Traversal | 91 | 0 | 7 | 92.9 |
| Code Injection / RFI | 72 | 0 | 6 | 92.3 |
| SQL Injection | 134 | 0 | 15 | 89.9 |
| Open Redirect | 73 | 0 | 12 | 85.9 |
| Server-Side Request Forgery | 74 | 2 | 15 | 83.1 |
| Security Misconfiguration | 144 | 0 | 54 | 72.7 |
| Cross-Site Scripting | 135 | 0 | 65 | 67.5 |
| Denial of Service | 46 | 0 | 38 | 54.8 |
| Hardcoded Credentials | 109 | 1 | 94 | 53.7 |
| Missing Authentication / Authorization | 86 | 3 | 76 | 53.1 |
| Sensitive Data Exposure | 156 | 0 | 159 | 49.5 |
| Broken Access Control / IDOR | 104 | 0 | 117 | 47.1 |
| Other | 849 | 2 | 962 | 46.9 |
§
LLM operational metrics
102,958
Avg input tokens
35,387
Avg output tokens
1,460,633
Avg total tokens
519s
Avg latency / repo
0.0%
JSON repair rate
140
Total runs
§
Cost
$106
Total cost
$0.78
Cost / run
$0.015
Cost / 100 LOC
713,729
Lines of code scanned
136
Successful runs