We measure our own miss rate.
222 real vulnerabilities from the GitHub Advisory Database. Assay scanned each project's code as it existed before the fix. The headline is the blind number: recall on the 164 CVEs no rule was learned from. Below: what it caught, what it missed, and how the number moves as the system learns from its misses. Most benchmarks hide the misses. The misses are the roadmap.
Last run 2026-07-20 · re-runs automatically every Monday · raw results TSV
The blind-holdout number, run by run
Every benchmark run appends a row to a public TSV. Each point below is a real run, on the holdout slice only; the labels are what changed between them. The holdout itself grows as the corpus expands — a bigger denominator moves the number down without the detectors getting worse.
Three ways to count a catch. We lead with the strictest.
A finding only counts as a true catch if it lands on the code the fix later changed, and stops firing once the fix is applied. Looser definitions inflate the number; we show all three — computed on the 164 blind holdout CVEs.
A finding inside the fix-changed region, from a detector that goes quiet after the fix. It flagged the actual vulnerability.
A finding inside the fix-changed region, but the detector still fires on the patched code too. Right place, weaker signal.
A finding anywhere in a changed file. This is the number most tools would report. We don't lead with it.
The fine print, before you ask
- Taught CVEs are labeled, not hidden. 58 of the 222 CVEs fed the rule catalog, so they can't count as blind catches. On those, recall is 58.6%. The headline 32.9% is the 164 untouched holdout CVEs only.
- Generalization: demonstrated, but small. Rules learned from one CVE have caught 5 holdout CVEs they were never taught from:
- CVE-2026-9495 — Access-control flaw in @koa/router (CWE-284), via turn-2 rule learned from a different advisory
- CVE-2026-48054 — Code injection (CWE-94), via turn-2 rule learned from a different advisory
- GHSA-72r4-9c5j-mj57 — pnpm patch-removal path traversal (CWE-22), via turn-1 rule learned from a different advisory
- GHSA-534h-c3cw-v3h9 — Nuxt vite-node world-connectable IPC socket (CWE-276), via three corroborating turn-1 rules
- CVE-2026-59731 — Authorization via non-canonical URL paths (CWE-647), via turn-1 rule learned from a different advisory
- Noisy detectors don't get credit. A detector firing more than 20 times in one file is dropped for that file. Without this cap the benchmark reports a meaningless 100%. We know because our first version did.
What we miss today
Holdout misses from the latest run. This list is the work queue for the next teaching cycle.
What the loop learned last cycle
- Denial of service via unbounded recursion in nested-object parsing and property traversal
- Prototype pollution through user-controlled keys in nested object assignment
- Authentication bypass when polluted prototypes shadow built-in object methods
- Memory exhaustion from unvalidated request-body size limits in HTTP handling
- Credential injection via unsanitized configuration objects passed to HTTP clients
- Resource exhaustion via attacker-controlled deep nesting in form serialization
Each rule was extracted from a confirmed CVE fix, then validated before promotion. 375 candidates failed validation and were discarded.
How it works
- 1.Pull recent security advisories with linked fix commits from the GitHub Advisory Database (npm and pip).
- 2.For each advisory, fetch every changed file twice: as it existed before the fix (vulnerable) and after (patched).
- 3.Scan the vulnerable version with deterministic detectors only — pattern rules and learned rules, no LLM at scan time.
- 4.Score a catch only when a finding overlaps the code the fix changed, and the detector goes quiet on the patched version.
- 5.Feed the misses back: harvest their fix diffs into new candidate rules, validate, promote survivors, re-run.
Methodology & limitations
- Dataset
- GitHub Advisory Database — npm and pip advisories with linked fix commits
- Corpus
- 222 CVEs scored (npm + pip, up to 500 discovered per ecosystem)
- Split
- 164 blind holdout · 58 taught (rules were learned from their fixes)
- Detectors
- 437 promoted learned rules + claimless checks, deterministic only — no LLM at scan time
- Run date
- 2026-07-20
- Rule admission
- Candidate rules must fire on their source vulnerability and pass synthetic precision/recall validation before promotion; 375 of 422 candidates were rejected last cycle
- Noise cap
- A detector firing more than 20 times in one file is dropped for that file — its region hits carry no localization signal
- JavaScript/TypeScript and Python files only
- A catch requires line-region overlap with the fix; a correct finding elsewhere in the repo counts as a miss
- Holdout CVEs predate the rules (retrospective). The prospective test — advisories published after the rules were learned — is the weekly Monday run
- Numbers are internal, not independently validated
Raw data: results-recall.tsv — one row per run, appended automatically.
Run the same detectors on your own repo.
npx tryassay assess .