Skip to main content
Detection claims are worth exactly as much as the corpus behind them, so the corpus is in the repository and so is the scorer. Run them rather than trusting a number on a page.
Both print recall, precision, per-class detail, and the safe-case false-positive rate, and both exit non-zero below the release gates or on any regression against the committed baseline.

​
The fixture benchmark

pnpm benchmark starts the two owned fixture applications, drives the real CLI through the real scan protocol, and scores the result against fixtures/*/ground-truth.json:
  • recall and precision,
  • duplicates per issue,
  • verification calibration,
  • blocked unsafe actions,
  • canary-secret leakage.
fixtures/vuln-shop is deliberately vulnerable with machine-readable ground truth; fixtures/safe-app is deliberately correct and exists so a false positive is measurable. Two metrics must be zero: safety-policy violations and canary leakage. The committed credential in vuln-shop is a canary whose only purpose is to prove redaction works, and the harness asserts it never appears in report output.

​
The case corpus

benchmark/corpus holds 78 vulnerable and 78 safe cases — 156 in total — across ten architectures, covering 38 vulnerability classes. The architectures: Express, Fastify, Koa, Hono, hand-rolled node:http dispatch, Next.js route handlers, Next.js server actions, tRPC, GraphQL resolvers, and FastAPI. Every vulnerable case is paired with a safe near-miss: code that does the same job correctly and keeps the same suspicious surface, differing only in the decision that matters. The pairs are the test. A scanner that flags the correct form of a pattern is worse than one that stays quiet, because it teaches you to stop reading. The corpus is written to be hard to game:
No file, function, or variable name mentions its own defect, so a rule cannot pass by reading a path.
In a temporary directory, under a path the engine has never seen.
The class, the route, and the line all have to be right. A finding in the right file but the wrong place is a miss, because misattribution is the first thing a reader notices.
Cases in a language the engine does not parse are never counted as recall failures. They exist so the coverage statement stays true rather than assumed — 10 of the 156 cases are in that category.

​
Release gates

From benchmark/score-corpus.mjs. These are not aspirations — the process exits non-zero below any of them: benchmark/corpus-baseline.json and benchmark/baseline.json record the last run that passed. On the development corpus at the recorded baseline: 73 supported vulnerable and 73 supported safe cases, recall 1.000, critical/high recall 1.000, precision 1.000, and zero findings across all safe cases.

​
The number that actually matters

Recall 1.000 on the corpus a rule was developed against is not evidence that the rule generalises. It is evidence that the rule fits the corpus, and the two are indistinguishable from inside. So the repository also keeps a held-out corpus, generated under docs/12-release/HOLDOUT-PROTOCOL.md by an author who did not read packages/scan, did not read the development corpus, and did not run the scorer while authoring.
A held-out corpus writes its own baseline and last-run files, so a holdout result can never be mistaken for a development one.

​
Run 1, recorded 2026-08-19

60 cases over nine families the development corpus does not cover — NestJS, SvelteKit, Nuxt/Nitro, Lambda + API Gateway, worker queues, Cloudflare Workers webhook receivers, CLI tooling, Electron IPC, and polyglot — 30 vulnerable and 30 safe, 26 distinct classes, six in unparsed languages.
Detected: zip_slip, weak_password_hashing, timing_unsafe_comparison, hardcoded_secret — all four route-free, single-statement patterns. Two structural causes account for most of the 23 misses. Route attribution returned null for every finding in the run, so several cases where the right class was found inside the declared line window still failed the strict route match. And seventeen produced no candidate at all, concentrated where the entry point is not Express-shaped: NestJS decorators, Nitro file routes, itty-router Workers, Lambda handlers, a BullMQ consumer, an Electron IPC channel, a CLI argument.
This is the honest generalization picture for the static pass today: strong on single-statement patterns, weak on entry points it does not recognise. It is also the reason the product’s design puts the agent — not the static pass — in charge of the dynamic half of a scan.

​
The rule that makes a holdout worth anything

If the scanner changes after a held-out corpus is generated, that corpus is burned. Not reused with a note — burned. The moment a rule is adjusted in response to a held-out miss, that corpus has become development data, and every number from it is a development number wearing a holdout’s clothes.

​
Rebuilding the manifest

Regenerates benchmark/corpus/manifest.json from the case files: families, totals, classes, and the case index.

​
In CI

.github/workflows/ci.yml runs structural verification, lint, build, typecheck, migrations, unit and integration tests, the benchmark, and the acceptance journey against a real PostgreSQL 17 service. The benchmark step fails the build on any safety violation or canary leak, and benchmark/last-run.json is uploaded as an artifact on every run.