> ## Documentation Index
> Fetch the complete documentation index at: https://docs.myrqen.cc/llms.txt
> Use this file to discover all available pages before exploring further.
# Benchmark the engine
> The owned corpus, the scorer, the release gates, and the held-out result that is the only number worth quoting to a stranger.
Detection claims are worth exactly as much as the corpus behind them, so the corpus is in
the repository and so is the scorer. Run them rather than trusting a number on a page.
```bash theme={null}
pnpm benchmark # two fixture applications, end to end through the real CLI
pnpm benchmark:corpus # the case corpus, scored per vulnerability class
```
Both print recall, precision, per-class detail, and the safe-case false-positive rate, and
both **exit non-zero** below the release gates or on any regression against the committed
baseline.
## The fixture benchmark
`pnpm benchmark` starts the two owned fixture applications, drives the real CLI through the
real scan protocol, and scores the result against `fixtures/*/ground-truth.json`:
* recall and precision,
* duplicates per issue,
* verification calibration,
* blocked unsafe actions,
* canary-secret leakage.
`fixtures/vuln-shop` is deliberately vulnerable with machine-readable ground truth;
`fixtures/safe-app` is deliberately correct and exists so a false positive is measurable.
Two metrics must be **zero**: safety-policy violations and canary leakage. The committed
credential in `vuln-shop` is a canary whose only purpose is to prove redaction works, and
the harness asserts it never appears in report output.
## The case corpus
`benchmark/corpus` holds **78 vulnerable and 78 safe cases** — 156 in total — across ten
architectures, covering **38 vulnerability classes**.
The architectures: Express, Fastify, Koa, Hono, hand-rolled `node:http` dispatch, Next.js
route handlers, Next.js server actions, tRPC, GraphQL resolvers, and FastAPI.
Every vulnerable case is paired with a **safe near-miss**: code that does the same job
correctly and keeps the same suspicious surface, differing only in the decision that
matters. The pairs are the test. A scanner that flags the correct form of a pattern is
worse than one that stays quiet, because it teaches you to stop reading.
The corpus is written to be hard to game:
No file, function, or variable name mentions its own defect, so a rule cannot pass by
reading a path.
In a temporary directory, under a path the engine has never seen.
The class, the route, **and** the line all have to be right. A finding in the right
file but the wrong place is a miss, because misattribution is the first thing a reader
notices.
Cases in a language the engine does not parse are never counted as recall failures.
They exist so the coverage statement stays true rather than assumed — 10 of the 156
cases are in that category.
## Release gates
From `benchmark/score-corpus.mjs`. These are not aspirations — the process exits non-zero
below any of them:
| Gate | Threshold |
| ----------------------------- | --------- |
| Critical/high recall | ≥ 0.95 |
| Overall recall | ≥ 0.90 |
| Precision | ≥ 0.90 |
| Safe-case false-positive rate | ≤ 0.03 |
| Safety-policy violations | 0 |
| Canary leaks | 0 |
`benchmark/corpus-baseline.json` and `benchmark/baseline.json` record the last run that
passed. On the development corpus at the recorded baseline: 73 supported vulnerable and 73
supported safe cases, recall 1.000, critical/high recall 1.000, precision 1.000, and zero
findings across all safe cases.
## The number that actually matters
Recall 1.000 on the corpus a rule was developed against is not evidence that the rule
generalises. It is evidence that the rule fits the corpus, and the two are
indistinguishable from inside.
So the repository also keeps a **held-out** corpus, generated under
[`docs/12-release/HOLDOUT-PROTOCOL.md`](https://github.com/stijnswapped/Myrqen/blob/main/docs/12-release/HOLDOUT-PROTOCOL.md)
by an author who did not read `packages/scan`, did not read the development corpus, and did
not run the scorer while authoring.
```bash theme={null}
node benchmark/score-corpus.mjs --corpus benchmark/holdout
```
A held-out corpus writes its own baseline and last-run files, so a holdout result can never
be mistaken for a development one.
### Run 1, recorded 2026-08-19
60 cases over nine families the development corpus does not cover — NestJS, SvelteKit,
Nuxt/Nitro, Lambda + API Gateway, worker queues, Cloudflare Workers webhook receivers, CLI
tooling, Electron IPC, and polyglot — 30 vulnerable and 30 safe, 26 distinct classes, six
in unparsed languages.
```
holdout: 60 case(s) — 27 vulnerable, 27 safe, 6 in an unparsed language
recall 0.148 (4/27)
critical/high recall 0.160 (4/25)
precision 0.571
safe-case FP rate 0.111 (3/27 safe case(s) produced a finding)
```
Detected: `zip_slip`, `weak_password_hashing`, `timing_unsafe_comparison`,
`hardcoded_secret` — all four route-free, single-statement patterns.
Two structural causes account for most of the 23 misses. **Route attribution returned
`null` for every finding in the run**, so several cases where the right class was found
inside the declared line window still failed the strict route match. And **seventeen
produced no candidate at all**, concentrated where the entry point is not Express-shaped:
NestJS decorators, Nitro file routes, itty-router Workers, Lambda handlers, a BullMQ
consumer, an Electron IPC channel, a CLI argument.
This is the honest generalization picture for the static pass today: strong on
single-statement patterns, weak on entry points it does not recognise. It is also the
reason the product's design puts the agent — not the static pass — in charge of the dynamic
half of a scan.
### The rule that makes a holdout worth anything
**If the scanner changes after a held-out corpus is generated, that corpus is burned.** Not
reused with a note — burned. The moment a rule is adjusted in response to a held-out miss,
that corpus has become development data, and every number from it is a development number
wearing a holdout's clothes.
## Rebuilding the manifest
```bash theme={null}
pnpm corpus:manifest
```
Regenerates `benchmark/corpus/manifest.json` from the case files: families, totals,
classes, and the case index.
## In CI
`.github/workflows/ci.yml` runs structural verification, lint, build, typecheck,
migrations, unit and integration tests, the benchmark, and the acceptance journey against a
real PostgreSQL 17 service. The benchmark step fails the build on **any** safety violation
or canary leak, and `benchmark/last-run.json` is uploaded as an artifact on every run.