The fixture benchmark
pnpm benchmark starts the owned fixture applications, drives the real CLI through the
real scan protocol, and scores the result against fixtures/*/ground-truth.json:
- recall and precision,
- duplicates per issue,
- verification calibration,
- blocked unsafe actions,
- canary-secret leakage.
fixtures/vuln-shop is deliberately vulnerable with machine-readable ground truth;
fixtures/safe-app is deliberately correct and exists so a false positive is measurable.
fixtures/vuln-fastapi and fixtures/vuln-flask are the Python equivalents of
vuln-shop. They start through python3 and are skipped, with a note in the output,
when the packages in their requirements.txt are not installed. Myrqen itself never
needs Python.
Two metrics must be zero: safety-policy violations and canary leakage. The committed
credential in vuln-shop is a canary whose only purpose is to prove redaction works, and
the harness asserts it never appears in report output.
The case corpus
benchmark/corpus holds 78 vulnerable and 78 safe cases — 156 in total — across ten
architectures, covering 38 vulnerability classes.
The architectures: Express, Fastify, Koa, Hono, hand-rolled node:http dispatch, Next.js
route handlers, Next.js server actions, tRPC, GraphQL resolvers, and FastAPI.
Every vulnerable case is paired with a safe near-miss: code that does the same job
correctly and keeps the same suspicious surface, differing only in the decision that
matters. The pairs are the test. A scanner that flags the correct form of a pattern is
worse than one that stays quiet, because it teaches you to stop reading.
The corpus is written to be hard to game:
No filename tells the answer
No filename tells the answer
No file, function, or variable name mentions its own defect, so a rule cannot pass by
reading a path.
Each case is scanned alone
Each case is scanned alone
In a temporary directory, under a path the engine has never seen.
Matching is strict
Matching is strict
The class, the route, and the line all have to be right. A finding in the right
file but the wrong place is a miss, because misattribution is the first thing a reader
notices.
Unparsed languages are counted separately
Unparsed languages are counted separately
Cases in a language the engine does not parse are never counted as recall failures.
They exist so the coverage statement stays true rather than assumed — 10 of the 156
cases are in that category.
The Python corpora
Python has its own corpora, scored the same way, one case at a time:benchmark/corpus-python(28 vulnerable, 28 safe) is the development corpus the FastAPI and Flask rules were written against.benchmark/holdout-python(28 vulnerable, 30 safe) was written blind and scored once, then read on purpose and fixed against. It is development data now; its one blind score is recorded in the held-out protocol as Python Run 1.benchmark/holdout-python-2(31 vulnerable, 31 safe) was written blind while those fixes were made, and scored once afterwards: recall 0.774, precision 0.923, and 2 of 31 deliberately correct files flagged (Python Run 2). It was then read and fixed against in a second round, so it is development data too. Run 2 is still the Python figure to quote, as the most recent blind run rather than a measurement of the current engine; the next blind Python figure is pending a new held-out set. Its cases cover only the classes the Python engine was built for, so it is not comparable with the JavaScript held-out figure. See Scan a Python app.
Release gates
Frombenchmark/score-corpus.mjs. These are not aspirations — the process exits non-zero
below any of them:
benchmark/corpus-baseline.json and benchmark/baseline.json record the last run that
passed. On the development corpus at the recorded baseline: 73 supported vulnerable and 73
supported safe cases, recall 1.000, critical/high recall 1.000, precision 1.000, and zero
findings across all safe cases.
The number that actually matters
Recall 1.000 on the corpus a rule was developed against is not evidence that the rule generalises. It is evidence that the rule fits the corpus, and the two are indistinguishable from inside. So the repository also keeps a held-out corpus, generated underdocs/12-release/HOLDOUT-PROTOCOL.md
by an author who did not read packages/scan, did not read the development corpus, and did
not run the scorer while authoring.
Run 1, recorded 2026-08-19
60 cases over nine families the development corpus does not cover — NestJS, SvelteKit, Nuxt/Nitro, Lambda + API Gateway, worker queues, Cloudflare Workers webhook receivers, CLI tooling, Electron IPC, and polyglot — 30 vulnerable and 30 safe as authored, of which 27 and 27 are in languages the engine parses; 26 distinct classes.zip_slip, weak_password_hashing, timing_unsafe_comparison,
hardcoded_secret — all four route-free, single-statement patterns.
Two structural causes account for most of the 23 misses. Route attribution returned
null for every finding in the run, so several cases where the right class was found
inside the declared line window still failed the strict route match. And seventeen
produced no candidate at all, concentrated where the entry point is not Express-shaped:
NestJS decorators, Nitro file routes, itty-router Workers, Lambda handlers, a BullMQ
consumer, an Electron IPC channel, a CLI argument.
That was the generalization picture on 2026-08-19, and it is recorded here as history
rather than as a current claim: strong on single-statement patterns, weak on entry points
it does not recognise. Run 1’s diagnosis is what produced the framework entry-point
adapters, and it is why this corpus is now burned — see the rule below. For the current
figures, see the box at the top of this page.
Run 2, recorded 2026-08-20
Written blind by five authors, one architecture family each, after Run 1 was burned. 76 cases over django-drf, spring-boot, rails-api, go-chi and modern-js — chosen to avoid every architecture the engine already had an adapter for.modern-js, and a ratio out of eleven is a weak estimate —
quoted here with that caveat rather than without it.
Ten of the eleven misses produced no candidate at all. Not a wrong line, not a
wrong class, nothing. The reason was the same for every one: no entry point was
recognised, so no binding was caller-controlled, so every data-flow check in the
engine stayed silent.
That is the most useful sentence either of the first two runs produced:
The entry-point adapter layer works. But its coverage is strictly per-architecture, and an application built on a framework with no adapter is very nearly invisible to this engine.Burned deliberately, in the same session, to build adapters for the architectures it named.
Run 3, recorded 2026-08-21
Six families, 34 vulnerable and 34 safe cases, every one JavaScript or TypeScript. The application source was written by blind authors, one family each; the ground truth in one pass by a subagent whose prompt carried the corpus specification and nothing about the implementation.holdout-3 has since been
tuned against, so it is history rather than a current claim.
Run 3 established what Run 2 could not: the adapter layer had fixed the blindness —
all six architectures produced candidates — and the rules were now the limit. The
classes that produced nothing at all were the stateful and semantic ones: ownership,
atomicity, whether anyone is authenticated, whether a comparison is constant-time.
Those are not shapes an adapter reveals.
Burned the same day. Nine changes were made in response, taking the same corpus to
0.471 — a development number, reported beside the blind one and never instead of it.
Run 4, recorded 2026-08-24
Five families no previous corpus had used, and three of them — AdonisJS, Convex, Effect — frameworks the engine has no adapter for at all. 30 vulnerable and 30 safe cases, written in parallel by five blind authors.0.033, is what the scorer returned.
Class-blind recall, 0.333, counts a case as seen when any finding lands inside
the expected lines whatever it is called — a weaker question than recall asks, and not
comparable to the 0.294 above.
The gap between them is the specification defect. What the run does establish is the
other twenty cases: two thirds of the files produced nothing at all, nearly all of
them reporting no recognised entry point. That is Run 2’s conclusion reproduced on
five fresh frameworks, and it is why holdout-4 has not been burned — a corpus of
architectures the engine cannot see is exactly the right thing to measure adapter work
against.
Nothing from Run 4 is published outside this page and the protocol document. Deciding
what the honest public sentence is has not been done, and inventing one here would be
the opposite of the point.
The rule that makes a holdout worth anything
If the scanner changes after a held-out corpus is generated, that corpus is burned. Not reused with a note — burned. The moment a rule is adjusted in response to a held-out miss, that corpus has become development data, and every number from it is a development number wearing a holdout’s clothes.Rebuilding the manifest
benchmark/corpus/manifest.json from the case files: families, totals,
classes, and the case index.
In CI
.github/workflows/ci.yml runs structural verification, lint, build, typecheck,
migrations, unit and integration tests, the benchmark, and the acceptance journey against a
real PostgreSQL 17 service. The benchmark step fails the build on any safety violation
or canary leak, and benchmark/last-run.json is uploaded as an artifact on every run.