Skip to main content
Detection claims are worth exactly as much as the corpus behind them, so the corpus is in the repository and so is the scorer. Run them rather than trusting a number on a page.
Seven held-out runs have been recorded and Run 1 below is not the current result. The current figures come from Run 7, scored 2026-08-25 on benchmark/holdout-7 — 32 vulnerable and 32 deliberately-correct cases across five frameworks and five data layers no earlier corpus had used, written in parallel by authors who had never read the engine, against a build frozen at a recorded source hash and verified unchanged at scoring time. For the static pass alone:
Read the intervals, not the point estimates — including when comparing our own runs. Every figure above is a proportion over 32 vulnerable and 32 deliberately-correct cases, so each carries a 95% interval roughly 0.3 wide. On a sample this size a difference below about 0.346 is not measurable at all.That is not a caveat, it is the main thing to know about this table. Three runs are directly comparable — they score the figure this project named before any of the corpora existed — and they read 0.300, 0.333, 0.250. Their intervals all overlap. Earliest against latest is p = 0.66.The false-positive rate over the same three runs reads 0.267, 0.069, 0.125. Earliest against latest is p = 0.16. Also not distinguishable.So the honest summary of every held-out measurement this project has taken is that none of the changes between runs is distinguishable from noise at this sample size. Detecting a genuine five-point improvement would need roughly 1,400 cases; ten points needs 330. The corpora hold thirty.Two things follow, and we would rather say them than have a reader work them out. First, earlier write-ups of these runs described a declining trend and a generalised precision win, and neither is supported — the comparisons are now fixed in advance (earliest comparable run against latest) and computed in scripts/generate-benchmark-summary.mjs rather than chosen after the fact, because with three runs there are three pairs and one of them does reach p < 0.05. Second, the ruler is the bottleneck rather than the engine, and fixing it comes before any further detection work. The corpora and the scorer are both in the repository; the scorer now prints the interval beside every figure it produces.
Run 7 is lower than the run before it, and that is the point of running it. Run 6 read a higher figure, recorded in the run history below. Work done between the two runs raised the development corpora it was measured against and a fresh corpus declined to reproduce the gain. Two different corpora are two samples and not a trend, so neither number is evidence of a direction — but no claim that detection improved is supported, and none is made.What Run 7 established. Splitting its cases by what the dangerous value reaches explains the whole result: 0.875 (7 of 8) where the sink is a language or runtime primitive — fetch, path/fs, child_process, eval — and 0.042 (1 of 24) where the sink is framework, store or identity vocabulary. Server-side request forgery and path traversal generalise to an unfamiliar stack because fetch and path.join are the same everywhere. A SQL injection reached through db.rawQuery, em.getConnection().execute or strapi.db.connection.raw does not, and three were missed on a corpus with SQL in it. Coverage of a framework’s own vocabulary is the binding constraint, not the quality of the rules.Two recall figures, both defined before any corpus they were applied to existed. A case’s author states the line the defect is on and a range around it, deliberately widened where a reviewer could reasonably disagree about whether the defect belongs on the composition or on the sink. The stricter figure requires a finding to agree with the case’s own CWE and land within eight lines of the line the author named; the looser one allows it anywhere in the marked range. On Run 7 the two agree at 0.250, which they are not obliged to do — on Run 5 the difference was not academic, because three findings credited as detections turned out to be false positives sitting further down the same handler. The stricter figure is the one to judge the scanner by.Why precision is absent rather than reported. Precision here is computed from exact matches on the defect’s class name, and thirty independent authors named the thirty defects in their own vocabulary — so the ratio is zero by construction and says nothing about the engine. The false-positive rate on the deliberately-correct half of the corpus is the precision signal that survives, and it is the row above.holdout-6 has since been tuned against, so anything the same corpus reports today is a development number, by the rule stated further down this page. The full history of all six runs is in docs/12-release/HOLDOUT-PROTOCOL.md, and the live numbers are generated from the scoring artifacts at myrqen.cc/benchmarks.Run 4, scored 2026-08-24, is recorded below and is not in that table. Its specification had a defect that made its recall figure measure class naming rather than detection, so quoting it either way would mislead; the run and the defect are both written up in full further down.
Both print recall, precision, per-class detail, and the safe-case false-positive rate, and both exit non-zero below the release gates or on any regression against the committed baseline.

The fixture benchmark

pnpm benchmark starts the owned fixture applications, drives the real CLI through the real scan protocol, and scores the result against fixtures/*/ground-truth.json:
  • recall and precision,
  • duplicates per issue,
  • verification calibration,
  • blocked unsafe actions,
  • canary-secret leakage.
fixtures/vuln-shop is deliberately vulnerable with machine-readable ground truth; fixtures/safe-app is deliberately correct and exists so a false positive is measurable. fixtures/vuln-fastapi and fixtures/vuln-flask are the Python equivalents of vuln-shop. They start through python3 and are skipped, with a note in the output, when the packages in their requirements.txt are not installed. Myrqen itself never needs Python. Two metrics must be zero: safety-policy violations and canary leakage. The committed credential in vuln-shop is a canary whose only purpose is to prove redaction works, and the harness asserts it never appears in report output.

The case corpus

benchmark/corpus holds 78 vulnerable and 78 safe cases — 156 in total — across ten architectures, covering 38 vulnerability classes. The architectures: Express, Fastify, Koa, Hono, hand-rolled node:http dispatch, Next.js route handlers, Next.js server actions, tRPC, GraphQL resolvers, and FastAPI. Every vulnerable case is paired with a safe near-miss: code that does the same job correctly and keeps the same suspicious surface, differing only in the decision that matters. The pairs are the test. A scanner that flags the correct form of a pattern is worse than one that stays quiet, because it teaches you to stop reading. The corpus is written to be hard to game:
No file, function, or variable name mentions its own defect, so a rule cannot pass by reading a path.
In a temporary directory, under a path the engine has never seen.
The class, the route, and the line all have to be right. A finding in the right file but the wrong place is a miss, because misattribution is the first thing a reader notices.
Cases in a language the engine does not parse are never counted as recall failures. They exist so the coverage statement stays true rather than assumed — 10 of the 156 cases are in that category.

The Python corpora

Python has its own corpora, scored the same way, one case at a time:
  • benchmark/corpus-python (28 vulnerable, 28 safe) is the development corpus the FastAPI and Flask rules were written against.
  • benchmark/holdout-python (28 vulnerable, 30 safe) was written blind and scored once, then read on purpose and fixed against. It is development data now; its one blind score is recorded in the held-out protocol as Python Run 1.
  • benchmark/holdout-python-2 (31 vulnerable, 31 safe) was written blind while those fixes were made, and scored once afterwards: recall 0.774, precision 0.923, and 2 of 31 deliberately correct files flagged (Python Run 2). It was then read and fixed against in a second round, so it is development data too. Run 2 is still the Python figure to quote, as the most recent blind run rather than a measurement of the current engine; the next blind Python figure is pending a new held-out set. Its cases cover only the classes the Python engine was built for, so it is not comparable with the JavaScript held-out figure. See Scan a Python app.

Release gates

From benchmark/score-corpus.mjs. These are not aspirations — the process exits non-zero below any of them: benchmark/corpus-baseline.json and benchmark/baseline.json record the last run that passed. On the development corpus at the recorded baseline: 73 supported vulnerable and 73 supported safe cases, recall 1.000, critical/high recall 1.000, precision 1.000, and zero findings across all safe cases.

The number that actually matters

Recall 1.000 on the corpus a rule was developed against is not evidence that the rule generalises. It is evidence that the rule fits the corpus, and the two are indistinguishable from inside. So the repository also keeps a held-out corpus, generated under docs/12-release/HOLDOUT-PROTOCOL.md by an author who did not read packages/scan, did not read the development corpus, and did not run the scorer while authoring.
A held-out corpus writes its own baseline and last-run files, so a holdout result can never be mistaken for a development one.

Run 1, recorded 2026-08-19

60 cases over nine families the development corpus does not cover — NestJS, SvelteKit, Nuxt/Nitro, Lambda + API Gateway, worker queues, Cloudflare Workers webhook receivers, CLI tooling, Electron IPC, and polyglot — 30 vulnerable and 30 safe as authored, of which 27 and 27 are in languages the engine parses; 26 distinct classes.
Detected: zip_slip, weak_password_hashing, timing_unsafe_comparison, hardcoded_secret — all four route-free, single-statement patterns. Two structural causes account for most of the 23 misses. Route attribution returned null for every finding in the run, so several cases where the right class was found inside the declared line window still failed the strict route match. And seventeen produced no candidate at all, concentrated where the entry point is not Express-shaped: NestJS decorators, Nitro file routes, itty-router Workers, Lambda handlers, a BullMQ consumer, an Electron IPC channel, a CLI argument.
That was the generalization picture on 2026-08-19, and it is recorded here as history rather than as a current claim: strong on single-statement patterns, weak on entry points it does not recognise. Run 1’s diagnosis is what produced the framework entry-point adapters, and it is why this corpus is now burned — see the rule below. For the current figures, see the box at the top of this page.

Run 2, recorded 2026-08-20

Written blind by five authors, one architecture family each, after Run 1 was burned. 76 cases over django-drf, spring-boot, rails-api, go-chi and modern-js — chosen to avoid every architecture the engine already had an adapter for.
54 of the 76 cases are Python, Java, Ruby or Go, which the engine does not parse and which are excluded rather than counted as misses. That leaves a scored population of eleven, almost all from modern-js, and a ratio out of eleven is a weak estimate — quoted here with that caveat rather than without it. Ten of the eleven misses produced no candidate at all. Not a wrong line, not a wrong class, nothing. The reason was the same for every one: no entry point was recognised, so no binding was caller-controlled, so every data-flow check in the engine stayed silent. That is the most useful sentence either of the first two runs produced:
The entry-point adapter layer works. But its coverage is strictly per-architecture, and an application built on a framework with no adapter is very nearly invisible to this engine.
Burned deliberately, in the same session, to build adapters for the architectures it named.

Run 3, recorded 2026-08-21

Six families, 34 vulnerable and 34 safe cases, every one JavaScript or TypeScript. The application source was written by blind authors, one family each; the ground truth in one pass by a subagent whose prompt carried the corpus specification and nothing about the implementation.
All four gates failed. This was the figure quoted across this site and the README until Run 6 replaced it on 2026-08-25 — it was the first number in this project’s history that could be described as a held-out result, and holdout-3 has since been tuned against, so it is history rather than a current claim. Run 3 established what Run 2 could not: the adapter layer had fixed the blindness — all six architectures produced candidates — and the rules were now the limit. The classes that produced nothing at all were the stateful and semantic ones: ownership, atomicity, whether anyone is authenticated, whether a comparison is constant-time. Those are not shapes an adapter reveals. Burned the same day. Nine changes were made in response, taking the same corpus to 0.471 — a development number, reported beside the blind one and never instead of it.

Run 4, recorded 2026-08-24

Five families no previous corpus had used, and three of them — AdonisJS, Convex, Effect — frameworks the engine has no adapter for at all. 30 vulnerable and 30 safe cases, written in parallel by five blind authors.
This run measured the wrong thing, and the fault is in its specification rather than in the engine. Its authors were told to name the defect they saw and were given no list of categories, on the reasoning that a class the engine has no rule for is the most useful thing a holdout can contain. The scorer matches a finding to a case by exact equality on the class name. Five independent authors produced thirty distinct names, and the engine cannot match any of them however well it does. So the run is recorded with two figures. Recall, 0.033, is what the scorer returned. Class-blind recall, 0.333, counts a case as seen when any finding lands inside the expected lines whatever it is called — a weaker question than recall asks, and not comparable to the 0.294 above. The gap between them is the specification defect. What the run does establish is the other twenty cases: two thirds of the files produced nothing at all, nearly all of them reporting no recognised entry point. That is Run 2’s conclusion reproduced on five fresh frameworks, and it is why holdout-4 has not been burned — a corpus of architectures the engine cannot see is exactly the right thing to measure adapter work against. Nothing from Run 4 is published outside this page and the protocol document. Deciding what the honest public sentence is has not been done, and inventing one here would be the opposite of the point.

The rule that makes a holdout worth anything

If the scanner changes after a held-out corpus is generated, that corpus is burned. Not reused with a note — burned. The moment a rule is adjusted in response to a held-out miss, that corpus has become development data, and every number from it is a development number wearing a holdout’s clothes.

Rebuilding the manifest

Regenerates benchmark/corpus/manifest.json from the case files: families, totals, classes, and the case index.

In CI

.github/workflows/ci.yml runs structural verification, lint, build, typecheck, migrations, unit and integration tests, the benchmark, and the acceptance journey against a real PostgreSQL 17 service. The benchmark step fails the build on any safety violation or canary leak, and benchmark/last-run.json is uploaded as an artifact on every run.