How the VerifyMCP scanner benchmark works

VerifyMCP publishes numbers about other people’s software. A scanner that has never been measured against anything is asking to be trusted on its own say-so, which is exactly what we tell people not to do with MCP servers.

This page describes the benchmark we are building to measure it, and the rules that govern what may be published from it.

What the benchmark measures

One thing: for a server whose defects a human has established, does VerifyMCP report the findings that human expected, and does it report anything they did not.

A case names reason codes, not scores. The score is a weighted aggregate of the findings, so measuring the aggregate would tell you that something went wrong without telling you what. A finding is a specific claim about a specific server, and it is the unit that can be right or wrong.

What it deliberately does not measure

Whether the weights are correct. That a destructive tool exposed to anonymous callers costs more of the total than an absent HSTS header is a judgement about what matters. It can be argued with, and it is argued with in the rubric, but no corpus settles it. A benchmark can only measure whether the finding fired.
Whether a server is safe to use. A case says a specific defect is present or absent. A server with no findings against it is a server we found nothing wrong with in the checks the case scoped, which is a much smaller statement.
How VerifyMCP compares to other scanners. Comparing tools fairly needs a corpus both parties accept, and ours is not one. Other people have built corpora and measured scanners against them: the MCPZoo study (Chen et al., July 2026) assembled 64,611 unique MCP servers, ran eight scanners over them, and found on manual validation that fewer than half the sampled alerts were true positives. Running our own corpus against another tool would measure how closely that tool agrees with our reason codes, and we wrote the reason codes.
Anything about the two tracks jointly. Remote endpoints and packages are scored on different rubrics over different evidence. Results are reported per track and never pooled.

How cases are selected, and by whom

Selection is the easiest place to cheat in a benchmark and the hardest thing to detect from outside, so the rules are written down and the reasoning for each case is committed with it.

Synthetic fixtures are written to cover a specific check. One fixture, one flaw, and a recorded statement of what would change the answer. They are proposed by whoever is working on the check and reviewed by someone else, because a fixture written by reading the implementation documents the implementation rather than the requirement.

Real servers are selected before they are scored, never after. A server that VerifyMCP has already scored and that produced an interesting-looking result is the worst possible candidate: the reviewer knows the answer they are meant to reach. Each case records in its provenance how that particular server came to be reviewed, so a reader can judge the selection for themselves rather than taking a claim of impartiality on trust.

Every case names its reviewer, and the reviewer is a person rather than a team or a tool. Whoever it names is willing to be named in anything published from the case.

How ground truth is established

The two kinds of case have different evidentiary strength, and the difference is not smoothed over.

For a synthetic fixture, ground truth is not in dispute. We built the flaw, so we know it is there, and we know what else is not. The cost is that a fixture only exercises the checks that read a captured MCP contract. It has no domain, no certificate and no dependency tree, so it can say nothing about the endpoint, infrastructure and supply-chain checks.

For a real server, ground truth is one named person’s reading of somebody else’s software, recorded in terms a stranger can check. “The tool is dangerous” is not ground truth. “The tool declares destructiveHint: true, the handshake is open, and an unauthenticated call reached tool lookup” is, because every clause of it can be verified against the artefact.

Results from the two are reported separately and never merged into one figure. Averaging certain ground truth with contested ground truth produces a number that means neither.

False positives and false negatives, per reason code

Each case declares a scope: the reason-code prefixes it asserts ground truth over. Within that scope, and only within it:

a true positive is a code the reviewer expected and the engine emitted;
a false positive is a code the engine emitted that no reviewer expected;
a false negative is a code the reviewer expected and the engine did not emit.

Findings outside the scope are reported and counted in nothing. Without that rule a synthetic fixture would be impossible. The DNSSEC result for a server with no domain is a fact about the test harness, and counting it against the scanner would report a corpus defect as a product defect.

There is no false-positive rate, because there is no true-negative population. A case records the findings a reviewer expects; it does not record the hundreds of codes they considered and rejected. What the corpus supports is a false discovery rate, false positives over all in-scope findings, and that is what will be published, under that name. A denominator nobody counted is not made real by dividing by it.

And matching is by code, not by value. A measurement code such as tool description coverage fires whether coverage is 12% or 98%. A case expecting it asserts that the measurement was made, not that it came out well. Expressing an expected value is a later version of the case format, and until it exists, no result should be read as evidence about measurement accuracy.

What reproducible means here, and where it stops

The fixture path is reproducible in full. A fixture is a static capture, the remote checks are pure functions of it, and the harness runs them with no network access at all. The same fixture under the same rubric version produces the same reason codes on any machine, and if it ever does not, that is a bug in VerifyMCP and this is how you would find it.

The harness only scores remote-track fixtures. It refuses a registry target, and it refuses a package-track case, because the package checks fetch their own supply-chain data from third parties rather than reading a capture, so there is nothing a synthetic file could stand in for. A manually reviewed case is scored by the production pipeline against the pinned version, and the codes that pipeline emits are what get diffed against the manifest. The harness is not a second implementation of the scanner and must not become one.

The manually reviewed path is not reproducible, and cannot be made so. Scoring a published server means observing the world at a moment, and these parts of the world do not hold still.

A remote endpoint can change its tools, its authorization posture, its certificate or its protocol version between two runs, and can be down during one of them. Pinning a version pins what the registry says, not what the endpoint serves.
OSV publishes advisories continuously, so a package with no known CVE today can have one tomorrow with nothing about the package having changed.
Socket’s malware verdicts are theirs, are revised, and need an API key. A run without one reports the malware signal as unverified, which is a different result from a clean verdict and must not be compared against one.
GitHub rate limits apply per token, so repository reachability can resolve differently for a third party running the same corpus at the same moment.
DNSSEC validation depends on the resolver path from wherever the run happens.
The AI-judged signal is an LLM call. It is pinned to a versioned rubric and cached by schema hash, which makes it stable in practice for a given schema, but it is not a deterministic function and we will not claim it is.
The package track runs the published artefact in a container, where install resolution, native dependencies and timing all vary.

So a third party will not reproduce a manually reviewed result byte for byte, and should not expect to. What they can do is run the same corpus, publish what they got, and compare the disagreements. Those are where either the scanner or the ground truth is wrong, and both are worth knowing about. Any result published from a manually reviewed case carries the run date, the rubric version, and which third-party signals were available during the run.

Rubric versions

Every published result carries the rubric version it was produced under, or it is not published.

Scores and findings move when the rubric moves, and it moves several times a year: a category is reweighted, a signal is split, a spec version is pinned. A case reviewed under one rubric version and scored under a later one is a different measurement. The human judgement in it may still hold, but nobody has checked that it does, so the harness reports the mismatch on every run and the case is re-reviewed before anything is published from it.

This applies to comparisons over time as well. Two results under different rubric versions are not a trend line, and we will not draw one.

Disputing a case

A case is a human judgement and can be wrong. Ours included.

Anyone can dispute one, and a server author disputing a case about their own server does not need to establish standing first. A dispute names the case id and says which specific claim is wrong and why, in terms of the artefact.

The case’s reviewer answers it in the open. Where they agree the case is wrong, it is corrected or withdrawn, and any published result computed from it is republished without it, with a note saying so. Where they disagree, the disagreement is recorded on the case and the case stays, because a corpus that quietly drops every contested case is a corpus of things nobody objected to.

A case about a real server is withdrawn on request while a dispute is open. The asymmetry is deliberate: a wrong published finding about a named person’s software does them real harm, and a benchmark result is not worth that.

Why nothing has been published

Because three fixtures are not a corpus.

Producing a figure now would mean choosing the servers, grading them, running our own scanner over them and publishing our own mark. That is not a benchmark, and the credibility it would borrow is the thing a benchmark is supposed to earn. So the corpus, the harness and these rules are published first, and in this order on purpose. The method can be argued with before there are any results to defend.

The corpus and the harness live in the benchmark/ directory of the repository, along with the README each directory carries saying what belongs in it.