Scoring changelog: what has changed in how we score
Scores move for two reasons. Usually a server changed. Occasionally we changed, because we added a check, corrected one, or reweighted the categories. This page records the second kind, in date order, so any movement you cannot explain from your own release history can be explained from ours.
We version each rubric internally so that a stored score always names the rules it was computed under. Those identifiers are an implementation detail and are not published; this page is the public record instead.
26 September 2026: fixes across remote endpoints and every package ecosystem
No category was added and no weight changed. Instead we corrected a set of checks on remote endpoints and on every package ecosystem that were reading the right data the wrong way, or not reading it at all. As with the previous batch, they ship as one change, so each moved component gets one changelog entry rather than several.
Authorisation on remote endpoints. We now treat an endpoint as open only when it answers our anonymous request with a success. A 404, 405, 429 or server error says nothing about authorisation, so it is now inconclusive rather than “no authorisation required”. A success reached with a credential we supplied is inconclusive too, since it proves nothing about an anonymous caller. A tool that credibly declares itself read-only is no longer treated as destructive, unless its name or description says otherwise.
Reaching remote endpoints the way their clients do. Servers that answer only on the older SSE transport are now probed with the request they actually answer, so their authorisation, TLS and plain-HTTP checks measure the server rather than a refusal. Headers a registry entry declares with a fixed or default value are now sent to that server, and only to that server; secret ones never are. If a server requires a header we cannot supply and rejects us with a 400 or 422, that reads as needing credentials rather than as a fault. A URL that is a template, such as one with a tenant or region in it, is now filled in from the value or default the registry entry declares for each variable, so those servers are probed and can rise. Only a template with a variable nobody has filled in is not probed, and its transport reads as not yet verified.
Stability on remote endpoints. A dropped authorisation requirement is charged only when our latest capture proves the endpoint is now open, and a transport change only when it is a real downgrade. Moving from SSE to streamable HTTP, or a passing failure on the old transport, is no longer charged.
npm. A version its publisher has deprecated now scores zero for maintenance,
however recently it was published. A deprecation with an empty message is not
one, as npm itself treats it. Licences written as common names, such as “MIT
License”, now resolve, and “SEE LICENSE IN” or a URL reads as a licence we could
not check rather than a non-OSI one. Repository links in shorthand forms such as
github:owner/repo now resolve, so the repository and provenance checks can read
them. Dependency trees include peer dependencies, as npm installs them, and use
the version npm would pick: the latest tag when it fits, and a version that is
not deprecated over one that is. A registry record with an unusual field no
longer fails to read, and packages whose command is a script with a shebang now
start in our sandbox.
NuGet. Dependencies are graded at the lowest version a range allows, which is the version restore actually installs, rather than the highest. For many packages that can lower their dependency and vulnerability results. Packages published only under SemVer 2.0 versions are now found, where before they could read as unpublished. Vulnerabilities are looked up under the package id’s registered spelling, so a dependency written in a different letter case no longer hides its advisories. An unlisted version’s placeholder publish date is no longer read as a real release date. The install-script scan of MSBuild files now reads encodings and spellings it used to miss, and a build file we cannot read in full, or a task that loads code we have not reviewed, is reported as unreviewed.
Bundles (MCPB). A bundle published as a GitHub release asset, with a GitHub
build-provenance attestation over its exact bytes, now earns
build provenance. When a bundle’s registry
entry declares a SHA-256, we check the bundle against it, and do not grade a
file that does not match. A bundle whose archive would unpack differently depending on
the tool that unpacks it is not read, and is reported as inconclusive rather than
guessed at. uv bundles are graded on the versions their uv.lock pins, and
bundled node_modules folders and requirements files are read more accurately.
When a bundle’s release URL changes between versions, its stability history now
carries over instead of starting again, and more bundles start in our sandbox.
Container images. A signed SBOM, produced in CI from the image’s declared repository and naming the exact image digest, now counts for the vulnerability check. Language packages it lists are looked up, and operating-system packages only for Linux distribution releases the advisory database covers. An unsigned SBOM still does not count. Build provenance is now also found where cosign stores it, and across referrer listings that run to more than one page. Each image is now one component per repository rather than a new one for every release tag, so its stability history starts again from this change and then builds up normally.
Python (PyPI). Build provenance on PyPI is verified for the first time. A
release earns it only when every one of its files carries a verified attestation
(SLSA provenance or PyPI’s publish attestation) from the declared repository. On
PyPI the publish attestation counts as well as SLSA provenance, because it is
signed by the Trusted Publisher’s CI identity in the declared repository and
states that this exact file was published from that repository’s workflow. A
release with only some files attested does not earn it. A release with more than
16 files is never verified, and is inconclusive unless we find a file without an
attestation. Install scripts are now graded: a wheel-only release
with nothing that runs at start-up is conclusively clean, while a source release
is reported as unreviewed when, among other cases, it has a setup.py, an
in-tree build backend, build hooks, a backend such as scikit-build-core or
meson-python that runs the project’s own build scripts, metadata computed at
build time that we cannot prove static, a shipped .git or .hg folder, or a
pyproject.toml we cannot parse. A licence given only as a classifier that is not
OSI-approved is reported as a licence we could not check, rather than as no
licence. Dependency trees follow Python’s own rules for version specifiers,
environment markers and package names, and packages published only as source can
now start in our sandbox.
Every package ecosystem. A vulnerability now counts as fixable only when the advisory lists a fix for the package we asked about, not for another package it also covers. Our sandbox no longer passes a package argument that its registry entry marks as secret.
Tool safety, on both tracks. Tools that differ only in their digits, such
as query_db1 and query_db2 with the same description, no longer leave a
component reading as only partly checked for manipulation. The
tool safety guide describes exactly what counts as
manipulation.
What will move. On remote endpoints, servers that answered our authorisation probe with an error, speak only SSE, declare headers, or have a templated URL we can now fill in move. On packages, the largest movers are PyPI releases, which can now earn provenance and are graded for install scripts, NuGet packages whose dependencies now resolve lower, bundles with a GitHub attestation, and container images with a signed SBOM. Some components move down: container images whose stability history restarts, and npm versions whose publisher has deprecated them, which now score zero for maintenance. The package rubric and remote rubric describe each check as it now stands.
24 September 2026: a batch of measurement fixes on both tracks
No category was added and no weight changed. Instead we corrected a set of checks that were reading the right data the wrong way, or not reading it at all. Each fix is small on its own, but together they move a lot of scores, so they ship as one change and each moved component gets one changelog entry rather than several.
Build provenance on packages. Only a signed SLSA provenance statement now counts as build provenance, and it has to name the source repository in the certificate’s own repository field. A plain signature over something else, such as a cosign image signature, proves who pushed an artefact rather than how it was built, and no longer earns the credit. Container images are scored for provenance for the first time: the attestation was being found and verified, then discarded. We now check the multi-platform index the image was published under as well as its linux/amd64 image, since publishers attest one or the other. On NuGet, the signature nuget.org adds to every package is read as the repository’s signature, not the author’s, so it no longer counts as author-signed. That changes the finding’s wording on almost every NuGet package, but not the score.
Licences on PyPI and NuGet. We now read the licence the way publishers declare it: PyPI’s PEP 639 licence expression first, then the licence field, then the OSI-approved trove classifiers. A NuGet package that declares its licence only as a URL or a packed file is inconclusive rather than “no licence”.
Maintenance on container images. Reproducible builds stamp a fixed creation date (usually 1 January 1970) so that the image digest does not depend on when it was built. We were reading that as a release from decades ago. It is now treated as an unknown date.
Dependencies. The packages listed in a container image’s SBOM carry no registry facts, such as deprecation or a source repository, so dependency health is now inconclusive for them instead of charging every one of them with having no repository. A bundle (MCPB) is now graded on the dependency versions it actually ships rather than the newest versions its ranges allow, which can move it in either direction. An SBOM we cannot parse no longer hides a usable one attached after it.
Install scripts in bundles. A bundle whose install surface we could not read
(a Python, binary or uv bundle, or one with no readable package.json) used to get
full credit for having no install scripts. It is now inconclusive.
More packages get a tool list. Bundles now start the way their host starts them, from the launch configuration in their manifest, and bundles that ship a native binary run too. Packages are launched with the environment variables and container arguments they declare in the registry, and PyPI packages with the extras they declare. Packages that could not start in our sandbox before now produce their first capture, which moves their schema, coverage, stability and tool safety categories.
Tool listings on remote endpoints. A server whose tool listing failed, or was cut short, is no longer scored as though we had read all of its tools, so it cannot raise its tool coverage or tool safety by listing only part of what it exposes. A server that does not declare the tools capability is still asked for its tools, and only a clear “no such method”, an empty list or a refused request counts as having none. Stability only compares captures in which we read the tool list in full, on both tracks.
What will move. On packages, container images move the most: provenance can now be earned, and SBOM dependency health goes to inconclusive. Python and binary bundles lose the install-script credit they were given by mistake, and packages that start in our sandbox for the first time gain their capture categories. On remote endpoints, only servers whose tool listing we could not read in full move. The package rubric and remote rubric describe each check as it now stands.
29 August 2026: a new category for what your tool text tells the model
The shared core gained a fifth category, tool safety, worth 6% of every score on both tracks. It reads the text your server hands a host model, meaning the instructions returned on connect, the name, title and description of every tool, and the description of every parameter, and it asks what that text tells the model to do.
That is a different question from the categories either side of it. Tool coverage asks whether the text is there. Schema quality asks whether it reads well. A server can be documented completely and read beautifully while its descriptions quietly instruct the model to ignore its operator, and until this change nothing in the score said so.
Three signals. Half the category’s weight sits on a scan for injection markers: a fixed list of phrases and characters that address the model reading them rather than the person choosing a tool. A quarter sits on destructive annotations, comparing what a tool’s name and leading description plainly imply against the MCP annotations it declares. The last quarter is the judged reading that was already happening, described below.
Every other shared category paid for it. Schema quality and AI usability fell from 20% to 18%, stability and change management from 15% to 14%, tool coverage from 10% to 8%, and capabilities from 5% to 4%. Nothing was taken from either track’s own half: endpoint security is still 40% and transport 10% on a remote endpoint, supply chain still 30% and provenance still 20% on a package. The shared core still comes to half of every score, and both rubrics still add up to 100.
The weight is small deliberately. Two of the three signals are fixed text scans that prefer precision to recall. They are built to stay quiet unless they are fairly sure, and to be worth listening to when they are not. A category built that way should say something specific and cost a few points; it should not decide a server’s grade on a pattern match. Tool safety never caps a score.
Attempted manipulation is now reported here as well, and it costs a little more. Since 9 August a server that writes text aimed at the scorer rather than at an agent has scored zero for schema quality’s judged signal. That zero was indistinguishable from a server that had simply written no descriptions, which are opposite findings sharing one number. Tool safety does not take that penalty over, it adds a second one: a flagged component still scores zero inside schema quality and now also scores zero for this category’s judged reading. In practice that is roughly a point and a half on top of what a flagged server already lost. The judgement itself is unchanged, the same reading by the same model, so no new server becomes flagged by this release. What changes is that the finding is now stated plainly instead of hiding inside a low schema-quality number.
As before, describing an attack is not performing one. A security or red-team server that documents injection techniques is scanned on the same fixed list as everything else, and a phrase quoted inside your own text does not count against you.
What will move. Most components will shift by a point or two from the
re-weighting alone, in either direction depending on where they were already
strong. The real movement is on the annotation gap: the MCP spec treats an absent
destructiveHint as true, and most servers today declare nothing at all, so a
server with tools that delete, publish, transfer or execute will lose part of
this category until it declares them. That is a fixable gap, and the fix is an
annotation rather than a rewrite. The
tool safety guide covers every finding.
The checks landed in our code on 23 August 2026 and reach published scores on 29 August 2026.
9 August 2026: instruction quality is judged tool by tool
Schema quality includes one judgement of how clear and actionable a server’s descriptions are. Until 9 August 2026 that was a single verdict on the whole server, read in one go. From that date each tool is judged on its own, along with the server instructions and the resource list, and the results are averaged.
Every tool now counts for the same amount. Under the old method a server with forty good tools and five undescribed ones was read as one body of text, and what the five cost it depended on how they struck the reader. Now each tool carries an equal share of the result, so a poorly described tool costs a predictable amount rather than whatever an overall impression happened to give it. On a large server that share is still small, because the average is taken over every tool; what has changed is that it is consistent, and it no longer depends on how the rest of the server reads. Servers with a few poorly described tools among many will generally score slightly lower; small, uniformly documented servers will barely move.
Tools are not weighted by how much they have written. A long description is not worth more than a short one, and a terse but complete description is not penalised for being short.
Attempted manipulation now scores zero, and you will see it for the first time. Some servers embed text aimed at the scorer rather than at an agent, such as an instruction to ignore the rubric or to assign a particular score. That has scored zero in our code since 8 August 2026, but existing judgements were not recomputed at the time, so it reaches published scores on 9 August 2026. Describing an attack is not performing one: a security or red-team server that documents injection techniques is judged normally, on the same rubric as everything else. Where a server does try it, the whole component scores zero for this signal, not just the offending tool.
The model that reads the descriptions has changed. This signal is the one place in the score where a language model makes the judgement, so the model is part of the ruler, and replacing it moves the number by itself. We changed it in the same release as the two changes above, deliberately, so that this signal moved once rather than three times in a week. The rubric it is asked to apply is word for word the one described here, and we re-judged every server against it rather than mixing old and new verdicts. Expect small movements in either direction on servers whose descriptions sat near a rubric boundary.
Nothing else moved. No category was reweighted, and no check gained or lost a signal. Only the instruction-quality number is measured differently, so its share of the total is unchanged.
One practical note, since it is the reason this landed when it did: judging a server as a single block meant that when a hosting platform edited one sentence in a boilerplate tool it ships with every server it hosts, we re-judged every one of those servers in full. Judging tool by tool means shared text is judged once.
31 July 2026: one shared core for both tracks
The largest change we have made. Both rubrics were rebuilt around a shared core: the four categories that measure the MCP contract itself (schema quality, stability, tool coverage and capabilities) are now scored by the same checks, at the same weights, whether the component is a package or a remote endpoint. That core is worth half of every score. The other half is what only that kind of component can be judged on.
Every weight changed. On remote endpoints, endpoint security rose from 25% to 40%, schema quality fell from 25% to 20%, stability from 20% to 15%, tool coverage from 15% to 10%, and capabilities from 15% to 5%.
Transport modernity also moved out of capabilities into its own category, transport and reachability, worth 10%. It measures the endpoint rather than the MCP contract, and it was the one signal a package could never earn, because a package’s transport is always stdio.
No remote check gained, lost or changed a signal, so every remote score moved purely by re-weighting.
Packages changed more. They gained the two shared categories they did not previously have, capabilities and stability, while supply chain fell from 36% to 30% and provenance from 24% to 20%. Capabilities cost nothing to add, because it is read from the sandbox capture we already take. Stability did not: it needs two verified captures to compare, and until this change nothing reliably kept the older one. So almost every package now reports insufficient history, scores 0 for that category and keeps its 15% weight. A package that scored 100 before this change lands near 85.
That is deliberate. We price the gap rather than hiding it, and it resolves by itself for each package as its second capture lands.
We also fixed a bug the same day: the package stability check could compare a capture against itself, which meant a package captured once could publish a stability figure it had not earned.
28 July 2026: support for MCP 2026-07-28
Three changes moved published numbers, and all three were the rubric catching up with a new spec revision rather than any server changing.
2025-06-18 left the supported window. We give full protocol-recency marks to the current revision and the one before it. With 2026-07-28 as current, that window became 2026-07-28 and 2025-11-25, so servers on 2025-06-18 dropped from full marks to partial credit.
HTTP+SSE was marked down further, because the revision reclassified it as deprecated under the spec’s lifecycle policy, making it eligible for removal.
A new signal covers client registration. An authorization server offering only the deprecated RFC 7591 dynamic client registration is capped, against full marks for Client ID Metadata Documents.
There is more detail in what MCP 2026-07-28 changes.
26 July 2026: we stopped discounting what we cannot read
Until this change, a category we could not verify dropped out of the calculation entirely, and the remaining categories were re-normalised over what was left. The effect was backwards: a component we could learn nothing about was scored only on the parts we could see, so a package with no readable schema could publish a very high number having proved almost nothing.
Now every category always counts. If we cannot verify one, it scores 0, keeps its weight, and says exactly why it could not be checked.
Packages without a verified sandbox capture fell by roughly 40 points. This was intended, and it is transitional: sandbox extraction runs for every ecosystem, and each component recovers what it can prove as its capture lands.
On remote endpoints only one case moved. A server we could fully read that listed no tools at all used to be scored as though tool coverage did not apply to it, which was a discount for having nothing to show. It is now scored like any other category we cannot credit.
24 July 2026: package schema and tool coverage start counting
We had been capturing what a package’s server actually serves, by installing and running it in an isolated sandbox, and showing the results without scoring them. From this date, schema quality and tool coverage count toward a package’s score at the same weights the remote track uses.
Only packages with a verified capture moved.
23 July 2026: stability ramps over the first 30 days
Stability used to be close to a binary gate on 30 days of observation. A server we had watched for a day was treated much like one we had watched for a month.
It now ramps: a clean history earns credit in proportion to how long we have actually observed it, and churn is scaled by the same window. This means a young server that changes nothing can no longer be outscored by a young server that removed tools, which was possible before.
Young servers moved across the board, by design.
16 July 2026: “tool quality” became “tool coverage”
A naming fix, not a scoring change. The category has always measured something deterministic: how much of the tool surface carries descriptions, parameter documentation and output schemas. It has never been a judgement of whether your tools are any good, and the old name implied more than the checks assess.
No check, weight or threshold changed.
14 July 2026: authorisation scored once, and scoring beyond npm
Authorisation stopped being counted twice. Two separate signals were reading the same underlying fact and penalising it in both places. They became one graded signal, which also gives credit to a server that protects tool calls while leaving discovery open.
We started testing the call path, not just the handshake. Authorisation used to be measured on discovery alone, so a server that answered discovery anonymously but rejected actual tool calls was scored as though it were wide open. We now probe a deliberately non-existent tool inside the live session, which has no side effect and proves whether calls are gated.
Package scoring expanded beyond npm to PyPI, NuGet, OCI and MCPB, with real dependency resolution and ecosystem-native provenance verification.
13 July 2026: the score is the result
The status field used to carry a verdict of its own (verified, partial or inconclusive) alongside the number, which meant two things claimed to summarise one component. Status now records only whether a critical failure fired. The 0–100 score is the result.
Several scoring corrections landed with it:
- The context-footprint curve was wrong, and could score a perfectly lean server close to 0.
- A missing destructive-operation hint is now treated as unknown rather than safe, because the spec defaults it to destructive. The old behaviour rewarded leaving it off.
- Serving MCP over plaintext HTTP became a critical failure.
- Three facts that were being counted in two categories at once were removed from one of them.
- Tool-name style consistency was dropped. It measured house style, not a trust property.
11 July 2026: tool coverage becomes its own category
Tool surface documentation was pulled out into a category of its own, with deterministic signals for description coverage, parameter documentation and output schemas, and the remote weights were rebalanced around it.
28 June 2026: trust scoring goes live
The first public rubric. Signals aggregate into categories, categories into a single 0–100 score, and a small number of critical findings cap the total regardless of everything else. Within days it gained stable reason codes, so every number could be explained down to the individual check, and live verification rather than metadata alone.