Package rubric: npm, PyPI, NuGet, OCI, MCPB

A packaged server is code you download and run, so half the score is verifiable supply-chain and provenance evidence: 30% supply chain, 20% provenance and transparency.

The other half is the shared core, the five categories that measure the MCP contract itself and are scored identically on packages and remote endpoints: 18% schema quality and AI usability, 14% stability and change management, 8% tool coverage, 6% tool safety, 4% capabilities. A package does not publish its schema anywhere we can read, so we install and run it in an isolated sandbox and score the schema it serves there.

All seven categories are always scored and always shown. When we do not yet have a capture, the shared five score 0 and say why, exactly like any other check we cannot confirm. Sandbox capture is enabled for npm, PyPI, NuGet, OCI and MCPB, and we are working through the indexed fleet, so a package indexed recently may sit at that 0 for a short while before its capture lands. A clean, well-provenanced package whose captured schema is well documented can reach 100.

Ceilings vary by ecosystem, today

We resolve the full dependency tree for npm, PyPI, NuGet and MCPB, and for OCI only when the publisher attaches an SBOM (an SPDX or CycloneDX JSON document, plain or as a signed attestation, found through the OCI Referrers API, the sha256-<digest>.sbom tag, cosign’s sha256-<digest>.att tag or the attestation manifests BuildKit adds with --sbom). Most images attach none, so their dependency-derived signals stay inconclusive rather than being scored as clean. An unsigned SBOM leaves both signals inconclusive: a clean CVE result over the publisher’s own unverified inventory proves nothing. An SBOM signed in CI from your own repository (the same keyless Sigstore verification build provenance uses, over the same image digest) is graded by the known-CVEs check, including its Alpine, Debian, Ubuntu, AlmaLinux, Rocky Linux, Wolfi and Chainguard system packages (for the distribution releases on our list of those our advisory source covers; a release it holds only a token few advisories for, such as Ubuntu 24.10, is not on it) and its Go, Rust, Maven, RubyGems, Composer, npm, PyPI and NuGet packages. A signed SBOM is trusted as your attributable claim about the image and is not checked for completeness. Every package it lists must be checked before it can read as clean, so packages we cannot look up keep the result partial. Dependency health stays inconclusive either way, because an SBOM carries none of the registry facts it scores. Any advisory found against a package an unsigned SBOM lists is still shown on the page, marked as not graded. Malware coverage reaches npm, PyPI and NuGet but not OCI or MCPB, because our vendor has no package-URL coverage there. So the highest score an ecosystem can currently reach still varies sharply. This is an honest limitation of the analysis we have built so far, not a judgement about those ecosystems; the fix is to extend the same analysis to the ecosystems it misses.

The limitation is entirely in the package-only half of the rubric (supply chain and provenance). The shared core is scored from a sandbox run, which works the same way for every ecosystem, so it is not where the ceilings come from.

These are estimates read off the signal weights and each ecosystem’s known permanent gaps, not measurements from a scoring sweep. That sweep, scoring a fixed set of real components per ecosystem, has not been run, which is why this table gives bands rather than the two-decimal figures the arithmetic would happily produce. Treat the shape as reliable and the precision as absent.

An ecosystem we can analyse less thoroughly still reaches the middle of the range on the strength of a well-documented schema, because the shared core is half the score and is measured from a sandbox run that works the same way everywhere. That is the point of scoring the two halves separately.

A component we have not yet captured in the sandbox keeps the package-only half, so roughly 50% of the figures above: all five shared categories score 0 and keep their weight. That is a transitional state, not a property of the ecosystem.

Supply chain (30%)

Whether the code is safe to run today.

Culpability, not exposure

The CVE check resolves your production dependency tree (dev and peer dependencies excluded) and queries public advisory databases in batch across the whole tree, not just the root. Findings are attributed as direct (a dependency you declare) versus transitive (something deeper), and each names the vulnerable package, its version, and the path from your package to it; direct findings weigh more than transitive ones.

A vulnerability counts against your score only when a fixed version exists: when your declared range still resolves to a vulnerable version and an upgrade is on the shelf. An advisory with no published fix is shown as information, not scored as a failure: nobody can install their way out of it, so charging for it would measure luck, not diligence. If the tree only partly resolves, the CVE and dependency-health signals are scored on what resolved and marked partial; a half-walked tree never rounds up to a clean bill of health.

Sources: our malware-scanning provider for the malware verdict, public advisory databases for CVEs across the resolved tree, and our own resolution of the full dependency tree for dependency health (and for the tree the CVE check walks). A source with no data for your package leaves its signal inconclusive: we can’t confirm it, so it scores 0 and counts against your score (see below), not merely “not failed”. Likewise, when no malware-scanning provider is configured on our side the malware signal is marked inconclusive rather than silently scoring on.

Install scripts are classified by their command string, not merely their presence: no hook or a recognised native-build command (node-gyp, tsc, prebuild-install, and similar, with plain arguments) earns full marks; an unambiguous dropper (a curl | sh, node -e, base64 decode, chmod +x, an inline URL) scores 0; anything else opaque scores 0.5 (a hook runs and we haven’t established what it does; we don’t open the tarball). PyPI has no command string, so it is graded on what the release ships instead: an sdist built by a recognised build backend (setuptools, Poetry, Hatch, Flit, PDM, maturin or uv) earns full marks, and so does a wheel-only release with nothing that runs at start-up. Any setup.py, a build backend loaded from inside the package itself, a backend that runs the package’s own build scripts (scikit-build-core runs its CMake files, meson-python its meson.build), an unrecognised backend or a pyproject.toml we cannot parse, a build hook (hatch_build.py, pdm_build.py, a Hatch build or metadata hook other than a known file-reading plugin such as hatch-vcs, a Poetry build script, a PDM custom hook, a Hatch or PDM version source that runs code, or a setuptools cmdclass table), a setuptools attr or Flit dynamic version or description we cannot prove is a plain string the backend reads without importing the package, VCS metadata such as a .git folder shipped in the sdist, a stray root-level module setuptools’ legacy mode could import (no build-backend declared), or a wheel that runs code at every Python start-up (an import line in a .pth file, or a sitecustomize or usercustomize module) scores 0.5, because we can see the hook but not what it does. A release we could not fully download or scan (including an archive with more entries than we walk, or one built so that pip and our reader could see different files), with nothing found in what we did read, is inconclusive rather than clean, and so is one whose version PyPI did not answer for. Dependency health scores the resolved production set (deprecated, stale, or no source repo) and names the specific unhealthy dependencies so you can act on them.

Provenance & transparency (20%)

Whether you can prove where the code came from and who stands behind it.

A reachable source repository underpins several of these (we read it for the security policy, and it anchors the provenance attestation) so linking a verifiable repository is often the highest-leverage single fix.

We deliberately don’t score “verified publisher.” npm performs no identity verification on accounts or organisations (both are free to create) so a namespace can’t prove who stands behind a package, and crediting it would reward a signal anyone could fake. The accountable-identity question is answered instead by build provenance: a cryptographically signed attestation, following the SLSA provenance model, that binds the artifact to the source repository and CI workflow that produced it. Both GitHub Actions and GitLab CI attestations are accepted. An attestation that is cryptographically sound but names a different repository than the registry metadata declares is reported as a stale-metadata mismatch, not an accusation of forgery: it usually means a moved repo or a stale repository field, so it earns no credit but accuses nobody.

The shared core (50%)

Whether an agent that installs your package can actually understand and rely on what it exposes. We run the published package in an isolated sandbox with strict resource limits, speak MCP to it over stdio, and capture the schema it serves: its tools, parameters, prompts, resources and server instructions. That capture is then scored by the same signals as the remote rubric, with the same reason codes:

• Schema quality & AI usability (18%): prompt and resource description completeness, an AI-judged instruction-clarity signal (fixed rubric, cached so an unchanged schema is never re-judged), the context-window token footprint, and usage examples.
• Stability & change management (14%): whether your tool surface stays stable across the versions we have captured. See below.
• Tool coverage (8%): tool description coverage, parameter documentation, and declared structured output schemas.
• Tool safety (6%): what that same tool text tells a host model to do. A scan for injection markers, whether tools that perform irreversible operations declare their MCP annotations, and an AI-judged reading for text written to manipulate the model. See tool safety.
• Capabilities (4%): how current the MCP spec revision you negotiate is, plus a small bonus if you ship UI template metadata.

How stability works on a package

A published version is immutable, so stability here is not about your package changing under anyone: it cannot. It is about what changes between versions, which is what someone meets when they upgrade. We compare the earliest capture we hold from the last 30 days against the most recent one, and a removed tool or an incompatibly changed parameter counts against you. Added tools do not.

Two consequences worth knowing:

• If both captures record the same version and they disagree, we report that we could not tell what happened, rather than counting it against you. Either our sandbox is unreliable for your package (the usual cause is a tool list that depends on a secret or a native dependency it resolved differently between runs), or the coordinate you publish under is one that can be re-pointed, like an OCI tag. We cannot tell those apart from the outside, so we say so instead of guessing either way. Two captures of one version that agree are simply stable.
• Until we hold two captures we say so and score 0, rather than calling an unmeasured surface stable. We began retaining the history this needs in July 2026, so most packages will show that for a while.

Unlike the remote rubric, nothing here penalises a dropped authentication requirement or a changed transport. Neither is something a package has.

When we cannot read your schema

All four are always scored and always shown. When we do not have a verified capture of your current version, whether because we have never run your package, our capture is of a version you have since superseded, the run yielded nothing, our storage trim cut the capture short, or your package declared that it provides tools and our run could not list them, each category records a single unverified finding naming which of those it was, and scores 0 while keeping its full weight. That is the same rule as everywhere else here: we do not credit what we cannot confirm, and we do not hide a category we could not read. A failed sandbox run is still our limitation rather than your defect, and the finding says so, but leaving the category out would overstate what we have actually verified about your package.

The sandbox holds no credentials of ours. If your registry entry declares environment variables (an API key or similar) that your server will not start without, a verified owner of the server can supply test values privately, from the server’s page under My servers in their account. They are encrypted before they are stored, only our scoring worker can read them, and they are removed from everything we store and publish about the run. (True/false and numeric settings for an MCPB bundle are the exception, unless the bundle marks them sensitive: they are settings rather than secrets, so they are left as your server prints them.) The sandbox still has no network access while your server runs, so a server that must reach the internet at start-up will not start whatever values it is given. Use a scoped, throwaway credential made for testing. The package is scored by the same rubric either way, and we do not mark which runs used owner-supplied values. If your server repeats a supplied value in its tool definitions, what we publish shows [redacted] in its place.

Capabilities is the one exception to the truncation rule: if a capture was too large to store in full we still read the protocol version out of it, because trimming only removes descriptive text.

When a capture succeeds and the package genuinely exposes no tools, prompts or resources, the findings say exactly that. Those are confirmed readings rather than gaps, so they read as findings, not as “unverified”.

The captured tool list, with per-tool token estimates, is shown on your server page’s MCP tools card.

What unverifiable signals do here

Some signals we simply couldn’t verify from public data: a source with no data for the package, an unpublished version, or a repository that isn’t publicly accessible. Because we only credit what we can confirm, they score 0 and count against your score: an unverifiable signal is never given the benefit of the doubt. Making the underlying evidence public (a reachable repo, a published version) is what lets the signal be verified and count in your favour.

Two of these gaps are ours rather than yours: an ecosystem we cannot yet analyse fully, and a sandbox run that fails on our side. They score 0 for the same reason (we only credit what we confirm) but there is nothing for you to fix, and the finding on your page says so.

Full machine spec: docs/scoring-package.md in the repository.