evalpair
PYPI · EVALPAIR
Run LLM eval suites on local or hosted models; compare runs with intervals and a noise floor.
Available components
Last scanned (5 hours ago)
Scores higher than 81% of PyPI servers
How this component scores in each security and reliability category. Every signal is checked automatically from public evidence about the published package, including repeated runs of it in an isolated sandbox, and we only credit what we can confirm. How we score → Why this is hard to score →
Supply Chain Security100
- No malware found by supply-chain analysis.Pass
- No known CVEs affecting this package version or its production dependencies.Pass
- Runs uv_build at install time, a recognised build step with no custom scripting around it. View diagnostics → Pass
- 1 of 28 dependencies flagged as unhealthy. View diagnostics → Partial
Provenance & Transparency97
- Source repository is publicly reachable at the declared URL. View diagnostics → Pass
- Cryptographically verified build provenance (signed, bound to Srikarmk/evalpair). View diagnostics → Pass
- Clear OSI-approved license (MIT).Pass
- Actively maintained (last published 0 days ago).Pass
- Disclosure check failed: no security disclosure policy was found in the source repository. See how to fix → Fail
Schema Quality & AI Usability72
- AI-judged instruction clarity (good).Pass
- Context-footprint check failed: tool/resource definitions use about 604 tokens (~120/item across 5 items; 5 tools + 0 resources), over budget; trim descriptions and params. See how to fix → Fail
- Usage-examples check failed: none of the tools include examples. See how to fix → Fail
Stability & Change Management0
- Stability not yet verified: not enough scan history yet (needs a 30-day window).Unverified
Tool Coverage98
- 100% of tools have a non-trivial description (not blank, and not just the tool's name).Pass
- 94% of tool parameters carry a description.Partial
- Structured output schemas are declared (100% of tools); any adoption earns full credit.Pass
Tool Safety100
- No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.Pass
- All 1 tool(s) whose name or description implies an irreversible operation declare an MCP destructiveHint annotation.Pass
- An AI judge read all 6 captured unit(s) of tool text and found none that tries to manipulate the model reading it.Pass
Capabilities100
- Implements a current MCP spec version (2026-07-28).Pass
Unverified: 1 category
A category scored 0 because we could not verify it: a data source with nothing on this package, evidence we could not reach, or a check we could not run. We only credit what we can confirm.
Every change we have recorded for this component, newest first. Security-relevant changes are always shown. ▲ marks a change for the better, ▼ a change for the worse; unmarked changes are neutral.
- 5 Nov 26 80
First indexed and scored.
Diagnostic detail from the automated scan of this channel: what the scanner observed at each step, so you can see exactly where a check passed or failed. It is informational only and never changes the trust score.
Captured 5 Nov 2026 · Analysed pypi/evalpair@0.1.1
Provenance Verified
A signed build attestation was found and verified, binding this exact artifact to the source repository it claims to come from.
| Result | Verified |
|---|---|
| Ecosystem | pypi |
| Reason | Verified |
| Discovered via | Registry attestation endpoint |
| Source repo | Srikarmk/evalpair |
| Certificate issuer | https://token.actions.githubusercontent.com |
| Certificate SAN | https://github.com/Srikarmk/evalpair/.github/workflows/publish.yml@refs/tags/v0.1.1 |
| Rekor log index | 3692683419 |
| Predicate type | PyPI publish attestation https://docs.pypi.org/attestations/publish/v1 |
| Subject digest | sha256:2a28b5b5ac3b501bd4339f3ed8bdc8c6a1b9faf9568ffed3dee4bc07c6785c98 |
Background: How many MCP packages publish verified provenance →
Install scripts 1 script
| Hook | Tier | Command |
|---|---|---|
| build_backend | allowlisted | uv_build |
Background: Why install scripts are a supply-chain risk →
Dependencies 28 packages
| Packages resolved | 28 |
|---|---|
| No linked repository | 1 |
| Tree resolution | Complete |
Background: SBOMs and build attestations, explained →
The tools this component advertises to a client, with an estimated token cost for each. Expand a tool to see its parameters and schema. The per-tool counts are indicative and are not scored directly; the schema's total context footprint is one signal in Schema Quality & AI Usability. A tool's description is untrusted text the model reads on every call, which is what makes this list a security surface and not just an inventory: how tool poisoning works →
compare_runs Compare two runs ~81
Use to compare two runs of the same suite item by item: score difference with a paired interval, flips each way, and whether answers moved more than the noise floor.
| Name | Type | Req | Description |
|---|---|---|---|
| a | string | yes | The baseline run id. |
| b | string | yes | The run to compare against it. |
| examples | integer | – | Items to show that got worse and better. |
Structured output declared, but exposes no named fields.
No examples provided.
list_runs List saved runs ~57
Use to find earlier runs to show or compare: run id, suite, model, item count, whether it has a repeat pass.
| Name | Type | Req | Description |
|---|---|---|---|
| limit | integer | – | How many, newest first. |
| suite | – | – | Only runs of this suite. |
Structured output declared, but exposes no named fields.
No examples provided.
list_suites List eval suites ~30
Use to see which eval suites can be run: name, item count, checker and what each measures.
Input schema present but exposes no named parameters.
Structured output declared, but exposes no named fields.
No examples provided.
run_eval Run an eval suite ~242
Use to run a suite against one model and save the run. Returns its score with an interval and its first failures. Calls the model; cached replies are reused.
| Name | Type | Req | Description |
|---|---|---|---|
| backend | string | – | 'ollama' for Ollama's API; 'openai' for any OpenAI-compatible server. |
| base_url | – | – | Server URL. Defaults: http://localhost:11434 (ollama), http://localhost:8000/v1 (openai). |
| label | – | – | Run id to save under (letters, digits, . _ -). |
| limit | – | – | Only the first N items; start small, runs take a while. |
| model | string | yes | Model name as the server knows it, e.g. 'llama3.1:8b' on Ollama. |
| repeat | boolean | – | Send every item twice to measure the noise floor. |
| seed | – | – | Sampling seed, where the server supports one. |
| suite | string | yes | A suite name from list_suites, or a path to a suite folder. |
| temperature | number | – | – |
| think | – | – | Ollama thinking on or off, for models that support it. |
Structured output declared, but exposes no named fields.
No examples provided.
show_run Show a run ~65
Use to see a saved run's score with its interval, the noise floor if it has a repeat, and the end of failed replies.
| Name | Type | Req | Description |
|---|---|---|---|
| failures | integer | – | How many failed items to include. |
| run | string | yes | A run id from list_runs or run_eval. |
Structured output declared, but exposes no named fields.
No examples provided.
How do I install the evalpair MCP server?
evalpair runs locally as a PyPI package, launched with uvx evalpair. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.
pypi · evalpair
claude mcp add srikarmk-evalpair -- uvx evalpair
What is the evalpair MCP server?
evalpair is an MCP server listed in the public MCP registry as io.github.Srikarmk/evalpair. Run LLM eval suites on local or hosted models; compare runs with intervals and a noise floor. This page covers its PyPI package (evalpair).
Is the evalpair MCP server safe to use?
evalpair scores 80 out of 100 on VerifyMCP. We found no known CVEs affecting it as of 5 November 2026. Its build provenance is signed and verified. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.
What tools does the evalpair MCP server expose?
evalpair exposes 5 tools: list_suites, list_runs, run_eval, show_run, compare_runs. Their descriptions and schemas cost roughly 475 tokens of context every time the server is loaded.
Is the evalpair MCP server still maintained?
evalpair is still listed as active in the MCP registry. We last reached this channel on 5 November 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.
What licence is the evalpair MCP server under?
evalpair declares the MIT licence, which is OSI-approved. That covers the source only, and says nothing about the cost of any service it calls.