# evalpair (pypi · evalpair)

Run LLM eval suites on local or hosted models; compare runs with intervals and a noise floor.

- Trust score: 80/100 (high trust)
- Registry status: active
- Liveness: live
- Owner verified: no
- Last scored: 2026-11-05

## Components

- pypi · `evalpair`: 80/100 (this document), [markdown](https://verifymcp.io/servers/srikarmk-evalpair/evalpair.md), [page](https://verifymcp.io/servers/srikarmk-evalpair/evalpair)

## Channel facts

- Registry: `pypi`
- Package: `evalpair`
- Version: `0.1.1`
- Transport: `stdio`

## Trust breakdown

How this component scores in each security and reliability category. Every signal is checked automatically from public evidence about the published package, including repeated runs of it in an isolated sandbox, and we only credit what we can confirm. Scores are 0–100 per category. Scoring method: https://verifymcp.io/docs/scoring (what has changed: https://verifymcp.io/docs/scoring/changelog)

Scored 2026-11-05.

- **Supply Chain Security**: 100/100
  - No malware found by supply-chain analysis.
  - No known CVEs affecting this package version or its production dependencies.
  - Runs uv_build at install time, a recognised build step with no custom scripting around it.
  - 1 of 28 dependencies flagged as unhealthy.
- **Provenance & Transparency**: 97/100
  - Source repository is publicly reachable at the declared URL.
  - Cryptographically verified build provenance (signed, bound to Srikarmk/evalpair).
  - Clear OSI-approved license (MIT).
  - Actively maintained (last published 0 days ago).
  - Disclosure check failed: no security disclosure policy was found in the source repository.
- **Schema Quality & AI Usability**: 72/100
  - AI-judged instruction clarity (good).
  - Context-footprint check failed: tool/resource definitions use about 604 tokens (~120/item across 5 items; 5 tools + 0 resources), over budget; trim descriptions and params.
  - Usage-examples check failed: none of the tools include examples.
- **Stability & Change Management**: 0/100
  - Stability not yet verified: not enough scan history yet (needs a 30-day window).
- **Tool Coverage**: 98/100
  - 100% of tools have a non-trivial description (not blank, and not just the tool's name).
  - 94% of tool parameters carry a description.
  - Structured output schemas are declared (100% of tools); any adoption earns full credit.
- **Tool Safety**: 100/100
  - No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.
  - All 1 tool(s) whose name or description implies an irreversible operation declare an MCP destructiveHint annotation.
  - An AI judge read all 6 captured unit(s) of tool text and found none that tries to manipulate the model reading it.
- **Capabilities**: 100/100
  - Implements a current MCP spec version (2026-07-28).

**Unverified: 1 category.** A category scored 0 because we could not verify it: a data source with nothing on this package, evidence we could not reach, or a check we could not run. We only credit what we can confirm.

## Install

### How do I install the evalpair MCP server?

evalpair runs locally as a PyPI package, launched with uvx evalpair. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.

### Claude

```bash
claude mcp add srikarmk-evalpair -- uvx evalpair
```

### Cursor

```json
{
  "mcpServers": {
    "srikarmk-evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ]
    }
  }
}
```

### VS Code

```json
{
  "servers": {
    "srikarmk-evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ]
    }
  }
}
```

### Codex

```bash
codex mcp add srikarmk-evalpair -- uvx evalpair
```

### opencode

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "srikarmk-evalpair": {
      "type": "local",
      "command": [
        "uvx",
        "evalpair"
      ],
      "enabled": true
    }
  }
}
```

### OpenClaw

```bash
openclaw mcp add srikarmk-evalpair --command uvx --arg evalpair
```

### Hermes

```yaml
mcp_servers:
  srikarmk-evalpair:
    command: "uvx"
    args: ["evalpair"]
```

### Netclaw

```json
{
  "McpServers": {
    "srikarmk-evalpair": {
      "Transport": "stdio",
      "Command": "uvx",
      "Arguments": [
        "evalpair"
      ]
    }
  }
}
```

### Vellum

```bash
assistant mcp add srikarmk-evalpair -t stdio -c uvx -a evalpair
```

### Other

```json
{
  "mcpServers": {
    "srikarmk-evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ]
    }
  }
}
```

## Changelog

Every change recorded for this component, newest first. Days that predate change tracking, or that we cannot explain, say so: "we were watching and nothing happened" and "we were not watching" are different claims.

### 2026-11-05 (score 80)

First indexed and scored.

## MCP tools (5)

### `list_suites` (~30 tokens)

List eval suites

Use to see which eval suites can be run: name, item count, checker and what each measures.

### `list_runs` (~57 tokens)

List saved runs

Use to find earlier runs to show or compare: run id, suite, model, item count, whether it has a repeat pass.

Input parameters:

- `limit` (integer): How many, newest first.
- `suite`: Only runs of this suite.

### `run_eval` (~242 tokens)

Run an eval suite

Use to run a suite against one model and save the run. Returns its score with an interval and its first failures. Calls the model; cached replies are reused.

Input parameters:

- `backend` (string): 'ollama' for Ollama's API; 'openai' for any OpenAI-compatible server.
- `base_url`: Server URL. Defaults: http://localhost:11434 (ollama), http://localhost:8000/v1 (openai).
- `label`: Run id to save under (letters, digits, . _ -).
- `limit`: Only the first N items; start small, runs take a while.
- `model` (string, required): Model name as the server knows it, e.g. 'llama3.1:8b' on Ollama.
- `repeat` (boolean): Send every item twice to measure the noise floor.
- `seed`: Sampling seed, where the server supports one.
- `suite` (string, required): A suite name from list_suites, or a path to a suite folder.
- `temperature` (number)
- `think`: Ollama thinking on or off, for models that support it.

### `show_run` (~65 tokens)

Show a run

Use to see a saved run's score with its interval, the noise floor if it has a repeat, and the end of failed replies.

Input parameters:

- `failures` (integer): How many failed items to include.
- `run` (string, required): A run id from list_runs or run_eval.

### `compare_runs` (~81 tokens)

Compare two runs

Use to compare two runs of the same suite item by item: score difference with a paired interval, flips each way, and whether answers moved more than the noise floor.

Input parameters:

- `a` (string, required): The baseline run id.
- `b` (string, required): The run to compare against it.
- `examples` (integer): Items to show that got worse and better.

## Diagnostics

Captured diagnostic sections: Provenance, Install scripts, Dependencies. The full working is on the page: https://verifymcp.io/servers/srikarmk-evalpair/evalpair#diagnostics

## Score history

- 2026-11-05: 80

## Common questions

### What is the evalpair MCP server?

evalpair is an MCP server listed in the public MCP registry as io.github.Srikarmk/evalpair. Run LLM eval suites on local or hosted models; compare runs with intervals and a noise floor. This page covers its PyPI package (evalpair).

### Is the evalpair MCP server safe to use?

evalpair scores 80 out of 100 on VerifyMCP. We found no known CVEs affecting it as of 5 November 2026. Its build provenance is signed and verified. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.

### What tools does the evalpair MCP server expose?

evalpair exposes 5 tools: list_suites, list_runs, run_eval, show_run, compare_runs. Their descriptions and schemas cost roughly 475 tokens of context every time the server is loaded.

### Is the evalpair MCP server still maintained?

evalpair is still listed as active in the MCP registry. We last reached this channel on 5 November 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.

### What licence is the evalpair MCP server under?

evalpair declares the MIT licence, which is OSI-approved. That covers the source only, and says nothing about the cost of any service it calls.

## Links

- PyPI project: https://pypi.org/project/evalpair/
- Socket report: https://socket.dev/pypi/package/evalpair
- Repository: https://github.com/Srikarmk/evalpair
- Changelog RSS feed: https://verifymcp.io/servers/srikarmk-evalpair/evalpair.xml
- Changelog JSON feed: https://verifymcp.io/servers/srikarmk-evalpair/evalpair.json
- HTML version of this page: https://verifymcp.io/servers/srikarmk-evalpair/evalpair