# io.github.diegoperezg7/evalsmith-mcp (npm · evalsmith-mcp)

Evaluate RAG retrieval, agent tool calls and prompt regressions with deterministic metrics.

- Trust score: 71/100 (medium)
- Change this week: 0
- Registry status: active
- Liveness: live
- Owner verified: no
- Last scored: 2026-11-05

## Components

- npm · `evalsmith-mcp`: 71/100 (this document), [markdown](https://verifymcp.io/servers/diegoperezg7-evalsmith-mcp/evalsmith-mcp.md), [page](https://verifymcp.io/servers/diegoperezg7-evalsmith-mcp/evalsmith-mcp)

## Channel facts

- Registry: `npm`
- Package: `evalsmith-mcp`
- Version: `0.1.0`
- Transport: `stdio`

## Trust breakdown

How this component scores in each security and reliability category. Every signal is checked automatically from public evidence about the published package, including repeated runs of it in an isolated sandbox, and we only credit what we can confirm. Scores are 0–100 per category. Scoring method: https://verifymcp.io/docs/scoring (what has changed: https://verifymcp.io/docs/scoring/changelog)

Scored 2026-11-05.

- **Supply Chain Security**: 98/100
  - No malware found by supply-chain analysis.
  - No known CVEs affecting this package version or its production dependencies.
  - No install/post-install scripts declared.
  - 34 of 107 dependencies flagged as unhealthy.
- **Provenance & Transparency**: 48/100
  - Source repository is publicly reachable at the declared URL.
  - Provenance check failed: no build-provenance attestation is published.
  - Clear OSI-approved license (MIT).
  - Actively maintained (last published 7 days ago).
  - Publishes a security disclosure policy (SECURITY.md).
- **Schema Quality & AI Usability**: 81/100
  - 100% of prompts and resources have a non-trivial description (not blank, and not just the item's name).
  - AI-judged instruction clarity (good).
  - Tool/resource definitions use about 1308 tokens (~72/item across 18 items; 5 tools + 13 resources), lean.
  - Usage-examples check failed: none of the tools include examples.
- **Stability & Change Management**: 0/100
  - Stability not yet verified: not enough scan history yet (needs a 30-day window).
- **Tool Coverage**: 97/100
  - 100% of tools have a non-trivial description (not blank, and not just the tool's name).
  - 91% of tool parameters carry a description.
  - Structured output schemas are declared (100% of tools); any adoption earns full credit.
- **Tool Safety**: 100/100
  - No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.
  - We read all 5 captured tool definition(s), and no name or description among them implies an irreversible operation.
  - An AI judge read all 7 captured unit(s) of tool text and found none that tries to manipulate the model reading it.
- **Capabilities**: 100/100
  - Implements a supported MCP spec version (2025-11-25); the latest is 2026-07-28.

**Unverified: 1 category.** A category scored 0 because we could not verify it: a data source with nothing on this package, evidence we could not reach, or a check we could not run. We only credit what we can confirm.

## Install

### How do I install the io.github.diegoperezg7/evalsmith-mcp server?

io.github.diegoperezg7/evalsmith-mcp runs locally as an npm package, launched with npx -y evalsmith-mcp. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.

### Claude

```bash
claude mcp add diegoperezg7-evalsmith-mcp -- npx -y evalsmith-mcp
```

### Cursor

```json
{
  "mcpServers": {
    "diegoperezg7-evalsmith-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "evalsmith-mcp"
      ]
    }
  }
}
```

### VS Code

```json
{
  "servers": {
    "diegoperezg7-evalsmith-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "evalsmith-mcp"
      ]
    }
  }
}
```

### Codex

```bash
codex mcp add diegoperezg7-evalsmith-mcp -- npx -y evalsmith-mcp
```

### opencode

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "diegoperezg7-evalsmith-mcp": {
      "type": "local",
      "command": [
        "npx",
        "-y",
        "evalsmith-mcp"
      ],
      "enabled": true
    }
  }
}
```

### OpenClaw

```bash
openclaw mcp add diegoperezg7-evalsmith-mcp --command npx --arg -y --arg evalsmith-mcp
```

### Hermes

```yaml
mcp_servers:
  diegoperezg7-evalsmith-mcp:
    command: "npx"
    args: ["-y", "evalsmith-mcp"]
```

### Netclaw

```json
{
  "McpServers": {
    "diegoperezg7-evalsmith-mcp": {
      "Transport": "stdio",
      "Command": "npx",
      "Arguments": [
        "-y",
        "evalsmith-mcp"
      ]
    }
  }
}
```

### Vellum

```bash
assistant mcp add diegoperezg7-evalsmith-mcp -t stdio -c npx -a -y evalsmith-mcp
```

### Other

```json
{
  "mcpServers": {
    "diegoperezg7-evalsmith-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "evalsmith-mcp"
      ]
    }
  }
}
```

## Changelog

Every change recorded for this component, newest first. Days that predate change tracking, or that we cannot explain, say so: "we were watching and nothing happened" and "we were not watching" are different claims.

### 2026-10-29 (score 71, +15)

- [security improvement] Malware scan: unverified → pass

### 2026-10-28 (score 56)

First indexed and scored.

## MCP tools (5)

### `evaluate_retrieval` (~204 tokens)

Evaluate retrieval

Score a ranked retrieval run against relevance judgements (qrels). Returns recall@k, precision@k, nDCG@k (graded, linear gain), hit rate@k and MRR, averaged over judged queries, plus per-query values. Deterministic; no model calls.

Input parameters:

- `k` (array): Cut-offs to evaluate. Defaults to [1, 3, 5, 10].
- `qrels` (object, required): Relevance judgements: { queryId: { docId: grade } }. Grade 0 means not relevant; higher grades are more relevant (used by nDCG). Example: { "q1": { "d3": 2, "d7": 1 } }
- `run` (object, required): Ranked retrieval output: { queryId: [docId, ...] }, best match first. Example: { "q1": ["d7", "d3", "d9"] }

Output parameters:

- `metrics` (object): Metric name to value, e.g. { "recall@5": 0.83, "mrr": 0.71 }
- `missingFromRun` (array): Judged queries absent from the run (scored as an empty ranking).
- `perQuery` (array)
- `queries` (integer): Queries with at least one relevant document.
- `skippedQueries` (array): Queries in qrels with no relevant documents (excluded from averages).

### `compare_configs` (~218 tokens)

Compare retrieval configurations

Compare 2-3 retrieval configurations on the same qrels. Ranks them on a primary metric, attaches seeded paired-bootstrap confidence intervals, and says whether the winner's lead over the runner-up is significant (interval of the gap excludes zero).

Input parameters:

- `bootstrap` (object): Paired bootstrap over queries. Defaults: 1000 samples, seed 42, 95% confidence.
- `configs` (array, required): Two or three runs over the same queries.
- `k` (array): Cut-offs to evaluate. Defaults to [1, 3, 5, 10].
- `metric` (string): Primary metric that decides the winner, e.g. ndcg@10, recall@5, mrr.
- `qrels` (object, required): Relevance judgements: { queryId: { docId: grade } }. Grade 0 means not relevant; higher grades are more relevant (used by nDCG). Example: { "q1": { "d3": 2, "d7": 1 } }

Output parameters:

- `bootstrap` (object)
- `configs` (array)
- `delta` (object): Paired bootstrap interval of winner minus runner-up.
- `metric` (string)
- `runnerUp` (string)
- `significant` (boolean): True when the delta interval excludes zero.
- `summary` (string)
- `winner` (string)

### `check_tool_calls` (~157 tokens)

Check agent tool calls

Compare an agent's tool-call trace with the expected calls: right tool, right order (strict or lax) and arguments that satisfy exact / regex / subset / any matchers. Returns a pass flag, a score and a reason for every mismatch.

Input parameters:

- `expected` (array, required): Expected calls. Argument matchers: {"type":"exact","value":...}, {"type":"regex","pattern":"..."}, {"type":"subset","value":{...}}, {"type":"any"}.
- `order` (string): strict: the trace must be exactly the expected calls, in order. lax: every expected call must appear somewhere; order and extra calls are allowed.
- `trace` (array, required): Tool calls the agent made, in order.

Output parameters:

- `order` (string): strict: the trace must be exactly the expected calls, in order. lax: every expected call must appear somewhere; order and extra calls are allowed.
- `passed` (boolean)
- `score` (number): Matched expected calls / expected calls.
- `steps` (array)
- `summary` (string)
- `unexpected` (array)

### `detect_prompt_regression` (~157 tokens)

Detect prompt regression

Run a golden set's deterministic checks (exact, regex, contains, JSON Schema, length) against baseline and candidate outputs. A regression is a case that passes on baseline and fails on candidate. An LLM judge can be enabled per call for a second opinion; the verdict only depends on the checks.

Input parameters:

- `baseline` (object, required): Outputs of the current prompt/model, keyed by case id.
- `candidate` (object, required): Outputs of the proposed prompt/model, keyed by case id.
- `golden` (array, required): Golden cases with deterministic checks.
- `judge` (object): Optional pairwise LLM judge on the server's configured OpenAI-compatible endpoint (e.g. Ollama). Off by default; it never changes the verdict.

Output parameters:

- `baselinePassRate` (number)
- `candidatePassRate` (number)
- `cases` (integer)
- `improvements` (array): Cases that fail on baseline and pass on candidate.
- `judge` (object)
- `perCase` (array)
- `regressions` (array): Cases that pass on baseline and fail on candidate.
- `summary` (string)
- `verdict` (string)

### `generate_report` (~147 tokens)

Generate report

Turn the results of the other evalsmith tools into a Markdown summary, a JSON document or JUnit XML for CI dashboards. Optional gates (metric -> minimum) decide the overall pass/fail.

Input parameters:

- `comparison` (object): structuredContent of compare_configs.
- `format` (string)
- `gates` (object): Minimum values, e.g. { "recall@5": 0.8, "tool_calls": 1 }.
- `name` (string)
- `regression` (object): structuredContent of detect_prompt_regression.
- `retrieval` (object): structuredContent of evaluate_retrieval.
- `toolCalls` (object): structuredContent of check_tool_calls.

Output parameters:

- `content` (string)
- `format` (string)
- `passed` (boolean)

## Diagnostics

Captured diagnostic sections: Provenance, Dependencies. The full working is on the page: https://verifymcp.io/servers/diegoperezg7-evalsmith-mcp/evalsmith-mcp#diagnostics

## Score history

- 2026-11-05: 71
- 2026-11-04: 71
- 2026-11-03: 71
- 2026-11-02: 71
- 2026-11-01: 71
- 2026-10-31: 71
- 2026-10-30: 71
- 2026-10-29: 71
- 2026-10-28: 56

## Common questions

### What is the io.github.diegoperezg7/evalsmith-mcp server?

io.github.diegoperezg7/evalsmith-mcp is listed in the public MCP registry as io.github.diegoperezg7/evalsmith-mcp. Evaluate RAG retrieval, agent tool calls and prompt regressions with deterministic metrics. This page covers its npm package (evalsmith-mcp).

### Is the io.github.diegoperezg7/evalsmith-mcp server safe to use?

io.github.diegoperezg7/evalsmith-mcp scores 71 out of 100 on VerifyMCP. We found no known CVEs affecting it as of 5 November 2026. It declares no install or post-install scripts. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.

### What tools does the io.github.diegoperezg7/evalsmith-mcp server expose?

io.github.diegoperezg7/evalsmith-mcp exposes 5 tools: evaluate_retrieval, compare_configs, check_tool_calls, detect_prompt_regression, generate_report. Their descriptions and schemas cost roughly 883 tokens of context every time the server is loaded.

### Is the io.github.diegoperezg7/evalsmith-mcp server still maintained?

io.github.diegoperezg7/evalsmith-mcp is still listed as active in the MCP registry. We last reached this channel on 5 November 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.

### What licence is the io.github.diegoperezg7/evalsmith-mcp server under?

io.github.diegoperezg7/evalsmith-mcp declares the MIT licence, which is OSI-approved. That covers the source only, and says nothing about the cost of any service it calls.

### What did VerifyMCP test on the io.github.diegoperezg7/evalsmith-mcp server?

VerifyMCP ran 15 checks on io.github.diegoperezg7/evalsmith-mcp, covering Malware scan, Known CVEs, Install scripts, Dependency health and 11 more. One item could not be completed, a gap on our side rather than a finding about io.github.diegoperezg7/evalsmith-mcp: Stability. One category scores 0 for that reason, because we only credit what we can confirm. Endpoint Security and Transport & Reachability do not apply to an npm package, so they were not tested. The score does not measure everything, and the scoring documentation lists what it leaves out.

## Links

- npm package: https://www.npmjs.com/package/evalsmith-mcp
- Socket report: https://socket.dev/npm/package/evalsmith-mcp
- Repository: https://github.com/diegoperezg7/evalsmith-mcp
- Changelog RSS feed: https://verifymcp.io/servers/diegoperezg7-evalsmith-mcp/evalsmith-mcp.xml
- Changelog JSON feed: https://verifymcp.io/servers/diegoperezg7-evalsmith-mcp/evalsmith-mcp/changelog.json
- Trust report (JSON): https://verifymcp.io/servers/diegoperezg7-evalsmith-mcp/evalsmith-mcp.json
- HTML version of this page: https://verifymcp.io/servers/diegoperezg7-evalsmith-mcp/evalsmith-mcp