Skip to content
verify mcp Beta VerifyMCP is currently in beta. If you notice any issues, get in touch and we’ll put it right.

io.github.dbsectrainer/mcp-eval-runner

NPM · MCP-EVAL-RUNNER

A standardized testing harness for MCP servers and agent workflows

Available components

0 this week 64 Trust /100

Last scanned (17 hours ago)

Scores higher than 37% of npm servers

Category Testing and QA

Trust breakdown (7 categories)

How this component scores in each security and reliability category. Every signal is checked automatically from public evidence about the published package, including repeated runs of it in an isolated sandbox, and we only credit what we can confirm. How we score → Why this is hard to score →

Supply Chain Security98
  • No malware found by supply-chain analysis.Pass
  • No known CVEs affecting this package version or its production dependencies.Pass
  • No install/post-install scripts declared.Pass
  • 34 of 110 dependencies flagged as unhealthy. View diagnostics → Partial
Provenance & Transparency48
  • Source repository is publicly reachable at the declared URL. View diagnostics → Pass
  • Provenance check failed: no build-provenance attestation is published. See how to fix → View diagnostics → Fail
  • Clear OSI-approved license (MIT).Pass
  • Actively maintained (last published 17 days ago).Pass
  • Publishes a security disclosure policy (SECURITY.md).Pass
Schema Quality & AI Usability48
  • 0% of prompts and resources have a non-trivial description (not blank, and not just the item's name).Fail
  • AI-judged instruction clarity (good).Pass
  • Tool/resource definitions use about 961 tokens (~96/item across 10 items; 10 tools + 0 resources), lean.Pass
  • Usage-examples check failed: none of the tools include examples. See how to fix → Fail
Stability & Change Management0
  • Stability not yet verified: not enough scan history yet (needs a 30-day window).Unverified
Tool Coverage100
  • 100% of tools have a non-trivial description (not blank, and not just the tool's name).Pass
  • 100% of tool parameters carry a description.Pass
Tool Safety75
  • No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.Pass
  • 0 of 1 tool(s) whose name or description implies an irreversible operation declare an MCP destructiveHint annotation; "run_suite" implies "execute" and declares no destructiveHint at all, which the MCP spec reads as destructive by default. See how to fix → Fail
  • An AI judge read all 10 captured unit(s) of tool text and found none that tries to manipulate the model reading it.Pass
Capabilities100
  • Implements a supported MCP spec version (2025-11-25); the latest is 2026-07-28.Pass

Unverified: 1 category

A category scored 0 because we could not verify it: a data source with nothing on this package, evidence we could not reach, or a check we could not run. We only credit what we can confirm.

What we tested

Checks we ran (15)

  • Malware scan
  • Known CVEs
  • Install scripts
  • Dependency health
  • Source repository
  • Provenance
  • License
  • Maintenance
  • Security disclosure
  • Schema quality
  • Tool coverage
  • Injection markers
  • Destructive annotations
  • Judged manipulation
  • MCP protocol

Checks we could not complete (1)

These are gaps on our side, not findings about this package. We only credit what we can confirm, so a category made up entirely of them scores 0.

  • Stability in Stability & Change Management (scores 0 for this)Stability not yet verified: not enough scan history yet (needs a 30-day window).

Does not apply to this package (2)

  • Endpoint SecurityScored on hosted endpoints only. A package has no live endpoint of its own for us to probe.
  • Transport & ReachabilityScored on hosted endpoints only. A package has no live endpoint of its own for us to probe.

The score does not measure everything. What the score does not measure →

Changelog

Every change we have recorded for this component, newest first. Security-relevant changes are always shown. ▲ marks a change for the better, ▼ a change for the worse; unmarked changes are neutral.

  • 10 Nov 26 +5
    • Source repository: unverified → pass ▲ security
  • 9 Nov 26 −5
    • Provenance: pass → unverified ▼ security
  • 5 Nov 26 +15
    • Malware scan: unverified → pass ▲ security
  • 4 Nov 26 +10
    • Malware scan: pass → unverified ▼ security
    • Injection markers: unverified → pass ▲ security
    • First check of Judged manipulation: pass security
    • Stability: Stability not yet verified: not enough scan history yet (needs a 30-day window). security
    • Tool coverage: unverified → 100 ▲ functional
    • MCP protocol: unverified → pass ▲ functional
    • Schema quality: unverified → 0 ▲ functional
    • First check of Schema quality: fail functional
    • First check of Destructive annotations: 0 functional
    • First check of Schema quality: pass functional
    • First check of Schema quality: good functional
    • First check of Tool coverage: 100 functional
    • Package version: 1.0.0 → 1.0.1 functional
  • 12 Oct 26 0
    • Security disclosure: unverified → pass ▲ functional
  • 11 Oct 26 0
    • Security disclosure: pass → unverified ▼ functional
  • 28 Sept 26 0
    • We updated how we score, so this day's move reflects our rubric, not a change to the server See what changed → functional
  • 25 Sept 26 0
    • We updated how we score, so this day's move reflects our rubric, not a change to the server See what changed → functional
Diagnostics

Diagnostic detail from the automated scan of this channel: what the scanner observed at each step, so you can see exactly where a check passed or failed. It is informational only and never changes the trust score.

Captured 21 Nov 2026 · Analysed npm/mcp-eval-runner@1.0.1

Provenance No attestation

The registry publishes no build provenance for this version, so there is nothing to verify.

Result No attestation
Ecosystem npm

Background: How many MCP packages publish verified provenance →

Dependencies 110 packages
Packages resolved 110
Stale 34
Tree resolution Complete

Background: SBOMs and build attestations, explained →

MCP tools · 10 exposed · ~961 tokens

The tools this component advertises to a client, with an estimated token cost for each. Expand a tool to see its parameters and schema. The per-tool counts are indicative and are not scored directly; the schema's total context footprint is one signal in Schema Quality & AI Usability. A tool's description is untrusted text the model reads on every call, which is what makes this list a security surface and not just an inventory: how tool poisoning works →

Tool Tokens
compare_results ~56

Diff two named run results by run ID. Shows regressions, fixes, new and removed cases.

NameTypeReqDescription
run_id_astringyesFirst run ID to compare
run_id_bstringyesSecond run ID to compare

No output schema declared.

No examples provided.

create_test_case ~150

Create a new YAML fixture file in the fixtures directory. Supported assertion types in each step's expect block: output_contains: "substring" — checks output includes this text output_equals: "exact" — checks exact output match tool_called: "tool_name" — verifies which tool was called latency_under: 500 — asserts response time under N ms schema_match: { type: 'object', ... } — validates output JSON against schema

NameTypeReqDescription
descriptionstring–Optional description for the test case
namestringyesThe name of the new test case
stepsarrayyesArray of steps for the test case

No output schema declared.

No examples provided.

discover_fixtures ~75

Discover fixture files across one or more directories. Returns a list of fixtures with their names, paths, and step counts. If no dirs are provided, uses the configured fixtures directory and FIXTURE_LIBRARY_DIRS env var (colon-separated list of paths).

NameTypeReqDescription
dirsarray–List of directories to scan for fixtures

No output schema declared.

No examples provided.

evaluate_deployment_gate ~117

CI gate — evaluate whether recent eval runs meet a minimum pass rate threshold. Fails (passed: false) if the pass rate of recent runs drops below min_pass_rate. Use this in CI pipelines to block deployments on regressions.

NameTypeReqDescription
lookback_runsinteger–Number of most-recent runs to consider (default: 10)
min_pass_ratenumberyesMinimum acceptable pass rate (0.0 – 1.0)
workflow_namestring–Optional suite/workflow name to filter runs by

No output schema declared.

No examples provided.

generate_html_report ~101

Generate a full single-file HTML report for a completed run. The report includes: suite summary (pass/fail counts, duration), per-case drill-down with per-assertion pass/fail details, and color-coded status (green=pass, red=fail, yellow=error). Use run_suite or run_case first, then pass the returned run_id here.

NameTypeReqDescription
run_idstringyesThe run ID returned by run_suite or run_case

No output schema declared.

No examples provided.

list_cases ~21

Enumerate all available fixtures with their step counts and assertion types.

Input schema present but exposes no named parameters.

No output schema declared.

No examples provided.

regression_report ~29

Compare current state to the last run and return what changed (regressions, fixes, new cases).

Input schema present but exposes no named parameters.

No output schema declared.

No examples provided.

run_case ~100

Run a single named test case from the fixtures directory. Assertion types supported in fixture expect blocks: output_contains: "substring" output_equals: "exact text" tool_called: "tool_name" latency_under: 500 schema_match: { type: "object", required: ["id"], properties: { id: { type: "number" } } }

NameTypeReqDescription
namestringyesThe name of the fixture to run

No output schema declared.

No examples provided.

run_suite ~182

Execute all fixtures in the fixtures directory. Returns a pass/fail summary. Execution modes: Live mode: fixture has a 'server' block with command/args — spawns the server and calls tools via real MCP stdio transport. Simulation mode: no server block — assertions run against expected_output fields. Assertion types supported in fixture expect blocks: output_contains: "substring" output_not_contains: "substring" output_equals: "exact text" output_matches: "regex" tool_called: "tool_name" latency_under: 500 schema_match: { type: "object", required: ["result"], properties: { result: { type: "string" } } } Step output piping: use {{steps.<step_id>.output}} in input values to reference the output of a previous step.

Input schema present but exposes no named parameters.

No output schema declared.

No examples provided.

scaffold_fixture ~130

Generate a boilerplate YAML fixture file in the fixtures directory. Each tool name provided becomes a documented step with placeholder input parameters and all supported assertion types pre-filled as comments. Example: scaffold_fixture({ name: 'search_test', tool_names: ['search', 'summarize'] }) Supported assertion types (pre-filled as TODOs in the generated file): output_contains, output_equals, tool_called, latency_under, schema_match

NameTypeReqDescription
namestringyesName for the new fixture (becomes filename)
tool_namesarrayyesList of tool names — each becomes a fixture step

No output schema declared.

No examples provided.

Install

How do I install the io.github.dbsectrainer/mcp-eval-runner server?

io.github.dbsectrainer/mcp-eval-runner runs locally as an npm package, launched with npx -y mcp-eval-runner. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.

npm · mcp-eval-runner

# add to Claude Code
claude mcp add dbsectrainer-mcp-eval-runner -- npx -y mcp-eval-runner
Common questions

What is the io.github.dbsectrainer/mcp-eval-runner server?

io.github.dbsectrainer/mcp-eval-runner is listed in the public MCP registry as io.github.dbsectrainer/mcp-eval-runner. A standardized testing harness for MCP servers and agent workflows. This page covers its npm package (mcp-eval-runner).

Is the io.github.dbsectrainer/mcp-eval-runner server safe to use?

io.github.dbsectrainer/mcp-eval-runner scores 64 out of 100 on VerifyMCP. We found no known CVEs affecting it as of 21 November 2026. It declares no install or post-install scripts. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.

What tools does the io.github.dbsectrainer/mcp-eval-runner server expose?

io.github.dbsectrainer/mcp-eval-runner exposes 10 tools: run_suite, run_case, list_cases, create_test_case, regression_report, and 5 more. Their descriptions and schemas cost roughly 961 tokens of context every time the server is loaded.

Is the io.github.dbsectrainer/mcp-eval-runner server still maintained?

io.github.dbsectrainer/mcp-eval-runner is still listed as active in the MCP registry. We last reached this channel on 21 November 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.

What licence is the io.github.dbsectrainer/mcp-eval-runner server under?

io.github.dbsectrainer/mcp-eval-runner declares the MIT licence, which is OSI-approved. That covers the source only, and says nothing about the cost of any service it calls.

What did VerifyMCP test on the io.github.dbsectrainer/mcp-eval-runner server?

VerifyMCP ran 15 checks on io.github.dbsectrainer/mcp-eval-runner, covering Malware scan, Known CVEs, Install scripts, Dependency health and 11 more. One item could not be completed, a gap on our side rather than a finding about io.github.dbsectrainer/mcp-eval-runner: Stability. One category scores 0 for that reason, because we only credit what we can confirm. Endpoint Security and Transport & Reachability do not apply to an npm package, so they were not tested. The score does not measure everything, and the scoring documentation lists what it leaves out.