io.github.dbsectrainer/mcp-eval-runner
NPM · MCP-EVAL-RUNNER
A standardized testing harness for MCP servers and agent workflows
Available components
Last scanned (17 hours ago)
Scores higher than 37% of npm servers
Category Testing and QA
How this component scores in each security and reliability category. Every signal is checked automatically from public evidence about the published package, including repeated runs of it in an isolated sandbox, and we only credit what we can confirm. How we score → Why this is hard to score →
Supply Chain Security98
- No malware found by supply-chain analysis.Pass
- No known CVEs affecting this package version or its production dependencies.Pass
- No install/post-install scripts declared.Pass
- 34 of 110 dependencies flagged as unhealthy. View diagnostics → Partial
Provenance & Transparency48
- Source repository is publicly reachable at the declared URL. View diagnostics → Pass
- Provenance check failed: no build-provenance attestation is published. See how to fix → View diagnostics → Fail
- Clear OSI-approved license (MIT).Pass
- Actively maintained (last published 17 days ago).Pass
- Publishes a security disclosure policy (SECURITY.md).Pass
Schema Quality & AI Usability48
- 0% of prompts and resources have a non-trivial description (not blank, and not just the item's name).Fail
- AI-judged instruction clarity (good).Pass
- Tool/resource definitions use about 961 tokens (~96/item across 10 items; 10 tools + 0 resources), lean.Pass
- Usage-examples check failed: none of the tools include examples. See how to fix → Fail
Stability & Change Management0
- Stability not yet verified: not enough scan history yet (needs a 30-day window).Unverified
Tool Coverage100
- 100% of tools have a non-trivial description (not blank, and not just the tool's name).Pass
- 100% of tool parameters carry a description.Pass
Tool Safety75
- No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.Pass
- 0 of 1 tool(s) whose name or description implies an irreversible operation declare an MCP destructiveHint annotation; "run_suite" implies "execute" and declares no destructiveHint at all, which the MCP spec reads as destructive by default. See how to fix → Fail
- An AI judge read all 10 captured unit(s) of tool text and found none that tries to manipulate the model reading it.Pass
Capabilities100
- Implements a supported MCP spec version (2025-11-25); the latest is 2026-07-28.Pass
Unverified: 1 category
A category scored 0 because we could not verify it: a data source with nothing on this package, evidence we could not reach, or a check we could not run. We only credit what we can confirm.
Checks we ran (15)
- Malware scan
- Known CVEs
- Install scripts
- Dependency health
- Source repository
- Provenance
- License
- Maintenance
- Security disclosure
- Schema quality
- Tool coverage
- Injection markers
- Destructive annotations
- Judged manipulation
- MCP protocol
Checks we could not complete (1)
These are gaps on our side, not findings about this package. We only credit what we can confirm, so a category made up entirely of them scores 0.
- Stability in Stability & Change Management (scores 0 for this)Stability not yet verified: not enough scan history yet (needs a 30-day window).
Does not apply to this package (2)
- Endpoint SecurityScored on hosted endpoints only. A package has no live endpoint of its own for us to probe.
- Transport & ReachabilityScored on hosted endpoints only. A package has no live endpoint of its own for us to probe.
The score does not measure everything. What the score does not measure →
Every change we have recorded for this component, newest first. Security-relevant changes are always shown. ▲ marks a change for the better, ▼ a change for the worse; unmarked changes are neutral.
- 10 Nov 26 +5
- Source repository: unverified → pass ▲ security
- 9 Nov 26 −5
- Provenance: pass → unverified ▼ security
- 5 Nov 26 +15
- Malware scan: unverified → pass ▲ security
- 4 Nov 26 +10
- Malware scan: pass → unverified ▼ security
- Injection markers: unverified → pass ▲ security
- First check of Judged manipulation: pass security
- Stability: Stability not yet verified: not enough scan history yet (needs a 30-day window). security
- Tool coverage: unverified → 100 ▲ functional
- MCP protocol: unverified → pass ▲ functional
- Schema quality: unverified → 0 ▲ functional
- First check of Schema quality: fail functional
- First check of Destructive annotations: 0 functional
- First check of Schema quality: pass functional
- First check of Schema quality: good functional
- First check of Tool coverage: 100 functional
- Package version: 1.0.0 → 1.0.1 functional
- 12 Oct 26 0
- Security disclosure: unverified → pass ▲ functional
- 11 Oct 26 0
- Security disclosure: pass → unverified ▼ functional
- 28 Sept 26 0
- We updated how we score, so this day's move reflects our rubric, not a change to the server See what changed → functional
- 25 Sept 26 0
- We updated how we score, so this day's move reflects our rubric, not a change to the server See what changed → functional
Diagnostic detail from the automated scan of this channel: what the scanner observed at each step, so you can see exactly where a check passed or failed. It is informational only and never changes the trust score.
Captured 21 Nov 2026 · Analysed npm/mcp-eval-runner@1.0.1
Provenance No attestation
The registry publishes no build provenance for this version, so there is nothing to verify.
| Result | No attestation |
|---|---|
| Ecosystem | npm |
Background: How many MCP packages publish verified provenance →
Dependencies 110 packages
| Packages resolved | 110 |
|---|---|
| Stale | 34 |
| Tree resolution | Complete |
Background: SBOMs and build attestations, explained →
The tools this component advertises to a client, with an estimated token cost for each. Expand a tool to see its parameters and schema. The per-tool counts are indicative and are not scored directly; the schema's total context footprint is one signal in Schema Quality & AI Usability. A tool's description is untrusted text the model reads on every call, which is what makes this list a security surface and not just an inventory: how tool poisoning works →
compare_results ~56
Diff two named run results by run ID. Shows regressions, fixes, new and removed cases.
| Name | Type | Req | Description |
|---|---|---|---|
| run_id_a | string | yes | First run ID to compare |
| run_id_b | string | yes | Second run ID to compare |
No output schema declared.
No examples provided.
create_test_case ~150
Create a new YAML fixture file in the fixtures directory. Supported assertion types in each step's expect block: output_contains: "substring" — checks output includes this text output_equals: "exact" — checks exact output match tool_called: "tool_name" — verifies which tool was called latency_under: 500 — asserts response time under N ms schema_match: { type: 'object', ... } — validates output JSON against schema
| Name | Type | Req | Description |
|---|---|---|---|
| description | string | – | Optional description for the test case |
| name | string | yes | The name of the new test case |
| steps | array | yes | Array of steps for the test case |
No output schema declared.
No examples provided.
discover_fixtures ~75
Discover fixture files across one or more directories. Returns a list of fixtures with their names, paths, and step counts. If no dirs are provided, uses the configured fixtures directory and FIXTURE_LIBRARY_DIRS env var (colon-separated list of paths).
| Name | Type | Req | Description |
|---|---|---|---|
| dirs | array | – | List of directories to scan for fixtures |
No output schema declared.
No examples provided.
evaluate_deployment_gate ~117
CI gate — evaluate whether recent eval runs meet a minimum pass rate threshold. Fails (passed: false) if the pass rate of recent runs drops below min_pass_rate. Use this in CI pipelines to block deployments on regressions.
| Name | Type | Req | Description |
|---|---|---|---|
| lookback_runs | integer | – | Number of most-recent runs to consider (default: 10) |
| min_pass_rate | number | yes | Minimum acceptable pass rate (0.0 – 1.0) |
| workflow_name | string | – | Optional suite/workflow name to filter runs by |
No output schema declared.
No examples provided.
generate_html_report ~101
Generate a full single-file HTML report for a completed run. The report includes: suite summary (pass/fail counts, duration), per-case drill-down with per-assertion pass/fail details, and color-coded status (green=pass, red=fail, yellow=error). Use run_suite or run_case first, then pass the returned run_id here.
| Name | Type | Req | Description |
|---|---|---|---|
| run_id | string | yes | The run ID returned by run_suite or run_case |
No output schema declared.
No examples provided.
list_cases ~21
Enumerate all available fixtures with their step counts and assertion types.
Input schema present but exposes no named parameters.
No output schema declared.
No examples provided.
regression_report ~29
Compare current state to the last run and return what changed (regressions, fixes, new cases).
Input schema present but exposes no named parameters.
No output schema declared.
No examples provided.
run_case ~100
Run a single named test case from the fixtures directory. Assertion types supported in fixture expect blocks: output_contains: "substring" output_equals: "exact text" tool_called: "tool_name" latency_under: 500 schema_match: { type: "object", required: ["id"], properties: { id: { type: "number" } } }
| Name | Type | Req | Description |
|---|---|---|---|
| name | string | yes | The name of the fixture to run |
No output schema declared.
No examples provided.
run_suite ~182
Execute all fixtures in the fixtures directory. Returns a pass/fail summary. Execution modes: Live mode: fixture has a 'server' block with command/args — spawns the server and calls tools via real MCP stdio transport. Simulation mode: no server block — assertions run against expected_output fields. Assertion types supported in fixture expect blocks: output_contains: "substring" output_not_contains: "substring" output_equals: "exact text" output_matches: "regex" tool_called: "tool_name" latency_under: 500 schema_match: { type: "object", required: ["result"], properties: { result: { type: "string" } } } Step output piping: use {{steps.<step_id>.output}} in input values to reference the output of a previous step.
Input schema present but exposes no named parameters.
No output schema declared.
No examples provided.
scaffold_fixture ~130
Generate a boilerplate YAML fixture file in the fixtures directory. Each tool name provided becomes a documented step with placeholder input parameters and all supported assertion types pre-filled as comments. Example: scaffold_fixture({ name: 'search_test', tool_names: ['search', 'summarize'] }) Supported assertion types (pre-filled as TODOs in the generated file): output_contains, output_equals, tool_called, latency_under, schema_match
| Name | Type | Req | Description |
|---|---|---|---|
| name | string | yes | Name for the new fixture (becomes filename) |
| tool_names | array | yes | List of tool names — each becomes a fixture step |
No output schema declared.
No examples provided.
How do I install the io.github.dbsectrainer/mcp-eval-runner server?
io.github.dbsectrainer/mcp-eval-runner runs locally as an npm package, launched with npx -y mcp-eval-runner. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.
npm · mcp-eval-runner
claude mcp add dbsectrainer-mcp-eval-runner -- npx -y mcp-eval-runner
What is the io.github.dbsectrainer/mcp-eval-runner server?
io.github.dbsectrainer/mcp-eval-runner is listed in the public MCP registry as io.github.dbsectrainer/mcp-eval-runner. A standardized testing harness for MCP servers and agent workflows. This page covers its npm package (mcp-eval-runner).
Is the io.github.dbsectrainer/mcp-eval-runner server safe to use?
io.github.dbsectrainer/mcp-eval-runner scores 64 out of 100 on VerifyMCP. We found no known CVEs affecting it as of 21 November 2026. It declares no install or post-install scripts. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.
What tools does the io.github.dbsectrainer/mcp-eval-runner server expose?
io.github.dbsectrainer/mcp-eval-runner exposes 10 tools: run_suite, run_case, list_cases, create_test_case, regression_report, and 5 more. Their descriptions and schemas cost roughly 961 tokens of context every time the server is loaded.
Is the io.github.dbsectrainer/mcp-eval-runner server still maintained?
io.github.dbsectrainer/mcp-eval-runner is still listed as active in the MCP registry. We last reached this channel on 21 November 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.
What licence is the io.github.dbsectrainer/mcp-eval-runner server under?
io.github.dbsectrainer/mcp-eval-runner declares the MIT licence, which is OSI-approved. That covers the source only, and says nothing about the cost of any service it calls.
What did VerifyMCP test on the io.github.dbsectrainer/mcp-eval-runner server?
VerifyMCP ran 15 checks on io.github.dbsectrainer/mcp-eval-runner, covering Malware scan, Known CVEs, Install scripts, Dependency health and 11 more. One item could not be completed, a gap on our side rather than a finding about io.github.dbsectrainer/mcp-eval-runner: Stability. One category scores 0 for that reason, because we only credit what we can confirm. Endpoint Security and Transport & Reachability do not apply to an npm package, so they were not tested. The score does not measure everything, and the scoring documentation lists what it leaves out.