retriEVAL
OCI · GHCR.IO/HCARRILLO001/RETRIEVAL-MCP:1.0.0
LLM evals as MCP tools: score outputs for faithfulness, relevancy, and hallucination.
Available components
Last scanned (23 hours ago)
Scores higher than 71% of OCI servers
Category Testing and QA
How this component scores in each security and reliability category. Every signal is checked automatically from public evidence about the published package, including repeated runs of it in an isolated sandbox, and we only credit what we can confirm. How we score → Why this is hard to score →
Supply Chain Security0
- No malware scan is available for this kind of package: the supply-chain vendors we use do not cover it. This is a permanent gap in our coverage, not a finding about the package.Unverified
- Known CVEs could not be checked: this artifact ships no SBOM or dependency manifest, so there is no dependency list to read.Unverified
- Install-script risk not yet assessed.Unverified
- Dependency health could not be checked: this artifact ships no SBOM or dependency manifest, so there is no dependency list to read.Unverified
Provenance & Transparency32
- Source repository is publicly reachable at the declared URL. View diagnostics → Pass
- Provenance check failed: no build-provenance attestation is published. See how to fix → View diagnostics → Fail
- License check failed: no license is declared. See how to fix → Fail
- Actively maintained (last published 72 days ago).Pass
- Disclosure check failed: no security disclosure policy was found in the source repository. See how to fix → Fail
Schema Quality & AI Usability75
- AI-judged instruction clarity (good).Pass
- Tool/resource definitions use about 1320 tokens (~94/item across 14 items; 14 tools + 0 resources), lean.Pass
- Usage-examples check failed: none of the tools include examples. See how to fix → Fail
Stability & Change Management100
- No destabilizing schema changes in the last 30 days.Pass
Tool Coverage67
- 100% of tools have a non-trivial description (not blank, and not just the tool's name).Pass
- 0% of tool parameters carry a description.Fail
Tool Safety75
- No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.Pass
- 0 of 1 tool(s) whose name or description implies an irreversible operation declare an MCP destructiveHint annotation; "run_eval" implies "eval" and declares no destructiveHint at all, which the MCP spec reads as destructive by default. See how to fix → Fail
- An AI judge read all 14 captured unit(s) of tool text and found none that tries to manipulate the model reading it.Pass
Capabilities100
- Implements a supported MCP spec version (2025-11-25); the latest is 2026-07-28.Pass
Unverified: 1 category
A category scored 0 because we could not verify it: a data source with nothing on this package, evidence we could not reach, or a check we could not run. We only credit what we can confirm.
Every change we have recorded for this component, newest first. Security-relevant changes are always shown. ▲ marks a change for the better, ▼ a change for the worse; unmarked changes are neutral.
- 22 Oct 26 +1
- Stability: 0.97 → pass security
- 20 Oct 26 −1
- Stability: pass → 0.93 functional
- 17 Oct 26 +3
- Stability: 0.80 → pass security
- 15 Oct 26 −3
- Stability: pass → 0.77 functional
- 14 Oct 26 +1
- Stability: 0.97 → pass security
- 12 Oct 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 87 to 93. That category is still filling its 30-day observation window: 26 days of observed history at the previous scan, 28 at this one. The score rises as the window fills, whether or not the server changes.
- 10 Oct 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 83 to 87. That category is still filling its 30-day observation window: 25 days of observed history at the previous scan, 26 at this one. The score rises as the window fills, whether or not the server changes.
- 9 Oct 26 −3
- Stability: pass → 0.83 functional
Diagnostic detail from the automated scan of this channel: what the scanner observed at each step, so you can see exactly where a check passed or failed. It is informational only and never changes the trust score.
Captured 4 Nov 2026 · Analysed oci/ghcr.io/hcarrillo001/retrieval-mcp:1.0.0
Provenance No attestation
The registry publishes no build provenance for this version, so there is nothing to verify.
| Result | No attestation |
|---|---|
| Ecosystem | oci |
| Reason | No attestation published |
Background: How many MCP packages publish verified provenance →
The tools this component advertises to a client, with an estimated token cost for each. Expand a tool to see its parameters and schema. The per-tool counts are indicative and are not scored directly; the schema's total context footprint is one signal in Schema Quality & AI Usability. A tool's description is untrusted text the model reads on every call, which is what makes this list a security surface and not just an inventory: how tool poisoning works →
author_metric ~70
Author a custom metric from a plain-language description (+ optional golden examples). The judge converts your criteria into explicit evaluation steps, which run as a G-Eval-style scorer in run_eval / evaluate_case.
| Name | Type | Req | Description |
|---|---|---|---|
| criteria | string | yes | – |
| examples | – | – | – |
| name | string | yes | – |
No output schema declared.
No examples provided.
compare_runs ~85
Compare several runs across all shared metrics. fmt="markdown" (default) returns a markdown matrix; fmt="text" returns plain ASCII; fmt="image" returns a PNG grouped-bar chart.
| Name | Type | Req | Description |
|---|---|---|---|
| fmt | string | – | – |
| golden_set | string | – | – |
| last_n | integer | – | – |
| run_ids | – | – | – |
| threshold | number | – | – |
No output schema declared.
No examples provided.
evaluate_case ~110
Score a single output inline, without loading a golden set. The result is saved as a one-case run (golden set "inline") so it has a dashboard link; pass save=False to score without recording it.
| Name | Type | Req | Description |
|---|---|---|---|
| actual_output | string | yes | – |
| context | – | – | – |
| expected_output | string | – | – |
| input | string | yes | – |
| metrics | array | yes | – |
| retrieval_context | – | – | – |
| save | boolean | – | – |
| threshold | number | – | – |
No output schema declared.
No examples provided.
get_budget ~30
Current judge spend, the cap (RETRIEVAL_BUDGET_USD), and remaining headroom.
Input schema present but exposes no named parameters.
No output schema declared.
No examples provided.
ground_against_url ~164
Fetch a web page and check OUTPUT for *consistency* with it: faithfulness (every claim supported by the page) and, if a question is given, answer relevancy. Use it to ask "does my answer agree with what this page says?". IMPORTANT — this is NOT a correctness check. There are no labels here: the page is an unverified source, not ground truth. A high score means the output matches the page, not that either is right. The returned `caveat` says so on every call; surface it to the user.
| Name | Type | Req | Description |
|---|---|---|---|
| output | string | yes | – |
| question | string | – | – |
| threshold | number | – | – |
| url | string | yes | – |
No output schema declared.
No examples provided.
list_golden_sets ~18
Show loaded golden sets and their sizes.
Input schema present but exposes no named parameters.
No output schema declared.
No examples provided.
list_metrics ~20
List built-in metrics and any custom metrics authored this session.
Input schema present but exposes no named parameters.
No output schema declared.
No examples provided.
list_runs ~40
List saved runs (most recent last) with their per-metric mean scores.
| Name | Type | Req | Description |
|---|---|---|---|
| golden_set | string | – | – |
| last_n | integer | – | – |
No output schema declared.
No examples provided.
load_golden_set ~84
Load a golden set from a file path, a JSON array string, or JSONL text. Field names are normalized (question->input, answer->actual_output, ground_truth->expected_output, contexts->context, passages->retrieval_context, ...).
| Name | Type | Req | Description |
|---|---|---|---|
| fmt | string | – | – |
| name | string | yes | – |
| source | string | yes | – |
No output schema declared.
No examples provided.
plot_metric_trend ~119
One metric's mean score across recent runs, with a threshold reference. fmt="markdown" (default) returns a rich markdown report (tables, inline bars, case strip, distribution); fmt="text" returns plain ASCII; fmt="image" returns a PNG line chart (only useful where images display).
| Name | Type | Req | Description |
|---|---|---|---|
| fmt | string | – | – |
| golden_set | string | – | – |
| last_n | integer | – | – |
| metric | string | yes | – |
| show_range | boolean | – | – |
| threshold | number | – | – |
No output schema declared.
No examples provided.
plot_run ~102
Every metric's mean score for a single run (default: latest). fmt="markdown" (default) returns a rich markdown report (tables, inline bars, case strip, distribution); fmt="text" returns plain ASCII; fmt="image" returns a PNG bar chart (only useful where images display).
| Name | Type | Req | Description |
|---|---|---|---|
| fmt | string | – | – |
| golden_set | string | – | – |
| run_id | string | – | – |
| threshold | number | – | – |
No output schema declared.
No examples provided.
reset_budget ~23
Reset the cumulative spend ledger to $0 (the cap itself is unchanged).
Input schema present but exposes no named parameters.
No output schema declared.
No examples provided.
run_eval ~382
Run metrics over a set of cases. Pass `cases` with the data inline (JSON, JSONL, CSV or TSV text, or a file path) to score in a single call — nothing is stored server-side and no prior load is needed. Or pass `golden_set` to reuse a set loaded earlier with load_golden_set. Inline `cases` is preferred: named sets live only in this server process, are shared with anyone else using it, and are lost when it restarts. All cases are scored, but only the `limit` lowest-scoring cases are returned by default (3) to keep replies short; `total_cases`/`shown` tell you how many more exist — call show_run_cases to page. Pass `generator_model` (which LLM produced the outputs) and `judge_model` (which LLM scored them) so the dashboard can compare across models; judge_model defaults to the configured judge. The run is saved to history (file or Supabase). IMPORTANT FOR CALLERS: the reply includes `view_url`, a link to the full visual report. ALWAYS include that link in your response to the user, even when you summarise everything else. `summary_md` is a ready-to-render markdown block (score table + link) that can be shown verbatim. Raises if the spend cap is hit mid-run; partial spend is still metered.
| Name | Type | Req | Description |
|---|---|---|---|
| cases | string | – | – |
| generator_model | string | – | – |
| golden_set | string | – | – |
| judge_model | string | – | – |
| label | string | – | – |
| limit | integer | – | – |
| metrics | array | yes | – |
| outputs | – | – | – |
| threshold | number | – | – |
No output schema declared.
No examples provided.
show_run_cases ~73
Page through the full per-case results of a saved run (the cases run_eval didn't show). Optionally sort by a single metric's score (worst first).
| Name | Type | Req | Description |
|---|---|---|---|
| limit | integer | – | – |
| metric | string | – | – |
| offset | integer | – | – |
| run_id | string | – | – |
No output schema declared.
No examples provided.
How do I install the retriEVAL MCP server?
retriEVAL runs locally as a container image, launched with docker run --rm -i -e ANTHROPIC_API_KEY -e RETRIEVAL_JUDGE_BACKEND -e RETRIEVAL_JUDGE_MODEL -e RETRIEVAL_BUDGET_USD ghcr.io/hcarrillo001/retrieval-mcp:1.0.0. Ready-made configuration for Claude, Cursor, VS Code, Codex and 3 more is on this page, copied from each client's own documentation.
oci · ghcr.io/hcarrillo001/retrieval-mcp:1.0.0
claude mcp add hcarrillo001-retrieval-mcp -- docker run --rm -i -e ANTHROPIC_API_KEY -e RETRIEVAL_JUDGE_BACKEND -e RETRIEVAL_JUDGE_MODEL -e RETRIEVAL_BUDGET_USD ghcr.io/hcarrillo001/retrieval-mcp:1.0.0
This image reads ANTHROPIC_API_KEY, RETRIEVAL_JUDGE_BACKEND, RETRIEVAL_JUDGE_MODEL and RETRIEVAL_BUDGET_USD. Set them in your client's env block for this server; docker run -e passes each one through to the container.
What is the retriEVAL MCP server?
retriEVAL is an MCP server listed in the public MCP registry as io.github.hcarrillo001/retrieval-mcp. LLM evals as MCP tools: score outputs for faithfulness, relevancy, and hallucination. This page covers its container image (ghcr.io/hcarrillo001/retrieval-mcp:1.0.0).
Is the retriEVAL MCP server safe to use?
retriEVAL scores 48 out of 100 on VerifyMCP. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.
What tools does the retriEVAL MCP server expose?
retriEVAL exposes 14 tools: list_metrics, load_golden_set, list_golden_sets, author_metric, run_eval, and 9 more. Their descriptions and schemas cost roughly 1,320 tokens of context every time the server is loaded.
Is the retriEVAL MCP server still maintained?
retriEVAL is still listed as active in the MCP registry. We last reached this channel on 4 November 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.