# LLM Eval Harness (oci · ghcr.io/razilzul/llm-eval-harness)

Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.

- Trust score: 39/100 (low)
- Registry status: active
- Liveness: live
- Owner verified: no
- Last scored: 2026-10-18

## Components

- oci · `ghcr.io/razilzul/llm-eval-harness`: 39/100 (this document), [markdown](https://verifymcp.io/servers/razilzul-llm-eval-harness/ghcr-io-razilzul-llm-eval-harness.md), [page](https://verifymcp.io/servers/razilzul-llm-eval-harness/ghcr-io-razilzul-llm-eval-harness)

## Channel facts

- Registry: `oci`
- Package: `ghcr.io/razilzul/llm-eval-harness`
- Version: `1.0.0`
- Transport: `stdio`

## Trust breakdown

How this component scores in each security and reliability category. Every signal is checked automatically from public evidence about the published package, including repeated runs of it in an isolated sandbox, and we only credit what we can confirm. Scores are 0–100 per category. Scoring method: https://verifymcp.io/docs/scoring (what has changed: https://verifymcp.io/docs/scoring/changelog)

Scored 2026-10-18.

- **Supply Chain Security**: 0/100
  - No malware scan is available for this kind of package: the supply-chain vendors we use do not cover it. This is a permanent gap in our coverage, not a finding about the package.
  - Known CVEs could not be checked: this artifact ships no SBOM or dependency manifest, so there is no dependency list to read.
  - Install-script risk not yet assessed.
  - Dependency health could not be checked: this artifact ships no SBOM or dependency manifest, so there is no dependency list to read.
- **Provenance & Transparency**: 45/100
  - Source repository is publicly reachable at the declared URL.
  - Provenance check failed: no build-provenance attestation is published.
  - Clear OSI-approved license (MIT).
  - Actively maintained (last published 0 days ago).
  - Disclosure check failed: no security disclosure policy was found in the source repository.
- **Schema Quality & AI Usability**: 74/100
  - AI-judged instruction clarity (good).
  - Context-footprint check failed: tool/resource definitions use about 333 tokens (~166/item across 2 items; 2 tools + 0 resources), over budget; trim descriptions and params.
  - Usage-examples check failed: none of the tools include examples.
- **Stability & Change Management**: 0/100
  - Stability observed for 0 of 30 days with no destabilising changes; credit accrues until the full window elapses.
- **Tool Coverage**: 100/100
  - 100% of tools have a non-trivial description (not blank, and not just the tool's name).
  - 100% of tool parameters carry a description.
- **Tool Safety**: 100/100
  - No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.
  - We read all 2 captured tool definition(s), and no name or description among them implies an irreversible operation.
  - An AI judge read all 2 captured unit(s) of tool text and found none that tries to manipulate the model reading it.
- **Capabilities**: 60/100
  - Spec-recency check failed: implements MCP spec 2025-06-18; the latest is 2026-07-28.

**Unverified: 2 categories.** Categories scored 0 because we could not verify them: a data source with nothing on this package, evidence we could not reach, or a check we could not run. We only credit what we can confirm.

## Install

### How do I install the LLM Eval Harness MCP server?

LLM Eval Harness runs locally as a container image, launched with docker run --rm -i -e ANTHROPIC_API_KEY -e GEMINI_API_KEY ghcr.io/razilzul/llm-eval-harness:1.0.0. Ready-made configuration for Claude, Cursor, VS Code, Codex and 3 more is on this page, copied from each client's own documentation.

### Claude

```bash
claude mcp add razilzul-llm-eval-harness -- docker run --rm -i -e ANTHROPIC_API_KEY -e GEMINI_API_KEY ghcr.io/razilzul/llm-eval-harness:1.0.0
```

This image reads ANTHROPIC_API_KEY and GEMINI_API_KEY. Set them in your client's env block for this server; docker run -e passes each one through to the container.

### Cursor

```json
{
  "mcpServers": {
    "razilzul-llm-eval-harness": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "-e",
        "ANTHROPIC_API_KEY",
        "-e",
        "GEMINI_API_KEY",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ]
    }
  }
}
```

This image reads ANTHROPIC_API_KEY and GEMINI_API_KEY. Set them in your client's env block for this server; docker run -e passes each one through to the container.

### VS Code

```json
{
  "servers": {
    "razilzul-llm-eval-harness": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "-e",
        "ANTHROPIC_API_KEY",
        "-e",
        "GEMINI_API_KEY",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ]
    }
  }
}
```

This image reads ANTHROPIC_API_KEY and GEMINI_API_KEY. Set them in your client's env block for this server; docker run -e passes each one through to the container.

### Codex

```bash
codex mcp add razilzul-llm-eval-harness -- docker run --rm -i -e ANTHROPIC_API_KEY -e GEMINI_API_KEY ghcr.io/razilzul/llm-eval-harness:1.0.0
```

This image reads ANTHROPIC_API_KEY and GEMINI_API_KEY. Set them in your client's env block for this server; docker run -e passes each one through to the container.

### opencode

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "razilzul-llm-eval-harness": {
      "type": "local",
      "command": [
        "docker",
        "run",
        "--rm",
        "-i",
        "-e",
        "ANTHROPIC_API_KEY",
        "-e",
        "GEMINI_API_KEY",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ],
      "enabled": true
    }
  }
}
```

This image reads ANTHROPIC_API_KEY and GEMINI_API_KEY. Set them in your client's env block for this server; docker run -e passes each one through to the container.

### Hermes

```yaml
mcp_servers:
  razilzul-llm-eval-harness:
    command: "docker"
    args: ["run", "--rm", "-i", "-e", "ANTHROPIC_API_KEY", "-e", "GEMINI_API_KEY", "ghcr.io/razilzul/llm-eval-harness:1.0.0"]
```

This image reads ANTHROPIC_API_KEY and GEMINI_API_KEY. Set them in your client's env block for this server; docker run -e passes each one through to the container.

### Netclaw

```json
{
  "McpServers": {
    "razilzul-llm-eval-harness": {
      "Transport": "stdio",
      "Command": "docker",
      "Arguments": [
        "run",
        "--rm",
        "-i",
        "-e",
        "ANTHROPIC_API_KEY",
        "-e",
        "GEMINI_API_KEY",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ]
    }
  }
}
```

This image reads ANTHROPIC_API_KEY and GEMINI_API_KEY. Set them in your client's env block for this server; docker run -e passes each one through to the container.

### Other

```json
{
  "mcpServers": {
    "razilzul-llm-eval-harness": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "-e",
        "ANTHROPIC_API_KEY",
        "-e",
        "GEMINI_API_KEY",
        "ghcr.io/razilzul/llm-eval-harness:1.0.0"
      ]
    }
  }
}
```

This image reads ANTHROPIC_API_KEY and GEMINI_API_KEY. Set them in your client's env block for this server; docker run -e passes each one through to the container.

## Changelog

Every change recorded for this component, newest first. Days that predate change tracking, or that we cannot explain, say so: "we were watching and nothing happened" and "we were not watching" are different claims.

### 2026-10-18 (score 39)

First indexed and scored.

## MCP tools (2)

### `run_golden_set` (~134 tokens)

Run one model over the golden set. Returns accuracy overall and on trap cases, mean latency and cost per request, and the failing cases.

Input parameters:

- `judge` (string): Optional judge model, same format. Default mock:judge (exact label match, free). Use a real model for the LLM-as-judge rubric.
- `model` (string, required): Model to evaluate, as provider:model, e.g. anthropic:claude-haiku-4-5, gemini:gemini-2.5-flash, mock:strong. Real models bill the API key in .env or the client config.

### `compare_models` (~199 tokens)

A/B two models on the golden set and return the gate verdict. The candidate must match or beat the baseline on accuracy, overall and on traps; only then do latency and cost matter.

Input parameters:

- `baseline` (string, required): Current model, as provider:model, e.g. anthropic:claude-haiku-4-5, gemini:gemini-2.5-flash, mock:strong. Real models bill the API key in .env or the client config.
- `candidate` (string, required): Proposed replacement, as provider:model, e.g. anthropic:claude-haiku-4-5, gemini:gemini-2.5-flash, mock:strong. Real models bill the API key in .env or the client config.
- `judge` (string): Optional judge model, same format. Default mock:judge (exact label match, free). Use a real model for the LLM-as-judge rubric.

## Diagnostics

Captured diagnostic sections: Provenance. The full working is on the page: https://verifymcp.io/servers/razilzul-llm-eval-harness/ghcr-io-razilzul-llm-eval-harness#diagnostics

## Score history

- 2026-10-18: 39

## Common questions

### What is the LLM Eval Harness MCP server?

LLM Eval Harness is an MCP server listed in the public MCP registry as io.github.razilzul/llm-eval-harness. Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request. This page covers its container image (ghcr.io/razilzul/llm-eval-harness).

### Is the LLM Eval Harness MCP server safe to use?

LLM Eval Harness scores 39 out of 100 on VerifyMCP. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.

### What tools does the LLM Eval Harness MCP server expose?

LLM Eval Harness exposes 2 tools: run_golden_set, compare_models. Their descriptions and schemas cost roughly 333 tokens of context every time the server is loaded.

### Is the LLM Eval Harness MCP server still maintained?

LLM Eval Harness is still listed as active in the MCP registry. We last reached this channel on 18 October 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.

### What licence is the LLM Eval Harness MCP server under?

LLM Eval Harness declares the MIT licence, which is OSI-approved. That covers the source only, and says nothing about the cost of any service it calls.

## Links

- GitHub Container Registry: https://ghcr.io/razilzul/llm-eval-harness
- Repository: https://github.com/razilzul/llm-eval-harness
- Changelog RSS feed: https://verifymcp.io/servers/razilzul-llm-eval-harness/ghcr-io-razilzul-llm-eval-harness.xml
- Changelog JSON feed: https://verifymcp.io/servers/razilzul-llm-eval-harness/ghcr-io-razilzul-llm-eval-harness.json
- HTML version of this page: https://verifymcp.io/servers/razilzul-llm-eval-harness/ghcr-io-razilzul-llm-eval-harness