# EvalForge Lite (pypi · evalforge-lite)

Compare text LLMs across OpenRouter, Bedrock, Vertex AI and Foundry with automated grading.

- Trust score: 75/100 (medium)
- Change this week: +7
- Registry status: active
- Liveness: live
- Owner verified: no
- Last scored: 2026-10-17

## Components

- pypi · `evalforge-lite`: 75/100 (this document), [markdown](https://verifymcp.io/servers/thejaredchapman-evalforge-lite/evalforge-lite.md), [page](https://verifymcp.io/servers/thejaredchapman-evalforge-lite/evalforge-lite)

## Channel facts

- Registry: `pypi`
- Package: `evalforge-lite`
- Version: `1.0.1`
- Transport: `stdio`

## Trust breakdown

How this component scores in each security and reliability category. Every signal is checked automatically from public evidence about the published package, including repeated runs of it in an isolated sandbox, and we only credit what we can confirm. Scores are 0–100 per category. Scoring method: https://verifymcp.io/docs/scoring (what has changed: https://verifymcp.io/docs/scoring/changelog)

Scored 2026-10-17.

- **Supply Chain Security**: 100/100
  - No malware found by supply-chain analysis.
  - No known CVEs affecting this package version or its production dependencies.
  - Runs setuptools.build_meta at install time, a recognised build step with no custom scripting around it.
  - 5 of 64 dependencies flagged as unhealthy.
- **Provenance & Transparency**: 45/100
  - Source repository is publicly reachable at the declared URL.
  - Provenance check failed: no build-provenance attestation is published.
  - Clear OSI-approved license (MIT).
  - Actively maintained (last published 2 days ago).
  - Disclosure check failed: no security disclosure policy was found in the source repository.
- **Schema Quality & AI Usability**: 73/100
  - AI-judged instruction clarity (good).
  - Tool/resource definitions use about 938 tokens (~104/item across 9 items; 9 tools + 0 resources), lean.
  - Usage-examples check failed: none of the tools include examples.
- **Stability & Change Management**: 53/100
  - Stability observed for 16 of 30 days with no destabilising changes; credit accrues until the full window elapses.
- **Tool Coverage**: 67/100
  - 100% of tools have a non-trivial description (not blank, and not just the tool's name).
  - 0% of tool parameters carry a description.
- **Tool Safety**: 100/100
  - No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.
  - We read all 9 captured tool definition(s), and no name or description among them implies an irreversible operation.
  - An AI judge read all 9 captured unit(s) of tool text and found none that tries to manipulate the model reading it.
- **Capabilities**: 100/100
  - Implements a current MCP spec version (2026-07-28).

## Install

### How do I install the EvalForge Lite MCP server?

EvalForge Lite runs locally as a PyPI package, launched with uvx evalforge-lite. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.

### Claude

```bash
claude mcp add thejaredchapman-evalforge-lite -- uvx evalforge-lite
```

### Cursor

```json
{
  "mcpServers": {
    "thejaredchapman-evalforge-lite": {
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ]
    }
  }
}
```

### VS Code

```json
{
  "servers": {
    "thejaredchapman-evalforge-lite": {
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ]
    }
  }
}
```

### Codex

```bash
codex mcp add thejaredchapman-evalforge-lite -- uvx evalforge-lite
```

### opencode

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "thejaredchapman-evalforge-lite": {
      "type": "local",
      "command": [
        "uvx",
        "evalforge-lite"
      ],
      "enabled": true
    }
  }
}
```

### OpenClaw

```bash
openclaw mcp add thejaredchapman-evalforge-lite --command uvx --arg evalforge-lite
```

### Hermes

```yaml
mcp_servers:
  thejaredchapman-evalforge-lite:
    command: "uvx"
    args: ["evalforge-lite"]
```

### Netclaw

```json
{
  "McpServers": {
    "thejaredchapman-evalforge-lite": {
      "Transport": "stdio",
      "Command": "uvx",
      "Arguments": [
        "evalforge-lite"
      ]
    }
  }
}
```

### Vellum

```bash
assistant mcp add thejaredchapman-evalforge-lite -t stdio -c uvx -a evalforge-lite
```

### Other

```json
{
  "mcpServers": {
    "thejaredchapman-evalforge-lite": {
      "command": "uvx",
      "args": [
        "evalforge-lite"
      ]
    }
  }
}
```

## Changelog

Every change recorded for this component, newest first. Days that predate change tracking, or that we cannot explain, say so: "we were watching and nothing happened" and "we were not watching" are different claims.

### 2026-10-16 (score 75, +1)

No change was recorded against any check on this day. Stability & Change Management went from 47 to 50. That category is still filling its 30-day observation window: 14 days of observed history at the previous scan, 15 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-10-14 (score 74, +1)

No change was recorded against any check on this day. Stability & Change Management went from 40 to 43. That category is still filling its 30-day observation window: 12 days of observed history at the previous scan, 13 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-10-13 (score 73, +32)

- [security improvement] Malware scan: unverified → pass
- [security improvement] Known CVEs: unverified → pass
- [functional improvement] Stability: unverified → 0.40
- [functional improvement] Dependency health: unverified → 0.97

### 2026-10-12 (score 41, −27)

- [security regression] Malware scan: pass → unverified
- [security regression] Known CVEs: pass → unverified
- [functional regression] Dependency health: 0.97 → unverified

### 2026-10-02 (score 68, +15)

- [security improvement] Malware scan: unverified → pass

### 2026-10-01 (score 53)

First indexed and scored.

## MCP tools (9)

### `list_models` (~92 tokens)

List every provider and model in the catalog, plus each provider's frontier (flagship) model.

Each model carries a `reasoning` flag (supports extended reasoning) and a `tags` list of
curated need/industry tag ids; `tags` in the result gives their labels, one-line "why"
explanations and the date they were verified. Tags are a starting point, not benchmarks.

### `suggest_models` (~29 tokens)

Suggest sibling models from the same family as the given model id.

Input parameters:

- `model_id` (string, required)

### `list_availability` (~45 tokens)

Return the current model-availability snapshot: live OpenRouter listing status
(refreshed at most every 6 hours) plus curated Bedrock/Vertex/Foundry region coverage.

### `set_policy` (~31 tokens)

Set the company policy text used to gate prompts before any model is called.

Input parameters:

- `policy_text` (string, required)

### `evaluate_prompt` (~223 tokens)

Get pre-run feedback on a prompt's clarity/specificity before running a comparison.

An explicit, separately-triggered LLM call (uses your credentials) — not run
automatically as part of run_comparison. Rate-limited independently from
run_comparison's 3-per-8h budget. Pass `creds` as {"openrouter"?: str,
"bedrock"?: {...}, "vertex"?: {...}, "foundry"?: {...}} to use Amazon Bedrock,
Google Vertex AI, or Microsoft Foundry; a bare `api_key` is treated as an
OpenRouter key. `judge_backend` picks which backend runs the evaluation.
If the operator has set server-side keys for a backend (see https://github.com/thejaredchapman/evalforge-lite/blob/main/docs/hosting-and-server-keys.md), those are used for it automatically and creds for it are not needed.

Input parameters:

- `api_key` (string)
- `creds`
- `judge_backend` (string)
- `prompt` (string, required)

### `run_comparison` (~365 tokens)

Run a set of test-case prompts against a set of models, scoring each response.

Each test case may include an optional "rubric" (scored by an LLM judge) and/or
"checks" (rule-based checks). Returns per-model grades, cost/latency stats, and an
overall verdict. Rate-limited to 3 calls per 8 hours.
Models are "<catalog id>" (OpenRouter) or "<catalog id>@bedrock" / "<catalog id>@vertex" /
"<catalog id>@foundry"; pass matching creds ({"openrouter"?, "bedrock"?, "vertex"?, "foundry"?})
or a bare OpenRouter api_key.
judge_backend picks which backend runs the judge and policy gate.
At most 4 models. priority (balanced|quality|fastest|cheapest) ranks the results;
repeats (1-3) re-sends each prompt for timing accuracy. Returns suggestions (same
provider and backend only), advice, ranking, and best_for_priority, plus a per-run
\`cost` total and, per cell, an `evaluation` (answered/quality/instruction_following/
completeness/helpfulness/safety scores, strengths, weaknesses, reasoning, overall).
If the operator has set server-side keys for a backend (see https://github.com/thejaredchapman/evalforge-lite/blob/main/docs/hosting-and-server-keys.md), those are used for it automatically and creds for it are not needed.

Input parameters:

- `api_key` (string)
- `creds`
- `judge_backend` (string)
- `models` (array, required)
- `priority` (string)
- `repeats` (integer)
- `test_cases` (array, required)

### `list_runs` (~21 tokens)

List metadata for the 5 most recent runs, newest first.

### `get_report` (~91 tokens)

Get a PDF report (base64-encoded) for a run. Defaults to the most recent run.

priority (balanced|quality|fastest|cheapest), if given, overrides the priority the
run was made with for the report's priority/best-pick line; otherwise the run's own
priority (from run_comparison) is used.

Input parameters:

- `priority`
- `run_id`

### `get_report_csv` (~41 tokens)

Get a CSV export for a run, one row per (test case x model) cell. Defaults to the most recent run.

Input parameters:

- `run_id`

## Diagnostics

Captured diagnostic sections: Provenance, Install scripts, Dependencies. The full working is on the page: https://verifymcp.io/servers/thejaredchapman-evalforge-lite/evalforge-lite#diagnostics

## Score history

- 2026-10-17: 75
- 2026-10-16: 75
- 2026-10-15: 74
- 2026-10-14: 74
- 2026-10-13: 73
- 2026-10-12: 41
- 2026-10-04: 68
- 2026-10-03: 68
- 2026-10-02: 68
- 2026-10-01: 53

## Common questions

### What is the EvalForge Lite MCP server?

EvalForge Lite is an MCP server listed in the public MCP registry as io.github.thejaredchapman/evalforge-lite. Compare text LLMs across OpenRouter, Bedrock, Vertex AI and Foundry with automated grading. This page covers its PyPI package (evalforge-lite).

### Is the EvalForge Lite MCP server safe to use?

EvalForge Lite scores 75 out of 100 on VerifyMCP. We found no known CVEs affecting it as of 17 October 2026. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.

### What tools does the EvalForge Lite MCP server expose?

EvalForge Lite exposes 9 tools: list_models, suggest_models, list_availability, set_policy, evaluate_prompt, and 4 more. Their descriptions and schemas cost roughly 938 tokens of context every time the server is loaded.

### Is the EvalForge Lite MCP server still maintained?

EvalForge Lite is still listed as active in the MCP registry. We last reached this channel on 17 October 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.

### What licence is the EvalForge Lite MCP server under?

EvalForge Lite declares the MIT licence, which is OSI-approved. That covers the source only, and says nothing about the cost of any service it calls.

## Links

- PyPI project: https://pypi.org/project/evalforge-lite/
- Socket report: https://socket.dev/pypi/package/evalforge-lite
- Repository: https://github.com/thejaredchapman/evalforge-lite
- Website: https://github.com/thejaredchapman/evalforge-lite/blob/main/docs/mcp-server.md
- Changelog RSS feed: https://verifymcp.io/servers/thejaredchapman-evalforge-lite/evalforge-lite.xml
- Changelog JSON feed: https://verifymcp.io/servers/thejaredchapman-evalforge-lite/evalforge-lite.json
- HTML version of this page: https://verifymcp.io/servers/thejaredchapman-evalforge-lite/evalforge-lite