EvalForge Lite
PYPI · EVALFORGE-LITE
Compare text LLMs across OpenRouter, Bedrock, Vertex AI and Foundry with automated grading.
Available components
Last scanned (11 hours ago)
Scores higher than 73% of PyPI servers
Category AI and Machine Learning
How this component scores in each security and reliability category. Every signal is checked automatically from public evidence about the published package, including repeated runs of it in an isolated sandbox, and we only credit what we can confirm. How we score → Why this is hard to score →
Supply Chain Security100
- No malware found by supply-chain analysis.Pass
- No known CVEs affecting this package version or its production dependencies.Pass
- Runs setuptools.build_meta at install time, a recognised build step with no custom scripting around it. View diagnostics → Pass
- 5 of 64 dependencies flagged as unhealthy. View diagnostics → Partial
Provenance & Transparency45
- Source repository is publicly reachable at the declared URL. View diagnostics → Pass
- Provenance check failed: no build-provenance attestation is published. See how to fix → View diagnostics → Fail
- Clear OSI-approved license (MIT).Pass
- Actively maintained (last published 2 days ago).Pass
- Disclosure check failed: no security disclosure policy was found in the source repository. See how to fix → Fail
Schema Quality & AI Usability73
- AI-judged instruction clarity (good).Pass
- Tool/resource definitions use about 938 tokens (~104/item across 9 items; 9 tools + 0 resources), lean.Pass
- Usage-examples check failed: none of the tools include examples. See how to fix → Fail
Stability & Change Management53
- Stability observed for 16 of 30 days with no destabilising changes; credit accrues until the full window elapses.Partial
Tool Coverage67
- 100% of tools have a non-trivial description (not blank, and not just the tool's name).Pass
- 0% of tool parameters carry a description.Fail
Tool Safety100
- No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.Pass
- We read all 9 captured tool definition(s), and no name or description among them implies an irreversible operation.Pass
- An AI judge read all 9 captured unit(s) of tool text and found none that tries to manipulate the model reading it.Pass
Capabilities100
- Implements a current MCP spec version (2026-07-28).Pass
Every change we have recorded for this component, newest first. Security-relevant changes are always shown. ▲ marks a change for the better, ▼ a change for the worse; unmarked changes are neutral.
- 16 Oct 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 47 to 50. That category is still filling its 30-day observation window: 14 days of observed history at the previous scan, 15 at this one. The score rises as the window fills, whether or not the server changes.
- 14 Oct 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 40 to 43. That category is still filling its 30-day observation window: 12 days of observed history at the previous scan, 13 at this one. The score rises as the window fills, whether or not the server changes.
- 13 Oct 26 +32
- Malware scan: unverified → pass ▲ security
- Known CVEs: unverified → pass ▲ security
- Stability: unverified → 0.40 ▲ functional
- Dependency health: unverified → 0.97 ▲ functional
- 12 Oct 26 −27
- Malware scan: pass → unverified ▼ security
- Known CVEs: pass → unverified ▼ security
- Dependency health: 0.97 → unverified ▼ functional
- 2 Oct 26 +15
- Malware scan: unverified → pass ▲ security
- 1 Oct 26 53
First indexed and scored.
Diagnostic detail from the automated scan of this channel: what the scanner observed at each step, so you can see exactly where a check passed or failed. It is informational only and never changes the trust score.
Captured 17 Oct 2026 · Analysed pypi/evalforge-lite@1.0.1
Provenance No attestation
The registry publishes no build provenance for this version, so there is nothing to verify.
| Result | No attestation |
|---|---|
| Ecosystem | pypi |
| Reason | No attestation published |
Background: How many MCP packages publish verified provenance →
Install scripts 1 script
| Hook | Tier | Command |
|---|---|---|
| build_backend | allowlisted | setuptools.build_meta |
Background: Why install scripts are a supply-chain risk →
Dependencies 64 packages
| Packages resolved | 64 |
|---|---|
| Stale | 4 |
| No linked repository | 1 |
| Tree resolution | Complete |
Background: SBOMs and build attestations, explained →
The tools this component advertises to a client, with an estimated token cost for each. Expand a tool to see its parameters and schema. The per-tool counts are indicative and are not scored directly; the schema's total context footprint is one signal in Schema Quality & AI Usability. A tool's description is untrusted text the model reads on every call, which is what makes this list a security surface and not just an inventory: how tool poisoning works →
evaluate_prompt ~223
Get pre-run feedback on a prompt's clarity/specificity before running a comparison. An explicit, separately-triggered LLM call (uses your credentials) — not run automatically as part of run_comparison. Rate-limited independently from run_comparison's 3-per-8h budget. Pass `creds` as {"openrouter"?: str, "bedrock"?: {...}, "vertex"?: {...}, "foundry"?: {...}} to use Amazon Bedrock, Google Vertex AI, or Microsoft Foundry; a bare `api_key` is treated as an OpenRouter key. `judge_backend` picks which backend runs the evaluation. If the operator has set server-side keys for a backend (see https://github.com/thejaredchapman/evalforge-lite/blob/main/docs/hosting-and-server-keys.md), those are used for it automatically and creds for it are not needed.
| Name | Type | Req | Description |
|---|---|---|---|
| api_key | string | – | – |
| creds | – | – | – |
| judge_backend | string | – | – |
| prompt | string | yes | – |
No output schema declared.
No examples provided.
get_report ~91
Get a PDF report (base64-encoded) for a run. Defaults to the most recent run. priority (balanced|quality|fastest|cheapest), if given, overrides the priority the run was made with for the report's priority/best-pick line; otherwise the run's own priority (from run_comparison) is used.
| Name | Type | Req | Description |
|---|---|---|---|
| priority | – | – | – |
| run_id | – | – | – |
No output schema declared.
No examples provided.
get_report_csv ~41
Get a CSV export for a run, one row per (test case x model) cell. Defaults to the most recent run.
| Name | Type | Req | Description |
|---|---|---|---|
| run_id | – | – | – |
No output schema declared.
No examples provided.
list_availability ~45
Return the current model-availability snapshot: live OpenRouter listing status (refreshed at most every 6 hours) plus curated Bedrock/Vertex/Foundry region coverage.
Input schema present but exposes no named parameters.
No output schema declared.
No examples provided.
list_models ~92
List every provider and model in the catalog, plus each provider's frontier (flagship) model. Each model carries a `reasoning` flag (supports extended reasoning) and a `tags` list of curated need/industry tag ids; `tags` in the result gives their labels, one-line "why" explanations and the date they were verified. Tags are a starting point, not benchmarks.
Input schema present but exposes no named parameters.
No output schema declared.
No examples provided.
list_runs ~21
List metadata for the 5 most recent runs, newest first.
Input schema present but exposes no named parameters.
No output schema declared.
No examples provided.
run_comparison ~365
Run a set of test-case prompts against a set of models, scoring each response. Each test case may include an optional "rubric" (scored by an LLM judge) and/or "checks" (rule-based checks). Returns per-model grades, cost/latency stats, and an overall verdict. Rate-limited to 3 calls per 8 hours. Models are "<catalog id>" (OpenRouter) or "<catalog id>@bedrock" / "<catalog id>@vertex" / "<catalog id>@foundry"; pass matching creds ({"openrouter"?, "bedrock"?, "vertex"?, "foundry"?}) or a bare OpenRouter api_key. judge_backend picks which backend runs the judge and policy gate. At most 4 models. priority (balanced|quality|fastest|cheapest) ranks the results; repeats (1-3) re-sends each prompt for timing accuracy. Returns suggestions (same provider and backend only), advice, ranking, and best_for_priority, plus a per-run `cost` total and, per cell, an `evaluation` (answered/quality/instruction_following/ completeness/helpfulness/safety scores, strengths, weaknesses, reasoning, overall). If the operator has set server-side keys for a backend (see https://github.com/thejaredchapman/evalforge-lite/blob/main/docs/hosting-and-server-keys.md), those are used for it automatically and creds for it are not needed.
| Name | Type | Req | Description |
|---|---|---|---|
| api_key | string | – | – |
| creds | – | – | – |
| judge_backend | string | – | – |
| models | array | yes | – |
| priority | string | – | – |
| repeats | integer | – | – |
| test_cases | array | yes | – |
No output schema declared.
No examples provided.
set_policy ~31
Set the company policy text used to gate prompts before any model is called.
| Name | Type | Req | Description |
|---|---|---|---|
| policy_text | string | yes | – |
No output schema declared.
No examples provided.
suggest_models ~29
Suggest sibling models from the same family as the given model id.
| Name | Type | Req | Description |
|---|---|---|---|
| model_id | string | yes | – |
No output schema declared.
No examples provided.
How do I install the EvalForge Lite MCP server?
EvalForge Lite runs locally as a PyPI package, launched with uvx evalforge-lite. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.
pypi · evalforge-lite
claude mcp add thejaredchapman-evalforge-lite -- uvx evalforge-lite
What is the EvalForge Lite MCP server?
EvalForge Lite is an MCP server listed in the public MCP registry as io.github.thejaredchapman/evalforge-lite. Compare text LLMs across OpenRouter, Bedrock, Vertex AI and Foundry with automated grading. This page covers its PyPI package (evalforge-lite).
Is the EvalForge Lite MCP server safe to use?
EvalForge Lite scores 75 out of 100 on VerifyMCP. We found no known CVEs affecting it as of 17 October 2026. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.
What tools does the EvalForge Lite MCP server expose?
EvalForge Lite exposes 9 tools: list_models, suggest_models, list_availability, set_policy, evaluate_prompt, and 4 more. Their descriptions and schemas cost roughly 938 tokens of context every time the server is loaded.
Is the EvalForge Lite MCP server still maintained?
EvalForge Lite is still listed as active in the MCP registry. We last reached this channel on 17 October 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.
What licence is the EvalForge Lite MCP server under?
EvalForge Lite declares the MIT licence, which is OSI-approved. That covers the source only, and says nothing about the cost of any service it calls.