Benchwarmers
PYPI · BENCHWARMERS · 2 COMPONENTS
Ask a collection of LLMs at once through your Benchwarmers server; verdicts pick the next panel.
Available components
Last scanned (19 hours ago)
Scores higher than 65% of PyPI servers
Category AI and Machine Learning
How this component scores in each security and reliability category. Every signal is checked automatically from public evidence about the published package, including repeated runs of it in an isolated sandbox, and we only credit what we can confirm. How we score → Why this is hard to score →
Supply Chain Security50
- Malware scan not yet available for this package.Unverified
- No known CVEs affecting this package version or its production dependencies.Pass
- Runs hatchling.build at install time, a recognised build step with no custom scripting around it. View diagnostics → Pass
- No production dependencies, so there is no dependency health to assess. View diagnostics → Pass
Provenance & Transparency100
- Source repository is publicly reachable at the declared URL. View diagnostics → Pass
- Cryptographically verified build provenance (signed, bound to WendellResearch-AI/benchwarmers). View diagnostics → Pass
- Clear OSI-approved license (MIT).Pass
- Actively maintained (last published 1 days ago).Pass
- Publishes a security disclosure policy (SECURITY.md).Pass
Schema Quality & AI Usability60
- AI-judged instruction clarity (good).Pass
- Context-footprint check failed: tool/resource definitions use about 702 tokens (~175/item across 4 items; 4 tools + 0 resources), over budget; trim descriptions and params. See how to fix → Fail
- Usage-examples check failed: none of the tools include examples. See how to fix → Fail
Stability & Change Management40
- Stability observed for 12 of 30 days with no destabilising changes; credit accrues until the full window elapses.Partial
Tool Coverage97
- 100% of tools have a non-trivial description (not blank, and not just the tool's name).Pass
- 92% of tool parameters carry a description.Partial
Tool Safety100
- No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.Pass
- We read all 4 captured tool definition(s), and no name or description among them implies an irreversible operation.Pass
- An AI judge read all 5 captured unit(s) of tool text and found none that tries to manipulate the model reading it.Pass
Capabilities100
- Implements a supported MCP spec version (2025-11-25); the latest is 2026-07-28.Pass
Every change we have recorded for this component, newest first. Security-relevant changes are always shown. ▲ marks a change for the better, ▼ a change for the worse; unmarked changes are neutral.
- 4 Nov 26 −15
- Malware scan: pass → unverified ▼ security
- 3 Nov 26 +16
- Malware scan: unverified → pass ▲ security
- 2 Nov 26 −15
- Malware scan: pass → unverified ▼ security
- Package version: 0.5.0 → 0.5.5 functional
- Package version: 0.5.0 → 0.5.4 functional
- Package version: 0.5.0 → 0.5.2 functional
- Package version: 0.5.0 → 0.5.1 functional
- 1 Nov 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 27 to 30. That category is still filling its 30-day observation window: 8 days of observed history at the previous scan, 9 at this one. The score rises as the window fills, whether or not the server changes.
- 31 Oct 26 +15
- Malware scan: unverified → pass ▲ security
- 30 Oct 26 −14
- Malware scan: pass → unverified ▼ security
- Package version: 0.4.0 → 0.5.0 functional
- 29 Oct 26 +15
- Malware scan: unverified → pass ▲ security
- 28 Oct 26 −14
- Malware scan: pass → unverified ▼ security
- Package version: 0.3.3 → 0.4.0 functional
Diagnostic detail from the automated scan of this channel: what the scanner observed at each step, so you can see exactly where a check passed or failed. It is informational only and never changes the trust score.
Captured 4 Nov 2026 · Analysed pypi/benchwarmers@0.5.5
Provenance Verified
A signed build attestation was found and verified, binding this exact artifact to the source repository it claims to come from.
| Result | Verified |
|---|---|
| Ecosystem | pypi |
| Reason | Verified |
| Discovered via | Registry attestation endpoint |
| Source repo | WendellResearch-AI/benchwarmers |
| Certificate issuer | https://token.actions.githubusercontent.com |
| Certificate SAN | https://github.com/WendellResearch-AI/benchwarmers/.github/workflows/release.yml@refs/tags/v0.5.5 |
| Rekor log index | 3648212493 |
| Predicate type | PyPI publish attestation https://docs.pypi.org/attestations/publish/v1 |
| Subject digest | sha256:0b27b4c96382c550dfcac00489e6fd2e2e3bafcb79f5bd3265701ec353253c2e |
Background: How many MCP packages publish verified provenance →
Install scripts 1 script
| Hook | Tier | Command |
|---|---|---|
| build_backend | allowlisted | hatchling.build |
Background: Why install scripts are a supply-chain risk →
Dependencies 0 packages
| Packages resolved | 0 |
|---|---|
| Tree resolution | Complete |
Background: SBOMs and build attestations, explained →
The tools this component advertises to a client, with an estimated token cost for each. Expand a tool to see its parameters and schema. The per-tool counts are indicative and are not scored directly; the schema's total context footprint is one signal in Schema Quality & AI Usability. A tool's description is untrusted text the model reads on every call, which is what makes this list a security surface and not just an inventory: how tool poisoning works →
ask Ask a collection ~216
Put a prompt to every seat of one collection at once and get every answer back, each under its seat and model, plus the seats as data (alias, seat, status, call id). Can take minutes. Ask a collection, not a favourite model. Give the seats the real text (code, diff, file contents) in the prompt: they cannot read your files. Verify every claim against the source before acting on it, name every seat that answered in what you tell the user, then send a verdict per seat you judged.
| Name | Type | Req | Description |
|---|---|---|---|
| collection | string | yes | A collection name, `core` or `bench/core` (list_collections). |
| merged | boolean | – | true: one reconciled answer, every seat still called and listed; false: every answer. Omit for the collection's own. |
| prompt | string | yes | The question, with everything it needs. |
| system | string | – | A system message for every seat. |
| with | array | – | Model aliases to seat in explorer chairs, this request only. |
No output schema declared.
No examples provided.
list_collections List collections ~63
The collections this Benchwarmers server offers: name, what each is for, tags, members, explorer seats, reply mode, and what the members handle together (smallest context window, shared capabilities, tightest rate limits). Pick by task before calling ask.
Input schema present but exposes no named parameters.
No output schema declared.
No examples provided.
pending_verdicts Seats still to judge ~101
For the Benchwarmers Claude Code plugin's Stop hook, which calls it at the end of a turn: the seats of this session's panels that have no verdict yet. You do not need to call it; send verdicts with `verdict`.
| Name | Type | Req | Description |
|---|---|---|---|
| active | – | – | The hook's stop_hook_active: true says nothing. |
| session | string | – | The Claude Code session id. |
| transcript | string | – | The session's transcript_path. |
No output schema declared.
No examples provided.
verdict Record a verdict ~160
Judge one seat's answer by its call id, once you have checked it against the source. One per seat you judged. List the claims it got right in `real` and the claims that are untrue in `false`, one sentence each: they feed routing and the 'confidently wrong' record, so a note alone is not enough.
| Name | Type | Req | Description |
|---|---|---|---|
| call | string | yes | The seat's call id, e.g. p20261022-101500-ab12.member1. |
| false | array | – | Claims it made that are untrue, one sentence each. |
| note | string | – | One or two sentences on why. |
| real | array | – | Claims it made that are true, one sentence each. |
| useful | boolean | yes | – |
No output schema declared.
No examples provided.
How do I install the Benchwarmers MCP server?
Benchwarmers runs locally as a PyPI package, launched with uvx benchwarmers. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.
pypi · benchwarmers
claude mcp add wendellresearch-ai-benchwarmers -- uvx benchwarmers
What is the Benchwarmers MCP server?
Benchwarmers is an MCP server listed in the public MCP registry as io.github.WendellResearch-AI/benchwarmers. Ask a collection of LLMs at once through your Benchwarmers server; verdicts pick the next panel. This page covers its PyPI package (benchwarmers).
Is the Benchwarmers MCP server safe to use?
Benchwarmers scores 69 out of 100 on VerifyMCP. We found no known CVEs affecting it as of 4 November 2026. Its build provenance is signed and verified. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.
What tools does the Benchwarmers MCP server expose?
Benchwarmers exposes 4 tools: list_collections, ask, verdict, pending_verdicts. Their descriptions and schemas cost roughly 540 tokens of context every time the server is loaded.
Is the Benchwarmers MCP server still maintained?
Benchwarmers is still listed as active in the MCP registry. We last reached this channel on 4 November 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.
What licence is the Benchwarmers MCP server under?
Benchwarmers declares the MIT licence, which is OSI-approved. That covers the source only, and says nothing about the cost of any service it calls.