# Benchwarmers (pypi · benchwarmers)

Ask a collection of LLMs at once through your Benchwarmers server; verdicts pick the next panel.

- Trust score: 85/100 (high trust)
- Change this week: +4
- Registry status: active
- Liveness: live
- Owner verified: no
- Last scored: 2026-11-05

## Components

- npm · `@wendellresearch-ai/benchwarmers`: 50/100, [markdown](https://verifymcp.io/servers/wendellresearch-ai-benchwarmers/wendellresearch-ai-benchwarmers.md), [page](https://verifymcp.io/servers/wendellresearch-ai-benchwarmers/wendellresearch-ai-benchwarmers)
- pypi · `benchwarmers`: 85/100 (this document), [markdown](https://verifymcp.io/servers/wendellresearch-ai-benchwarmers/benchwarmers.md), [page](https://verifymcp.io/servers/wendellresearch-ai-benchwarmers/benchwarmers)

## Channel facts

- Registry: `pypi`
- Package: `benchwarmers`
- Version: `0.5.5`
- Transport: `stdio`

## Trust breakdown

How this component scores in each security and reliability category. Every signal is checked automatically from public evidence about the published package, including repeated runs of it in an isolated sandbox, and we only credit what we can confirm. Scores are 0–100 per category. Scoring method: https://verifymcp.io/docs/scoring (what has changed: https://verifymcp.io/docs/scoring/changelog)

Scored 2026-11-05.

- **Supply Chain Security**: 100/100
  - No malware found by supply-chain analysis.
  - No known CVEs affecting this package version or its production dependencies.
  - Runs hatchling.build at install time, a recognised build step with no custom scripting around it.
  - No production dependencies, so there is no dependency health to assess.
- **Provenance & Transparency**: 100/100
  - Source repository is publicly reachable at the declared URL.
  - Cryptographically verified build provenance (signed, bound to WendellResearch-AI/benchwarmers).
  - Clear OSI-approved license (MIT).
  - Actively maintained (last published 2 days ago).
  - Publishes a security disclosure policy (SECURITY.md).
- **Schema Quality & AI Usability**: 60/100
  - AI-judged instruction clarity (good).
  - Context-footprint check failed: tool/resource definitions use about 702 tokens (~175/item across 4 items; 4 tools + 0 resources), over budget; trim descriptions and params.
  - Usage-examples check failed: none of the tools include examples.
- **Stability & Change Management**: 43/100
  - Stability observed for 13 of 30 days with no destabilising changes; credit accrues until the full window elapses.
- **Tool Coverage**: 97/100
  - 100% of tools have a non-trivial description (not blank, and not just the tool's name).
  - 92% of tool parameters carry a description.
- **Tool Safety**: 100/100
  - No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.
  - We read all 4 captured tool definition(s), and no name or description among them implies an irreversible operation.
  - An AI judge read all 5 captured unit(s) of tool text and found none that tries to manipulate the model reading it.
- **Capabilities**: 100/100
  - Implements a supported MCP spec version (2025-11-25); the latest is 2026-07-28.

## Install

### How do I install the Benchwarmers MCP server?

Benchwarmers runs locally as a PyPI package, launched with uvx benchwarmers. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.

### Claude

```bash
claude mcp add wendellresearch-ai-benchwarmers -- uvx benchwarmers
```

### Cursor

```json
{
  "mcpServers": {
    "wendellresearch-ai-benchwarmers": {
      "command": "uvx",
      "args": [
        "benchwarmers"
      ]
    }
  }
}
```

### VS Code

```json
{
  "servers": {
    "wendellresearch-ai-benchwarmers": {
      "command": "uvx",
      "args": [
        "benchwarmers"
      ]
    }
  }
}
```

### Codex

```bash
codex mcp add wendellresearch-ai-benchwarmers -- uvx benchwarmers
```

### opencode

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "wendellresearch-ai-benchwarmers": {
      "type": "local",
      "command": [
        "uvx",
        "benchwarmers"
      ],
      "enabled": true
    }
  }
}
```

### OpenClaw

```bash
openclaw mcp add wendellresearch-ai-benchwarmers --command uvx --arg benchwarmers
```

### Hermes

```yaml
mcp_servers:
  wendellresearch-ai-benchwarmers:
    command: "uvx"
    args: ["benchwarmers"]
```

### Netclaw

```json
{
  "McpServers": {
    "wendellresearch-ai-benchwarmers": {
      "Transport": "stdio",
      "Command": "uvx",
      "Arguments": [
        "benchwarmers"
      ]
    }
  }
}
```

### Vellum

```bash
assistant mcp add wendellresearch-ai-benchwarmers -t stdio -c uvx -a benchwarmers
```

### Other

```json
{
  "mcpServers": {
    "wendellresearch-ai-benchwarmers": {
      "command": "uvx",
      "args": [
        "benchwarmers"
      ]
    }
  }
}
```

## Changelog

Every change recorded for this component, newest first. Days that predate change tracking, or that we cannot explain, say so: "we were watching and nothing happened" and "we were not watching" are different claims.

### 2026-11-05 (score 85, +16)

- [security improvement] Malware scan: unverified → pass

### 2026-11-04 (score 69, −15)

- [security regression] Malware scan: pass → unverified

### 2026-11-03 (score 84, +16)

- [security improvement] Malware scan: unverified → pass

### 2026-11-02 (score 68, −15)

- [security regression] Malware scan: pass → unverified
- [functional] Package version: 0.5.0 → 0.5.5
- [functional] Package version: 0.5.0 → 0.5.4
- [functional] Package version: 0.5.0 → 0.5.2
- [functional] Package version: 0.5.0 → 0.5.1

### 2026-11-01 (score 83, +1)

No change was recorded against any check on this day. Stability & Change Management went from 27 to 30. That category is still filling its 30-day observation window: 8 days of observed history at the previous scan, 9 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-10-31 (score 82, +15)

- [security improvement] Malware scan: unverified → pass

### 2026-10-30 (score 67, −14)

- [security regression] Malware scan: pass → unverified
- [functional] Package version: 0.4.0 → 0.5.0

### 2026-10-29 (score 81, +15)

- [security improvement] Malware scan: unverified → pass

## MCP tools (4)

### `list_collections` (~63 tokens)

List collections

The collections this Benchwarmers server offers: name, what each is for, tags, members, explorer seats, reply mode, and what the members handle together (smallest context window, shared capabilities, tightest rate limits). Pick by task before calling ask.

### `ask` (~216 tokens)

Ask a collection

Put a prompt to every seat of one collection at once and get every answer back, each under its seat and model, plus the seats as data (alias, seat, status, call id). Can take minutes. Ask a collection, not a favourite model. Give the seats the real text (code, diff, file contents) in the prompt: they cannot read your files. Verify every claim against the source before acting on it, name every seat that answered in what you tell the user, then send a verdict per seat you judged.

Input parameters:

- `collection` (string, required): A collection name, `core` or `bench/core` (list_collections).
- `merged` (boolean): true: one reconciled answer, every seat still called and listed; false: every answer. Omit for the collection's own.
- `prompt` (string, required): The question, with everything it needs.
- `system` (string): A system message for every seat.
- `with` (array): Model aliases to seat in explorer chairs, this request only.

### `verdict` (~160 tokens)

Record a verdict

Judge one seat's answer by its call id, once you have checked it against the source. One per seat you judged. List the claims it got right in `real` and the claims that are untrue in `false`, one sentence each: they feed routing and the 'confidently wrong' record, so a note alone is not enough.

Input parameters:

- `call` (string, required): The seat's call id, e.g. p20261022-101500-ab12.member1.
- `false` (array): Claims it made that are untrue, one sentence each.
- `note` (string): One or two sentences on why.
- `real` (array): Claims it made that are true, one sentence each.
- `useful` (boolean, required)

### `pending_verdicts` (~101 tokens)

Seats still to judge

For the Benchwarmers Claude Code plugin's Stop hook, which calls it at the end of a turn: the seats of this session's panels that have no verdict yet. You do not need to call it; send verdicts with `verdict`.

Input parameters:

- `active`: The hook's stop_hook_active: true says nothing.
- `session` (string): The Claude Code session id.
- `transcript` (string): The session's transcript_path.

## Diagnostics

Captured diagnostic sections: Provenance, Install scripts, Dependencies. The full working is on the page: https://verifymcp.io/servers/wendellresearch-ai-benchwarmers/benchwarmers#diagnostics

## Score history

- 2026-11-05: 85
- 2026-11-04: 69
- 2026-11-03: 84
- 2026-11-02: 68
- 2026-11-01: 83
- 2026-10-31: 82
- 2026-10-30: 67
- 2026-10-29: 81
- 2026-10-28: 66
- 2026-10-27: 80
- 2026-10-26: 80
- 2026-10-25: 79
- 2026-10-24: 64
- 2026-10-23: 63

## Common questions

### What is the Benchwarmers MCP server?

Benchwarmers is an MCP server listed in the public MCP registry as io.github.WendellResearch-AI/benchwarmers. Ask a collection of LLMs at once through your Benchwarmers server; verdicts pick the next panel. This page covers its PyPI package (benchwarmers).

### Is the Benchwarmers MCP server safe to use?

Benchwarmers scores 85 out of 100 on VerifyMCP. We found no known CVEs affecting it as of 5 November 2026. Its build provenance is signed and verified. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.

### What tools does the Benchwarmers MCP server expose?

Benchwarmers exposes 4 tools: list_collections, ask, verdict, pending_verdicts. Their descriptions and schemas cost roughly 540 tokens of context every time the server is loaded.

### Is the Benchwarmers MCP server still maintained?

Benchwarmers is still listed as active in the MCP registry. We last reached this channel on 5 November 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.

### What licence is the Benchwarmers MCP server under?

Benchwarmers declares the MIT licence, which is OSI-approved. That covers the source only, and says nothing about the cost of any service it calls.

## Links

- PyPI project: https://pypi.org/project/benchwarmers/
- Socket report: https://socket.dev/pypi/package/benchwarmers
- Repository: https://github.com/WendellResearch-AI/benchwarmers
- Changelog RSS feed: https://verifymcp.io/servers/wendellresearch-ai-benchwarmers/benchwarmers.xml
- Changelog JSON feed: https://verifymcp.io/servers/wendellresearch-ai-benchwarmers/benchwarmers.json
- HTML version of this page: https://verifymcp.io/servers/wendellresearch-ai-benchwarmers/benchwarmers