# CompletionKit (remote · completionkit.com)

Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.

- Trust score: 73/100 (medium)
- Change this week: +12
- Registry status: active
- Liveness: live
- Owner verified: no
- Last scored: 2026-08-03

## Components

- remote · `completionkit.com`: 73/100 (this document), [markdown](https://verifymcp.io/servers/com-completionkit-evals/completionkit.md), [page](https://verifymcp.io/servers/com-completionkit-evals/completionkit)

## Channel facts

- Endpoint: `https://completionkit.com/mcp`
- Transports: `streamable-http`
- Auth: `required`
- Version: `1.0.0`

## Trust breakdown

How this component scores in each security and reliability category. Every signal is checked automatically against the live server, and we only credit what we can confirm. Scores are 0–100 per category. Scoring method: https://verifymcp.io/docs/scoring (what has changed: https://verifymcp.io/docs/scoring/changelog)

Scored 2026-08-03.

- **Endpoint Security**: 89/100
  - The endpoint's TLS certificate is valid, in date, and uses a strong key.
  - Authorisation is enforced on tool calls, but the challenge carries no valid RFC 9728 metadata, so a client cannot discover where to get a token.
  - HTTPS is enforced; there's no plaintext access path.
  - The HSTS (Strict-Transport-Security) header is present.
  - DNSSEC check failed: this domain isn't protected by DNSSEC.
- **Transport & Reachability**: 100/100
  - Verified streamable-http transport via a live MCP handshake.
- **Schema Quality & AI Usability**: 70/100
  - AI-judged instruction clarity (good).
  - Tool/resource definitions use about 2982 tokens (~55/item across 54 items; 54 tools + 0 resources), lean.
  - Usage-examples check failed: none of the tools include examples.
- **Stability & Change Management**: 27/100
  - Stability observed for 8 of 30 days with no destabilising changes; credit accrues until the full window elapses.
- **Tool Coverage**: 70/100
  - 100% of tools have a non-trivial description (not blank, and not just the tool's name).
  - 9% of tool parameters carry a description.
- **Capabilities**: 40/100
  - Spec-recency check failed: implements MCP spec 2025-03-26; the latest is 2026-07-28.

## Install

### Claude

```bash
claude mcp add --transport http com-completionkit-evals https://completionkit.com/mcp
```

### Codex

```toml
[mcp_servers.com-completionkit-evals]
url = "https://completionkit.com/mcp"
```

### opencode

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "com-completionkit-evals": {
      "type": "remote",
      "url": "https://completionkit.com/mcp",
      "enabled": true
    }
  }
}
```

### OpenClaw

```bash
openclaw mcp add com-completionkit-evals --url https://completionkit.com/mcp --transport streamable-http
```

### Hermes

```yaml
mcp_servers:
  com-completionkit-evals:
    url: "https://completionkit.com/mcp"
```

### Other

```json
{
  "mcpServers": {
    "com-completionkit-evals": {
      "type": "http",
      "url": "https://completionkit.com/mcp"
    }
  }
}
```

The mcpServers block is a cross-client convention. Remote transports vary, so check your client's docs.

## Changelog

Every change recorded for this component, newest first. Days that predate change tracking, or that we cannot explain, say so: "we were watching and nothing happened" and "we were not watching" are different claims.

### 2026-08-03 (score 73, +1)

No change was recorded against any check on this day. Stability & Change Management went from 23 to 27. That category is still filling its 30-day observation window: 7 days of observed history at the previous scan, 8 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-08-01 (score 72, +1)

No change was recorded against any check on this day. Stability & Change Management went from 17 to 20. That category is still filling its 30-day observation window: 5 days of observed history at the previous scan, 6 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-07-31 (score 71, +6)

- [functional] We updated how we score, so this day's move reflects our rubric, not a change to the server

### 2026-07-30 (score 65, 0)

- [functional] We updated how we score, so this day's move reflects our rubric, not a change to the server

### 2026-07-29 (score 65, +4)

- [functional regression] Schema quality: 2695 → 2982
- [functional] Schema quality: fair → good
- [functional] Server version: 0.28.13 → 0.28.23
- [functional] New tool “runs_regrade”
- [functional] New tool “runs_retry_failures”
- [functional] New tool “usage_get”
- [functional] New tool “runs_rerun”
- [cosmetic] “runs_update” added an optional parameter “metric_group_id”
- [cosmetic] “runs_create” added an optional parameter “metric_group_id”

### 2026-07-28 (score 61, 0)

- [functional] Server version: 0.28.11 → 0.28.13

### 2026-07-27 (score 61, +1)

- [functional] We updated how we score, so this day's move reflects our rubric, not a change to the server

### 2026-07-26 (score 60)

First indexed and scored.

## MCP tools (54)

### `prompts_list` (~12 tokens)

List all prompts

### `prompts_get` (~23 tokens)

Get a prompt by ID

Input parameters:

- `id` (integer, required): Prompt ID

### `prompts_create` (~50 tokens)

Create a prompt

Input parameters:

- `description` (string)
- `llm_model` (string, required)
- `name` (string, required)
- `tag_names` (array)
- `template` (string, required)

### `prompts_update` (~113 tokens)

Update a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing — promote it with prompts_publish — so an agent's edits don't go live without a gate. If it has no runs, it is updated in place.

Input parameters:

- `description` (string)
- `id` (integer, required)
- `llm_model` (string)
- `name` (string)
- `tag_names` (array)
- `template` (string)

### `prompts_delete` (~19 tokens)

Delete a prompt

Input parameters:

- `id` (integer, required)

### `prompts_publish` (~26 tokens)

Publish a prompt version, making it the current version

Input parameters:

- `id` (integer, required)

### `prompts_suggest_improvement` (~94 tokens)

Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it as a Suggestion. Requires a run that has a prompt (not a scoring-only run).

Input parameters:

- `run_id` (integer, required): The run whose results ground the improvement.

### `runs_list` (~12 tokens)

List all runs

### `runs_get` (~21 tokens)

Get a run by ID

Input parameters:

- `id` (integer, required)

### `runs_create` (~178 tokens)

Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.

Input parameters:

- `dataset_id` (integer)
- `expected_column` (string): Dataset column holding each row's answer key / ground truth, graded by checks with compare_to "expected" and passed to the judge; defaults to "expected_output".
- `judge_model` (string)
- `metric_group_id` (integer): Attach the metrics belonging to this metric group (its current metric_ids). Ignored when metric_ids is also given.
- `metric_ids` (array)
- `name` (string, required)
- `output_column` (string): Dataset column to grade when prompt_id is omitted; defaults to "actual_output".
- `prompt_id` (integer)
- `tag_names` (array)

### `runs_update` (~104 tokens)

Update a run

Input parameters:

- `dataset_id` (integer)
- `expected_column` (string)
- `id` (integer, required)
- `judge_model` (string)
- `metric_group_id` (integer): Replace the run's metrics with those belonging to this metric group. Ignored when metric_ids is also given.
- `metric_ids` (array)
- `name` (string)
- `output_column` (string)
- `tag_names` (array)

### `runs_delete` (~19 tokens)

Delete a run

Input parameters:

- `id` (integer, required)

### `runs_generate` (~52 tokens)

Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.

Input parameters:

- `id` (integer, required)

### `runs_regrade` (~45 tokens)

Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.

Input parameters:

- `id` (integer, required)

### `runs_rerun` (~54 tokens)

Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.

Input parameters:

- `id` (integer, required)

### `runs_retry_failures` (~44 tokens)

Re-run only the failed responses of a run, optionally limited to specific response ids via "only".

Input parameters:

- `id` (integer, required)
- `only` (array)

### `responses_list` (~22 tokens)

List responses for a run

Input parameters:

- `run_id` (integer, required)

### `responses_get` (~28 tokens)

Get a specific response

Input parameters:

- `id` (integer, required)
- `run_id` (integer, required)

### `datasets_list` (~12 tokens)

List all datasets

### `datasets_get` (~21 tokens)

Get a dataset by ID

Input parameters:

- `id` (integer, required)

### `datasets_create` (~124 tokens)

Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and "actual_output" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Every column is also available to the prompt as a variable.

Input parameters:

- `csv_data` (string, required)
- `name` (string, required)
- `tag_names` (array)

### `datasets_update` (~42 tokens)

Update a dataset

Input parameters:

- `csv_data` (string)
- `id` (integer, required)
- `name` (string)
- `tag_names` (array)

### `datasets_delete` (~19 tokens)

Delete a dataset

Input parameters:

- `id` (integer, required)

### `datasets_create_from_url` (~145 tokens)

Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked and the download is capped at 10MB. First row is the header; the "expected_output" (answer key) and "actual_output" (pre-made output) columns are recognized specially, overridable per run.

Input parameters:

- `name` (string, required)
- `tag_names` (array)
- `url` (string, required): Public http(s) URL of the CSV file to download.

### `metrics_list` (~12 tokens)

List all metrics

### `metrics_get` (~21 tokens)

Get a metric by ID

Input parameters:

- `id` (integer, required)

### `metrics_create` (~178 tokens)

Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.

Input parameters:

- `check_config` (object)
- `instruction` (string)
- `metric_type` (string)
- `name` (string, required)
- `rubric_bands` (array)
- `tag_names` (array)

### `metrics_update` (~182 tokens)

Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.

Input parameters:

- `check_config` (object)
- `id` (integer, required)
- `instruction` (string)
- `metric_type` (string)
- `name` (string)
- `rubric_bands` (array)
- `tag_names` (array)

### `metrics_delete` (~19 tokens)

Delete a metric

Input parameters:

- `id` (integer, required)

### `metrics_suggest_variants` (~134 tokens)

Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-metering hooks fire via ActiveSupport::Notifications under completion_kit.judge_suggestion.generated.

Input parameters:

- `count` (integer): How many variants to request (default 1, max 3). One focused rewrite beats five reworded copies.
- `metric_id` (integer, required)
- `model` (string): Override the model used to generate variants. Defaults to the configured judge model or an available judging model.

### `metric_groups_list` (~14 tokens)

List all metric groups

### `metric_groups_get` (~23 tokens)

Get a metric group by ID

Input parameters:

- `id` (integer, required)

### `metric_groups_create` (~44 tokens)

Create a metric group

Input parameters:

- `description` (string)
- `metric_ids` (array)
- `name` (string, required)
- `tag_names` (array)

### `metric_groups_update` (~51 tokens)

Update a metric group

Input parameters:

- `description` (string)
- `id` (integer, required)
- `metric_ids` (array)
- `name` (string)
- `tag_names` (array)

### `metric_groups_delete` (~21 tokens)

Delete a metric group

Input parameters:

- `id` (integer, required)

### `metric_versions_list` (~50 tokens)

List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.

Input parameters:

- `metric_id` (integer, required)

### `metric_versions_publish` (~80 tokens)

Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_bands back onto the metric so the judge grades against it.

Input parameters:

- `metric_version_id` (integer, required)

### `metric_versions_dismiss` (~60 tokens)

Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused — to demote a published version, publish a different one as current instead.

Input parameters:

- `metric_version_id` (integer, required)

### `provider_credentials_list` (~20 tokens)

List all provider credentials (API keys are not exposed)

### `provider_credentials_get` (~29 tokens)

Get a provider credential by ID (API key is not exposed)

Input parameters:

- `id` (integer, required)

### `provider_credentials_create` (~45 tokens)

Create a provider credential

Input parameters:

- `api_endpoint` (string)
- `api_key` (string, required)
- `api_version` (string)
- `provider` (string, required)

### `provider_credentials_update` (~52 tokens)

Update a provider credential

Input parameters:

- `api_endpoint` (string)
- `api_key` (string)
- `api_version` (string)
- `id` (integer, required)
- `provider` (string)

### `provider_credentials_delete` (~21 tokens)

Delete a provider credential

Input parameters:

- `id` (integer, required)

### `tags_list` (~12 tokens)

List all tags

### `tags_get` (~21 tokens)

Get a tag by ID

Input parameters:

- `id` (integer, required)

### `tags_create` (~25 tokens)

Create a tag. Color is auto-assigned.

Input parameters:

- `name` (string, required)

### `tags_update` (~26 tokens)

Rename a tag.

Input parameters:

- `id` (integer, required)
- `name` (string)

### `tags_delete` (~34 tokens)

Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset.

Input parameters:

- `id` (integer, required)

### `agreements_list` (~58 tokens)

List agreements. Filter by run_id, response_id, metric_id, or created_by.

Input parameters:

- `created_by` (string)
- `metric_id` (integer)
- `response_id` (integer)
- `run_id` (integer)

### `agreements_create` (~103 tokens)

Upsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'.

Input parameters:

- `corrected_score` (number)
- `created_by` (string)
- `metric_id` (integer, required)
- `note` (string)
- `response_id` (integer, required)
- `run_id` (integer, required)
- `verdict` (string, required)

### `judges_replay` (~115 tokens)

Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts.

Input parameters:

- `dataset_id` (integer, required)
- `judge_model` (string, required)
- `metric_id` (integer, required)
- `name` (string, required)
- `output_column` (string): Dataset column with the existing outputs to grade. Defaults to actual_output.

### `judges_compare` (~78 tokens)

Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.

Input parameters:

- `metric_id` (integer, required)
- `metric_version_a_id` (integer, required)
- `metric_version_b_id` (integer, required)

### `promptfoo_import` (~92 tokens)

Import a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become deterministic check metrics). Returns a summary of what mapped and what was skipped and why; nothing is dropped silently.

Input parameters:

- `config` (string, required): The full promptfooconfig.yaml contents.

### `usage_get` (~83 tokens)

Get this organization's plan usage and limits for the current billing period: runs and prompt fetches used, their limits, how many remain, and when the period resets. Call this to pre-check quota before starting runs. Runs are hard-blocked once the run limit is reached (with a small grace band), so a run over the limit will fail with run_limit_reached.

## Diagnostics

Captured diagnostic sections: TLS, DNSSEC, Authorisation, Transports. The full working is on the page: https://verifymcp.io/servers/com-completionkit-evals/completionkit#diagnostics

## Score history

- 2026-08-03: 73
- 2026-08-02: 72
- 2026-08-01: 72
- 2026-07-31: 71
- 2026-07-30: 65
- 2026-07-29: 65
- 2026-07-28: 61
- 2026-07-27: 61
- 2026-07-26: 60

## Links

- Remote endpoint: https://completionkit.com/mcp
- Authorisation metadata: https://completionkit.com/.well-known/oauth-protected-resource/mcp
- Repository: https://github.com/homemade-software-inc/completion-kit
- Website: https://completionkit.com/
- Changelog RSS feed: https://verifymcp.io/servers/com-completionkit-evals/completionkit/changelog.xml
- Changelog JSON feed: https://verifymcp.io/servers/com-completionkit-evals/completionkit/changelog.json
- HTML version of this page: https://verifymcp.io/servers/com-completionkit-evals/completionkit