# io.github.bch1212/modelwatch (npm · modelwatch-mcp)

Continuous behavioral drift monitoring for LLM apps — catches silent provider model updates.

- Trust score: 79/100 (medium)
- Change this week: +34
- Registry status: active
- Liveness: live
- Owner verified: no
- Last scored: 2026-08-03

## Components

- npm · `modelwatch-mcp`: 79/100 (this document), [markdown](https://verifymcp.io/servers/bch1212-modelwatch/modelwatch-mcp.md), [page](https://verifymcp.io/servers/bch1212-modelwatch/modelwatch-mcp)

## Channel facts

- Registry: `npm`
- Package: `modelwatch-mcp`
- Version: `0.1.1`
- Transport: `stdio`

## Trust breakdown

How this component scores in each security and reliability category. Every signal is checked automatically from public evidence about the published package, including repeated runs of it in an isolated sandbox, and we only credit what we can confirm. Scores are 0–100 per category. Scoring method: https://verifymcp.io/docs/scoring (what has changed: https://verifymcp.io/docs/scoring/changelog)

Scored 2026-08-03.

- **Supply Chain Security**: 86/100
  - No malware found by supply-chain analysis.
  - Only part of the dependency tree could be resolved (94 of 98), so this covers what we could see, not the whole tree.
  - No install/post-install scripts declared.
  - Only part of the dependency tree could be resolved (94 of 98), so this covers what we could see, not the whole tree.
- **Provenance & Transparency**: 97/100
  - Source repository is publicly reachable at the declared URL.
  - Cryptographically verified build provenance (signed, bound to bch1212/modelwatch).
  - Clear OSI-approved license (MIT).
  - Actively maintained (last published 85 days ago).
  - Disclosure check failed: no security disclosure policy was found in the source repository.
- **Schema Quality & AI Usability**: 77/100
  - AI-judged instruction clarity (excellent).
  - Tool/resource definitions use about 745 tokens (~82/item across 9 items; 9 tools + 0 resources), lean.
  - Usage-examples check failed: none of the tools include examples.
- **Stability & Change Management**: 27/100
  - Stability observed for 8 of 30 days with no destabilising changes; credit accrues until the full window elapses.
- **Tool Coverage**: 92/100
  - 100% of tools have a non-trivial description (not blank, and not just the tool's name).
  - 75% of tool parameters carry a description.
- **Capabilities**: 100/100
  - Implements a supported MCP spec version (2025-11-25); the latest is 2026-07-28.

## Install

### Claude

```bash
claude mcp add bch1212-modelwatch -- npx -y modelwatch-mcp
```

### Codex

```bash
codex mcp add bch1212-modelwatch -- npx -y modelwatch-mcp
```

### opencode

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "bch1212-modelwatch": {
      "type": "local",
      "command": [
        "npx",
        "-y",
        "modelwatch-mcp"
      ],
      "enabled": true
    }
  }
}
```

### OpenClaw

```bash
openclaw mcp add bch1212-modelwatch --command npx --arg -y --arg modelwatch-mcp
```

### Hermes

```yaml
mcp_servers:
  bch1212-modelwatch:
    command: "npx"
    args: ["-y", "modelwatch-mcp"]
```

### Other

```json
{
  "mcpServers": {
    "bch1212-modelwatch": {
      "command": "npx",
      "args": [
        "-y",
        "modelwatch-mcp"
      ]
    }
  }
}
```

## Changelog

Every change recorded for this component, newest first. Days that predate change tracking, or that we cannot explain, say so: "we were watching and nothing happened" and "we were not watching" are different claims.

### 2026-08-03 (score 79, +1)

No change was recorded against any check on this day. Stability & Change Management went from 23 to 27. That category is still filling its 30-day observation window: 7 days of observed history at the previous scan, 8 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-08-02 (score 78, +43)

- [security improvement] Known CVEs: unverified → partial
- [security improvement] Install scripts: unverified → pass
- [security improvement] Provenance: unverified → pass
- [security improvement] Malware scan: unverified → pass
- [security] The attested source repository moved: bch1212/modelwatch
- [functional improvement] Maintenance: unverified → pass
- [functional improvement] Dependency health: unverified → partial
- [functional improvement] License: unverified → pass
- [functional improvement] Stability: unverified → 0.23
- [functional] Licence: MIT

### 2026-08-01 (score 35, +30)

- [security] Stability: Stability not yet verified: not enough scan history yet (needs a 30-day window).
- [functional regression] Security disclosure: unverified → fail
- [functional improvement] MCP protocol: unverified → pass
- [functional improvement] Tool coverage: unverified → 100

### 2026-07-31 (score 5, −56)

- [functional] We updated how we score, so this day's move reflects our rubric, not a change to the server

### 2026-07-30 (score 61, −30)

- [security regression] Malware scan: pass → unverified
- [security regression] Known CVEs: partial → unverified
- [functional regression] Dependency health: partial → unverified

### 2026-07-29 (score 91, +65)

- [security improvement] Known CVEs: unverified → partial
- [security improvement] Install scripts: unverified → pass
- [security improvement] Provenance: unverified → pass
- [security] The attested source repository moved: bch1212/modelwatch
- [functional improvement] License: unverified → pass
- [functional improvement] Tool coverage: unverified → 100
- [functional improvement] Schema quality: unverified → excellent
- [functional improvement] Maintenance: unverified → pass
- [functional] Licence: MIT

### 2026-07-28 (score 26, −19)

- [functional regression] Tool coverage: 100 → unverified
- [functional improvement] Dependency health: unverified → partial
- [functional] First check of Schema quality: unverified

### 2026-07-27 (score 45)

First indexed and scored.

## MCP tools (9)

### `list_endpoints` (~37 tokens)

List the LLM endpoints currently monitored in this workspace. Returns id, name, provider, model, base_url, created_at for each.

### `create_endpoint` (~130 tokens)

Register an LLM endpoint to monitor. The workspace must already have a stored API key for the provider (use the dashboard to add one). Returns the new endpoint's id.

Input parameters:

- `base_url` (string): Optional. Override base URL for OpenAI-compatible endpoints (vLLM, LiteLLM, Together).
- `model` (string, required): Model identifier, e.g. 'gpt-4o-mini' or 'claude-sonnet-4-6'.
- `name` (string, required): Human-readable label, e.g. 'GPT-4o mini prod'.
- `provider` (string, required)

### `list_specs` (~72 tokens)

List behavioral specs in the workspace. A spec is a stored prompt + expectation that ModelWatch replays on a schedule and diffs against a baseline. Returns id, name, prompt, frequency, threshold, last_severity, and the parent endpoint_id.

Input parameters:

- `endpoint_id` (string): Optional. Filter by endpoint.

### `create_spec` (~167 tokens)

Create a behavioral spec. The first run after creation sets the baseline output; subsequent scheduled runs are scored against that baseline across 5 axes (semantic, format, refusal, length, contains). An alert is sent when the drift score crosses the threshold.

Input parameters:

- `endpoint_id` (string, required): Endpoint to monitor (from list_endpoints).
- `frequency` (string): How often to run the spec.
- `name` (string, required): Spec label, e.g. 'Refusal canary' or 'JSON schema check'.
- `prompt` (string, required): The exact prompt to send to the model.
- `threshold` (string): Severity at which to fire an alert. Buckets: low ≥0.05, medium ≥0.15, high ≥0.35, critical ≥0.6.

### `run_spec` (~84 tokens)

Run a spec on demand and return the drift score immediately. Use this to (1) set the baseline manually right after create_spec, or (2) sanity-check a spec without waiting for the next scheduled run. Returns the drift score, severity bucket, per-axis scores, and the drift_event_id if one was created.

Input parameters:

- `spec_id` (string, required): From list_specs.

### `reset_baseline` (~69 tokens)

Clear a spec's baseline. The next run will record a new baseline instead of being diffed against the old one. Use this after you've intentionally changed your prompt template, model version, or the behavior you expect — otherwise every future run will look like drift.

Input parameters:

- `spec_id` (string, required)

### `get_drift_events` (~92 tokens)

Fetch recent drift events across the workspace, newest first. Each event has spec_id, spec_name, severity, drift_score, axes breakdown, baseline_output, current_output, and detected_at. Use this for weekly review or to drive an automation.

Input parameters:

- `limit` (integer): Max events to return (default 20, max 100).
- `spec_id` (string): Optional. Filter to one spec.

### `get_spec_history` (~59 tokens)

Get the run history for a single spec. Returns each run's drift score, severity, axes breakdown, and timestamp — useful for trending charts and reasoning about when behavior shifted.

Input parameters:

- `limit` (integer)
- `spec_id` (string, required)

### `get_health` (~35 tokens)

Workspace KPIs: plan, spec count, runs this month, plan limits, active drift events. Useful as a daily status check.

## Diagnostics

Captured diagnostic sections: Provenance, Dependencies. The full working is on the page: https://verifymcp.io/servers/bch1212-modelwatch/modelwatch-mcp#diagnostics

## Score history

- 2026-08-03: 79
- 2026-08-02: 78
- 2026-08-01: 35
- 2026-07-31: 5
- 2026-07-30: 61
- 2026-07-29: 91
- 2026-07-28: 26
- 2026-07-27: 45

## Links

- npm package: https://www.npmjs.com/package/modelwatch-mcp
- Socket report: https://socket.dev/npm/package/modelwatch-mcp
- Repository: https://github.com/bch1212/modelwatch
- Website: https://modelwatch.app/
- Changelog RSS feed: https://verifymcp.io/servers/bch1212-modelwatch/modelwatch-mcp/changelog.xml
- Changelog JSON feed: https://verifymcp.io/servers/bch1212-modelwatch/modelwatch-mcp/changelog.json
- HTML version of this page: https://verifymcp.io/servers/bch1212-modelwatch/modelwatch-mcp