# AI Crawler Index (remote · www.pathwren.workers.dev)

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

- Trust score: 78/100 (medium)
- Change this week: +4
- Registry status: active
- Liveness: live
- Owner verified: no
- Last scored: 2026-09-21

## Components

- remote · `www.pathwren.workers.dev`: 78/100 (this document), [markdown](https://verifymcp.io/servers/dev-workers-pathwren-www-ai-crawler-index/c-mcp-registry-official-mcp.md), [page](https://verifymcp.io/servers/dev-workers-pathwren-www-ai-crawler-index/c-mcp-registry-official-mcp)

## Channel facts

- Endpoint: `https://www.pathwren.workers.dev/c/mcp-registry-official/mcp`
- Transports: `streamable-http`
- Auth: `none`
- Version: `1.6.0`

## Trust breakdown

How this component scores in each security and reliability category. Every signal is checked automatically against the live server, and we only credit what we can confirm. Scores are 0–100 per category. Scoring method: https://verifymcp.io/docs/scoring (what has changed: https://verifymcp.io/docs/scoring/changelog)

Scored 2026-09-21.

- **Endpoint Security**: 80/100
  - The endpoint's TLS certificate is valid, in date, and uses a strong key.
  - No authorisation is required to call this server. Every tool declares its destructiveHint and none is destructive, so open access doesn't expose one.
  - HTTPS is enforced; there's no plaintext access path.
  - The HSTS (Strict-Transport-Security) header is present.
  - DNSSEC check failed: this domain isn't protected by DNSSEC.
- **Transport & Reachability**: 100/100
  - Verified streamable-http transport via a live MCP handshake.
- **Schema Quality & AI Usability**: 55/100
  - 100% of prompts and resources have a non-trivial description (not blank, and not just the item's name).
  - AI-judged instruction clarity (poor).
  - Context-footprint check failed: tool/resource definitions use about 2923 tokens (~208/item across 14 items; 9 tools + 5 resources), over budget; trim descriptions and params.
  - Tools include usage examples.
- **Stability & Change Management**: 67/100
  - Stability observed for 20 of 30 days with no destabilising changes; credit accrues until the full window elapses.
- **Tool Coverage**: 100/100
  - 100% of tools have a non-trivial description (not blank, and not just the tool's name).
  - 100% of tool parameters carry a description.
  - Structured output schemas are declared (22% of tools); any adoption earns full credit.
- **Tool Safety**: 75/100
  - No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.
  - We read all 9 captured tool definition(s), and no name or description among them implies an irreversible operation.
  - Manipulation check failed: an AI judge found 10 of 11 captured unit(s) of tool text manipulative, the first being "server instructions".
- **Capabilities**: 100/100
  - Implements a current MCP spec version (2026-07-28).

## Install

### How do I install the AI Crawler Index MCP server?

AI Crawler Index is a hosted endpoint at https://www.pathwren.workers.dev/c/mcp-registry-official/mcp, so there is nothing to install locally. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.

### Claude

```bash
claude mcp add --transport http dev-workers-pathwren-www-ai-crawler-index 'https://www.pathwren.workers.dev/c/mcp-registry-official/mcp'
```

### Cursor

```json
{
  "mcpServers": {
    "dev-workers-pathwren-www-ai-crawler-index": {
      "url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp"
    }
  }
}
```

### VS Code

```json
{
  "servers": {
    "dev-workers-pathwren-www-ai-crawler-index": {
      "type": "http",
      "url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp"
    }
  }
}
```

### Codex

```toml
[mcp_servers.dev-workers-pathwren-www-ai-crawler-index]
url = "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp"
```

### opencode

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "dev-workers-pathwren-www-ai-crawler-index": {
      "type": "remote",
      "url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp",
      "enabled": true
    }
  }
}
```

### OpenClaw

```bash
openclaw mcp add dev-workers-pathwren-www-ai-crawler-index --url 'https://www.pathwren.workers.dev/c/mcp-registry-official/mcp' --transport streamable-http
```

### Hermes

```yaml
mcp_servers:
  dev-workers-pathwren-www-ai-crawler-index:
    url: "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp"
```

### Netclaw

```json
{
  "McpServers": {
    "dev-workers-pathwren-www-ai-crawler-index": {
      "Transport": "http",
      "Url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp"
    }
  }
}
```

### Vellum

```bash
assistant mcp add dev-workers-pathwren-www-ai-crawler-index -t streamable-http -u 'https://www.pathwren.workers.dev/c/mcp-registry-official/mcp'
```

### Other

```json
{
  "mcpServers": {
    "dev-workers-pathwren-www-ai-crawler-index": {
      "type": "http",
      "url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp"
    }
  }
}
```

The mcpServers block is a cross-client convention. Remote transports vary, so check your client's docs.

## Changelog

Every change recorded for this component, newest first. Days that predate change tracking, or that we cannot explain, say so: "we were watching and nothing happened" and "we were not watching" are different claims.

### 2026-09-21 (score 78, +1)

No change was recorded against any check on this day. Stability & Change Management went from 63 to 67. That category is still filling its 30-day observation window: 19 days of observed history at the previous scan, 20 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-09-19 (score 77, +1)

No change was recorded against any check on this day. Stability & Change Management went from 57 to 60. That category is still filling its 30-day observation window: 17 days of observed history at the previous scan, 18 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-09-17 (score 76, +1)

No change was recorded against any check on this day. Stability & Change Management went from 50 to 53. That category is still filling its 30-day observation window: 15 days of observed history at the previous scan, 16 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-09-15 (score 75, +1)

- [security] The server rewrote its instructions, which are the text every model session reads

### 2026-09-14 (score 74, 0)

- [security] The server rewrote its instructions, which are the text every model session reads

### 2026-09-13 (score 74, 0)

- [security] The server rewrote its instructions, which are the text every model session reads

### 2026-09-12 (score 74, +1)

- [security] The server rewrote its instructions, which are the text every model session reads

### 2026-09-11 (score 73, 0)

- [security] The server rewrote its instructions, which are the text every model session reads

## MCP tools (9)

### `whoami` (~296 tokens)

Who is calling? (no arguments)

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back describes YOU, the caller — the user-agent you sent, the address you came from, the class our instrument books you as, and whether this host has seen you before. Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and what the crawler index makes of the user-agent you sent, and whether the address you came from is inside a prefix its operator publishes — both run through this server's own classify_user_agent and is_verified_crawler_ip, on the same data a stranger can curl. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before.

Output parameters:

- `answered_by` (object): Which server answered, at which endpoint.
- `caveats` (array): What this answer does NOT establish — a user-agent is a claim.
- `independent` (boolean): This host is independent and unaffiliated.
- `license` (string)
- `this_call_touched` (object): Exactly which files were read to answer. No third party is contacted.
- `we_book_you_as` (object): The class this host's own instrument records for that user-agent.
- `we_have_seen_you` (object): Whether this user-agent appears in the published observation window.
- `you` (object): The user-agent you sent and the address you came from.

### `example` (~309 tokens)

Run this server's worked example (no arguments)

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back is this server's own worked example — classify_user_agent, run for real on a canned input from this host's own data — so you see the exact shape of an answer before you send anything of yours. Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — classify_user_agent on the exact user-agent string of the busiest client in this host's own published log (/data/observed-clients.json, 1164 requests in the window) — our data, and no third party is asked anything — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs classify_user_agent with {"user_agent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1. and returns its real answer.

Output parameters:

- `answer` (object): The real structuredContent of that call, not a mock.
- `answered_by` (object)
- `input_came_from` (string): Where the canned input came from — always this host's own data.
- `license` (string)
- `ran` (object): The tool name and the exact arguments that were run.
- `reproduce` (string): A command that reproduces this answer.
- `this_is_not_a_mock` (string)
- `what_it_shows` (string)

### `classify_user_agent` (~168 tokens)

Classify a user-agent string

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Identify the crawler behind a raw User-Agent header: operator, category, robots.txt stance, how to verify it, and what blocking it costs. A user-agent is a claim, not proof — confirm with is_verified_crawler_ip. Example: user_agent='GPTBot/1.2' returns GPTBot, OpenAI, ai-training. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/classify-ua?ua=GPTBot%2F1.2&s=client-dossiers

Input parameters:

- `user_agent` (string, required): Raw User-Agent header value.

### `lookup_crawler` (~116 tokens)

Look up one crawler

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The full index record for one crawler: operator, category, robots.txt token and stance, user-agent substring and example, verification method, mirrored IP-range endpoint, and what blocking it costs. An unknown slug returns near matches. Example: slug='claudebot'.

Input parameters:

- `slug` (string, required): Crawler slug, name or robots.txt token.

### `list_crawlers` (~194 tokens)

List and filter crawlers

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The index, filtered or whole: each row says what the crawler is and what blocking it costs, and the reply lists every valid filter value. Example: category='ai-training', limit=20.

Input parameters:

- `category` (string): ai-training, ai-search, user-fetch, dataset, search, seo, archive, preview or tool.
- `limit` (integer): Max rows, 1-200 (default 100).
- `operator` (string): Operator slug or name.
- `q` (string): Free text over slug, name and operator.
- `respects_robots_txt` (string): documented, disputed, by-design-no or n-a.
- `verification` (string): published-ranges, reverse-dns or none.

### `generate_robots_txt` (~131 tokens)

Generate a robots.txt for a stance

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — A ready-to-paste robots.txt block for one stance, with the reasoning and the crawlers it names. Omit the argument to get all eight stances described. Example: stance='block-ai-training'.

Input parameters:

- `stance` (string): One of allow-all, block-ai-training, block-all-ai, block-datasets, allow-ai-search-only, block-seo-tools, block-disputed, maximum-ai-visibility.

### `is_verified_crawler_ip` (~166 tokens)

Check an IP against operator-published crawler ranges

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Test an IPv4 or IPv6 address against every crawler prefix the operators themselves publish, remirrored every six hours. Returns the matching prefix, its source and the crawlers using it; a miss is reported as a miss, not as a forgery. Example: ip='20.171.206.5'. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verify-crawler?ip=20.171.206.5&s=client-dossiers

Input parameters:

- `ip` (string, required): IPv4 or IPv6 address to check.

### `whats_changed` (~121 tokens)

What changed in the index

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Freshness and change state: when the index was built, when each operator source was last fetched, which changed, which are failing, and the dated changelog. The cron check before regenerating a config from this data. Example: since='2026-08-01'.

Input parameters:

- `since` (string): ISO-8601 date or timestamp; omit for the current state.

### `changes_since` (~139 tokens)

What changed since your cursor

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Only what changed since your last call: prefixes added or removed, upstreams failed or recovered, records edited. Send back the `cursor` from the previous result; it advances only on a real change, and the data refreshes every 6h. Example: since='0'.

Input parameters:

- `limit` (integer): Events per page, default 100.
- `since` (string): Cursor from the last result, or an ISO date. Omit for all retained.

## Diagnostics

Captured diagnostic sections: TLS, DNSSEC, Authorisation, Transports. The full working is on the page: https://verifymcp.io/servers/dev-workers-pathwren-www-ai-crawler-index/c-mcp-registry-official-mcp#diagnostics

## Score history

- 2026-09-21: 78
- 2026-09-20: 77
- 2026-09-19: 77
- 2026-09-18: 76
- 2026-09-17: 76
- 2026-09-16: 75
- 2026-09-15: 75
- 2026-09-14: 74
- 2026-09-13: 74
- 2026-09-12: 74
- 2026-09-11: 73
- 2026-09-10: 73
- 2026-09-09: 72
- 2026-09-08: 72
- 2026-09-07: 71
- 2026-09-06: 71
- 2026-09-05: 70
- 2026-09-04: 70
- 2026-09-03: 78
- 2026-09-02: 77
- 2026-09-01: 77

## Common questions

### What is the AI Crawler Index MCP server?

AI Crawler Index is an MCP server listed in the public MCP registry as dev.workers.pathwren.www/ai-crawler-index. {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key. This page covers its hosted endpoint (https://www.pathwren.workers.dev/c/mcp-registry-official/mcp).

### Is the AI Crawler Index MCP server safe to use?

AI Crawler Index scores 78 out of 100 on VerifyMCP. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.

### What tools does the AI Crawler Index MCP server expose?

AI Crawler Index exposes 9 tools: whoami, example, classify_user_agent, lookup_crawler, list_crawlers, and 4 more. Their descriptions and schemas cost roughly 1,640 tokens of context every time the server is loaded.

### Does the AI Crawler Index MCP server require authentication?

No. We connected to AI Crawler Index without credentials and it answered, so anything it exposes is reachable by anyone who knows the address.

### Is the AI Crawler Index MCP server still maintained?

AI Crawler Index is still listed as active in the MCP registry. We last reached this channel on 21 September 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.

## Links

- Remote endpoint: https://www.pathwren.workers.dev/c/mcp-registry-official/mcp
- Website: https://www.pathwren.workers.dev/c/mcp-registry-official/mcp.html
- Changelog RSS feed: https://verifymcp.io/servers/dev-workers-pathwren-www-ai-crawler-index/c-mcp-registry-official-mcp.xml
- Changelog JSON feed: https://verifymcp.io/servers/dev-workers-pathwren-www-ai-crawler-index/c-mcp-registry-official-mcp.json
- HTML version of this page: https://verifymcp.io/servers/dev-workers-pathwren-www-ai-crawler-index/c-mcp-registry-official-mcp