# Crawlerpeek (npm · crawlerpeek)

Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only.

- Trust score: 75/100 (medium)
- Change this week: +18
- Registry status: active
- Liveness: live
- Owner verified: no
- Last scored: 2026-11-05

## Components

- npm · `crawlerpeek`: 75/100 (this document), [markdown](https://verifymcp.io/servers/crawlerpeek-crawlerpeek/crawlerpeek.md), [page](https://verifymcp.io/servers/crawlerpeek-crawlerpeek/crawlerpeek)

## Channel facts

- Registry: `npm`
- Package: `crawlerpeek`
- Version: `0.3.1`
- Transport: `stdio`

## Trust breakdown

How this component scores in each security and reliability category. Every signal is checked automatically from public evidence about the published package, including repeated runs of it in an isolated sandbox, and we only credit what we can confirm. Scores are 0–100 per category. Scoring method: https://verifymcp.io/docs/scoring (what has changed: https://verifymcp.io/docs/scoring/changelog)

Scored 2026-11-05.

- **Supply Chain Security**: 100/100
  - No malware found by supply-chain analysis.
  - No known CVEs affecting this package version or its production dependencies.
  - No install/post-install scripts declared.
  - No production dependencies, so there is no dependency health to assess.
- **Provenance & Transparency**: 48/100
  - Source repository is publicly reachable at the declared URL.
  - Provenance check failed: no build-provenance attestation is published.
  - Clear OSI-approved license (MIT).
  - Actively maintained (last published 6 days ago).
  - Publishes a security disclosure policy (SECURITY.md).
- **Schema Quality & AI Usability**: 68/100
  - AI-judged instruction clarity (excellent).
  - Context-footprint check failed: tool/resource definitions use about 1119 tokens (~186/item across 6 items; 6 tools + 0 resources), over budget; trim descriptions and params.
  - Usage-examples check failed: none of the tools include examples.
- **Stability & Change Management**: 37/100
  - Stability observed for 11 of 30 days with no destabilising changes; credit accrues until the full window elapses.
- **Tool Coverage**: 100/100
  - 100% of tools have a non-trivial description (not blank, and not just the tool's name).
  - 100% of tool parameters carry a description.
- **Tool Safety**: 100/100
  - No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.
  - We read all 6 captured tool definition(s), and no name or description among them implies an irreversible operation.
  - An AI judge read all 6 captured unit(s) of tool text and found none that tries to manipulate the model reading it.
- **Capabilities**: 100/100
  - Implements a supported MCP spec version (2025-11-25); the latest is 2026-07-28.

## Install

### How do I install the Crawlerpeek MCP server?

Crawlerpeek runs locally as an npm package, launched with npx -y crawlerpeek. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.

### Claude

```bash
claude mcp add crawlerpeek-crawlerpeek -- npx -y crawlerpeek
```

### Cursor

```json
{
  "mcpServers": {
    "crawlerpeek-crawlerpeek": {
      "command": "npx",
      "args": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}
```

### VS Code

```json
{
  "servers": {
    "crawlerpeek-crawlerpeek": {
      "command": "npx",
      "args": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}
```

### Codex

```bash
codex mcp add crawlerpeek-crawlerpeek -- npx -y crawlerpeek
```

### opencode

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "crawlerpeek-crawlerpeek": {
      "type": "local",
      "command": [
        "npx",
        "-y",
        "crawlerpeek"
      ],
      "enabled": true
    }
  }
}
```

### OpenClaw

```bash
openclaw mcp add crawlerpeek-crawlerpeek --command npx --arg -y --arg crawlerpeek
```

### Hermes

```yaml
mcp_servers:
  crawlerpeek-crawlerpeek:
    command: "npx"
    args: ["-y", "crawlerpeek"]
```

### Netclaw

```json
{
  "McpServers": {
    "crawlerpeek-crawlerpeek": {
      "Transport": "stdio",
      "Command": "npx",
      "Arguments": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}
```

### Vellum

```bash
assistant mcp add crawlerpeek-crawlerpeek -t stdio -c npx -a -y crawlerpeek
```

### Other

```json
{
  "mcpServers": {
    "crawlerpeek-crawlerpeek": {
      "command": "npx",
      "args": [
        "-y",
        "crawlerpeek"
      ]
    }
  }
}
```

## Changelog

Every change recorded for this component, newest first. Days that predate change tracking, or that we cannot explain, say so: "we were watching and nothing happened" and "we were not watching" are different claims.

### 2026-11-05 (score 75, +1)

No change was recorded against any check on this day. Stability & Change Management went from 33 to 37. That category is still filling its 30-day observation window: 10 days of observed history at the previous scan, 11 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-11-02 (score 74, +1)

No change was recorded against any check on this day. Stability & Change Management went from 23 to 27. That category is still filling its 30-day observation window: 7 days of observed history at the previous scan, 8 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-10-31 (score 73, +1)

No change was recorded against any check on this day. Stability & Change Management went from 17 to 20. That category is still filling its 30-day observation window: 5 days of observed history at the previous scan, 6 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-10-30 (score 72, +15)

- [security improvement] Malware scan: unverified → pass

### 2026-10-29 (score 57, −14)

- [security regression] Malware scan: pass → unverified
- [functional regression] Schema quality: 746 → 1119
- [functional] Package version: 0.2.0 → 0.3.1

### 2026-10-27 (score 71, +1)

No change was recorded against any check on this day. Stability & Change Management went from 3 to 7. That category is still filling its 30-day observation window: 1 days of observed history at the previous scan, 2 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-10-26 (score 70, +11)

- [security improvement] Known CVEs: unverified → pass
- [functional improvement] Dependency health: unverified → pass

### 2026-10-25 (score 59)

First indexed and scored.

## MCP tools (6)

### `check_ai_access` (~232 tokens)

Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others) may access a URL: robots.txt decision per bot plus the live HTTP status when the page is fetched with each bot's user-agent string. Read-only; one request per bot.

Input parameters:

- `bots` (array): Bot tokens to check, or omit for the default 15. Known: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extend…
- `http` (boolean): Also fetch the page with each bot's user agent (default true). false checks robots.txt only.
- `url` (string, required): Public http(s) URL of the page to check.

### `see_as_bot` (~197 tokens)

Fetch one URL with a single bot's user-agent string from this machine and report status, final URL after redirects (max 5), content type, size, page title and whether the response looks like a block or challenge page. Does not prove what the real bot gets (operators verify IP ranges).

Input parameters:

- `bot` (string, required): Bot token, for example GPTBot. Known: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, meta-ex…
- `url` (string, required): Public http(s) URL.

### `verify_bot_ip` (~224 tokens)

Check whether an IP address really belongs to a crawler (for example one seen in server logs claiming to be Googlebot), using only what the operator documents: its published IP-range list or forward-confirmed reverse DNS. Without bot, checks every bot that has such a method. Reads the operators' public lists; never connects to the IP. verified null means it could not be decided.

Input parameters:

- `bot` (string): Bot token to verify against, for example Googlebot. Omit to check all. Known: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Ext…
- `ip` (string, required): IPv4 or IPv6 address.

### `generate_robots_txt` (~93 tokens)

Generate a robots.txt that welcomes AI crawlers, with optional disallowed paths and bots to close. Pure text generation; makes no network requests.

Input parameters:

- `block_bots` (array): Bot tokens to close completely, e.g. ["CCBot"].
- `disallow` (array): Paths no bot should fetch, e.g. ["/admin/"].
- `sitemap` (string): Absolute sitemap URL.

### `lint_robots_txt` (~127 tokens)

Lint a robots.txt for mistakes: HTML served instead of robots.txt, misspelled fields or bot tokens, rules before any User-agent, relative paths or sitemap URLs, duplicate or empty groups, directives Google ignores, every AI crawler blocked. Give the site's URL (fetches its /robots.txt) or the file's text. Findings carry stable codes, line numbers and a severity.

Input parameters:

- `text` (string): The robots.txt content to lint, instead of url.
- `url` (string): Public http(s) URL of the site; its /robots.txt is fetched.

### `scan_site` (~246 tokens)

Check robots.txt for many pages of one site at once: takes the URLs from its sitemap (listed in robots.txt, else /sitemap.xml; a sitemap index is followed one level) and reports per bot which URLs robots.txt blocks. Same host only, up to 200 URLs, .gz sitemaps are skipped. Read-only; fetches only robots.txt and the sitemap files, never the pages.

Input parameters:

- `bots` (array): Bot tokens to check, or omit for the default 15. Known: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extend…
- `limit` (integer): How many sitemap URLs to check (default 25).
- `url` (string, required): Public http(s) URL on the site to scan.

## Diagnostics

Captured diagnostic sections: Provenance, Dependencies. The full working is on the page: https://verifymcp.io/servers/crawlerpeek-crawlerpeek/crawlerpeek#diagnostics

## Score history

- 2026-11-05: 75
- 2026-11-04: 74
- 2026-11-03: 74
- 2026-11-02: 74
- 2026-11-01: 73
- 2026-10-31: 73
- 2026-10-30: 72
- 2026-10-29: 57
- 2026-10-28: 71
- 2026-10-27: 71
- 2026-10-26: 70
- 2026-10-25: 59

## Common questions

### What is the Crawlerpeek MCP server?

Crawlerpeek is an MCP server listed in the public MCP registry as io.github.crawlerpeek/crawlerpeek. Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only. This page covers its npm package (crawlerpeek).

### Is the Crawlerpeek MCP server safe to use?

Crawlerpeek scores 75 out of 100 on VerifyMCP. We found no known CVEs affecting it as of 5 November 2026. It declares no install or post-install scripts. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.

### What tools does the Crawlerpeek MCP server expose?

Crawlerpeek exposes 6 tools: check_ai_access, see_as_bot, verify_bot_ip, generate_robots_txt, lint_robots_txt, scan_site. Their descriptions and schemas cost roughly 1,119 tokens of context every time the server is loaded.

### Is the Crawlerpeek MCP server still maintained?

Crawlerpeek is still listed as active in the MCP registry. We last reached this channel on 5 November 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.

### What licence is the Crawlerpeek MCP server under?

Crawlerpeek declares the MIT licence, which is OSI-approved. That covers the source only, and says nothing about the cost of any service it calls.

## Links

- npm package: https://www.npmjs.com/package/crawlerpeek
- Socket report: https://socket.dev/npm/package/crawlerpeek
- Repository: https://github.com/crawlerpeek/crawlerpeek
- Website: https://crawlerpeek.com/
- Changelog RSS feed: https://verifymcp.io/servers/crawlerpeek-crawlerpeek/crawlerpeek.xml
- Changelog JSON feed: https://verifymcp.io/servers/crawlerpeek-crawlerpeek/crawlerpeek.json
- HTML version of this page: https://verifymcp.io/servers/crawlerpeek-crawlerpeek/crawlerpeek