# Robots.txt & AI Crawler Checker (remote · robots-check.quietworklabs.workers.dev)

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

- Trust score: 74/100 (medium)
- Change this week: +4
- Registry status: active
- Liveness: live
- Owner verified: no
- Last scored: 2026-11-05

## Components

- remote · `robots-check.quietworklabs.workers.dev`: 74/100 (this document), [markdown](https://verifymcp.io/servers/dev-workers-quietworklabs-robots-check-robots-check/robots-check.md), [page](https://verifymcp.io/servers/dev-workers-quietworklabs-robots-check-robots-check/robots-check)

## Channel facts

- Endpoint: `https://robots-check.quietworklabs.workers.dev/mcp`
- Transports: `streamable-http`
- Auth: `none`
- Version: `1.0.0`

## Trust breakdown

How this component scores in each security and reliability category. Every signal is checked automatically against the live server, and we only credit what we can confirm. Scores are 0–100 per category. Scoring method: https://verifymcp.io/docs/scoring (what has changed: https://verifymcp.io/docs/scoring/changelog)

Scored 2026-11-05.

- **Endpoint Security**: 63/100
  - The endpoint's TLS certificate is valid, in date, and uses a strong key.
  - No authorisation is required to call this server. Every tool declares its destructiveHint and none is destructive, so open access doesn't expose one.
  - HTTPS enforcement could not be verified: the plaintext port answered with HTTP 405, which proves neither a plaintext path nor enforcement.
  - HSTS check failed: the Strict-Transport-Security header is absent.
  - DNSSEC check failed: this domain isn't protected by DNSSEC.
- **Transport & Reachability**: 100/100
  - Verified streamable-http transport via a live MCP handshake.
- **Schema Quality & AI Usability**: 67/100
  - AI-judged instruction clarity (excellent).
  - Context-footprint check failed: tool/resource definitions use about 672 tokens (~224/item across 3 items; 3 tools + 0 resources), over budget; trim descriptions and params.
  - Usage-examples check failed: none of the tools include examples.
- **Stability & Change Management**: 60/100
  - Stability observed for 18 of 30 days with no destabilising changes; credit accrues until the full window elapses.
- **Tool Coverage**: 100/100
  - 100% of tools have a non-trivial description (not blank, and not just the tool's name).
  - 100% of tool parameters carry a description.
- **Tool Safety**: 100/100
  - No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.
  - We read all 3 captured tool definition(s), and no name or description among them implies an irreversible operation.
  - An AI judge read all 4 captured unit(s) of tool text and found none that tries to manipulate the model reading it.
- **Capabilities**: 100/100
  - Implements a supported MCP spec version (2025-11-25); the latest is 2026-07-28.

## Install

### How do I install the Robots.txt & AI Crawler Checker MCP server?

Robots.txt & AI Crawler Checker is a hosted endpoint at https://robots-check.quietworklabs.workers.dev/mcp, so there is nothing to install locally. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.

### Claude

```bash
claude mcp add --transport http dev-workers-quietworklabs-robots-check-robots-ch 'https://robots-check.quietworklabs.workers.dev/mcp'
```

### Cursor

```json
{
  "mcpServers": {
    "dev-workers-quietworklabs-robots-check-robots-ch": {
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}
```

### VS Code

```json
{
  "servers": {
    "dev-workers-quietworklabs-robots-check-robots-ch": {
      "type": "http",
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}
```

### Codex

```toml
[mcp_servers.dev-workers-quietworklabs-robots-check-robots-ch]
url = "https://robots-check.quietworklabs.workers.dev/mcp"
```

### opencode

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "dev-workers-quietworklabs-robots-check-robots-ch": {
      "type": "remote",
      "url": "https://robots-check.quietworklabs.workers.dev/mcp",
      "enabled": true
    }
  }
}
```

### OpenClaw

```bash
openclaw mcp add dev-workers-quietworklabs-robots-check-robots-ch --url 'https://robots-check.quietworklabs.workers.dev/mcp' --transport streamable-http
```

### Hermes

```yaml
mcp_servers:
  dev-workers-quietworklabs-robots-check-robots-ch:
    url: "https://robots-check.quietworklabs.workers.dev/mcp"
```

### Netclaw

```json
{
  "McpServers": {
    "dev-workers-quietworklabs-robots-check-robots-ch": {
      "Transport": "http",
      "Url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}
```

### Vellum

```bash
assistant mcp add dev-workers-quietworklabs-robots-check-robots-ch -t streamable-http -u 'https://robots-check.quietworklabs.workers.dev/mcp'
```

### Other

```json
{
  "mcpServers": {
    "dev-workers-quietworklabs-robots-check-robots-ch": {
      "type": "http",
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}
```

The mcpServers block is a cross-client convention. Remote transports vary, so check your client's docs.

## Changelog

Every change recorded for this component, newest first. Days that predate change tracking, or that we cannot explain, say so: "we were watching and nothing happened" and "we were not watching" are different claims.

### 2026-11-05 (score 74, +1)

No change was recorded against any check on this day. Stability & Change Management went from 57 to 60. That category is still filling its 30-day observation window: 17 days of observed history at the previous scan, 18 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-11-03 (score 73, +1)

No change was recorded against any check on this day. Stability & Change Management went from 50 to 53. That category is still filling its 30-day observation window: 15 days of observed history at the previous scan, 16 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-11-01 (score 72, +1)

No change was recorded against any check on this day. Stability & Change Management went from 43 to 47. That category is still filling its 30-day observation window: 13 days of observed history at the previous scan, 14 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-10-30 (score 71, +1)

No change was recorded against any check on this day. Stability & Change Management went from 37 to 40. That category is still filling its 30-day observation window: 11 days of observed history at the previous scan, 12 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-10-28 (score 70, +1)

No change was recorded against any check on this day. Stability & Change Management went from 30 to 33. That category is still filling its 30-day observation window: 9 days of observed history at the previous scan, 10 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-10-26 (score 69, +1)

No change was recorded against any check on this day. Stability & Change Management went from 23 to 27. That category is still filling its 30-day observation window: 7 days of observed history at the previous scan, 8 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-10-23 (score 68, +1)

No change was recorded against any check on this day. Stability & Change Management went from 13 to 17. That category is still filling its 30-day observation window: 4 days of observed history at the previous scan, 5 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-10-21 (score 67, +1)

No change was recorded against any check on this day. Stability & Change Management went from 7 to 10. That category is still filling its 30-day observation window: 2 days of observed history at the previous scan, 3 at this one. The score rises as the window fills, whether or not the server changes.

## MCP tools (3)

### `robots_check` (~159 tokens)

May this user-agent crawl this URL? (RFC 9309)

Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.

Input parameters:

- `url` (string, required): Page URL to test
- `user_agent` (string): Crawler product token, e.g. GPTBot or MyAgent. Default *

### `ai_crawler_access` (~160 tokens)

Which AI crawlers a site allows or blocks

AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

Input parameters:

- `url` (string, required): Site or page URL

### `sitemap_check` (~130 tokens)

Find and validate an XML sitemap

Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

Input parameters:

- `url` (string, required): Site URL (discovers the sitemap) or a sitemap URL

## Diagnostics

Captured diagnostic sections: TLS, DNSSEC, Authorisation, Transports. The full working is on the page: https://verifymcp.io/servers/dev-workers-quietworklabs-robots-check-robots-check/robots-check#diagnostics

## Score history

- 2026-11-05: 74
- 2026-11-04: 73
- 2026-11-03: 73
- 2026-11-02: 72
- 2026-11-01: 72
- 2026-10-31: 71
- 2026-10-30: 71
- 2026-10-29: 70
- 2026-10-28: 70
- 2026-10-27: 69
- 2026-10-26: 69
- 2026-10-25: 68
- 2026-10-24: 68
- 2026-10-23: 68
- 2026-10-22: 67
- 2026-10-21: 67
- 2026-10-20: 66
- 2026-10-19: 66
- 2026-10-18: 65

## Common questions

### What is the Robots.txt & AI Crawler Checker MCP server?

Robots.txt & AI Crawler Checker is an MCP server listed in the public MCP registry as dev.workers.quietworklabs.robots-check/robots-check. robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check. This page covers its hosted endpoint (https://robots-check.quietworklabs.workers.dev/mcp).

### Is the Robots.txt & AI Crawler Checker MCP server safe to use?

Robots.txt & AI Crawler Checker scores 74 out of 100 on VerifyMCP. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.

### What tools does the Robots.txt & AI Crawler Checker MCP server expose?

Robots.txt & AI Crawler Checker exposes 3 tools: robots_check, ai_crawler_access, sitemap_check. Their descriptions and schemas cost roughly 449 tokens of context every time the server is loaded.

### Does the Robots.txt & AI Crawler Checker MCP server require authentication?

No. We connected to Robots.txt & AI Crawler Checker without credentials and it answered, so anything it exposes is reachable by anyone who knows the address.

### Is the Robots.txt & AI Crawler Checker MCP server still maintained?

Robots.txt & AI Crawler Checker is still listed as active in the MCP registry. We last reached this channel on 5 November 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.

## Links

- Remote endpoint: https://robots-check.quietworklabs.workers.dev/mcp
- Website: https://robots-check.quietworklabs.workers.dev/
- Changelog RSS feed: https://verifymcp.io/servers/dev-workers-quietworklabs-robots-check-robots-check/robots-check.xml
- Changelog JSON feed: https://verifymcp.io/servers/dev-workers-quietworklabs-robots-check-robots-check/robots-check.json
- HTML version of this page: https://verifymcp.io/servers/dev-workers-quietworklabs-robots-check-robots-check/robots-check