Robots.txt & AI Crawler Checker
REMOTE · ROBOTS-CHECK.QUIETWORKLABS.WORKERS.DEV
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
Available components
Last scanned (1 hour ago)
Scores higher than 62% of remote servers
Reachable. Last reached 5 Nov 2026.
Category Web Scraping
How this component scores in each security and reliability category. Every signal is checked automatically against the live server, and we only credit what we can confirm. How we score → Why this is hard to score →
Endpoint Security63
- The endpoint's TLS certificate is valid, in date, and uses a strong key. View diagnostics → Pass
- No authorisation is required to call this server. Every tool declares its destructiveHint and none is destructive, so open access doesn't expose one. See how to fix → View diagnostics → Partial
- HTTPS enforcement could not be verified: the plaintext port answered with HTTP 405, which proves neither a plaintext path nor enforcement. View diagnostics → Unverified
- HSTS check failed: the Strict-Transport-Security header is absent. See how to fix → View diagnostics → Fail
- DNSSEC check failed: this domain isn't protected by DNSSEC. See how to fix → View diagnostics → Fail
Transport & Reachability100
- Verified streamable-http transport via a live MCP handshake. View diagnostics → Pass
Schema Quality & AI Usability67
- AI-judged instruction clarity (excellent).Pass
- Context-footprint check failed: tool/resource definitions use about 672 tokens (~224/item across 3 items; 3 tools + 0 resources), over budget; trim descriptions and params. See how to fix → Fail
- Usage-examples check failed: none of the tools include examples. See how to fix → Fail
Stability & Change Management60
- Stability observed for 18 of 30 days with no destabilising changes; credit accrues until the full window elapses.Partial
Tool Coverage100
- 100% of tools have a non-trivial description (not blank, and not just the tool's name).Pass
- 100% of tool parameters carry a description.Pass
Tool Safety100
- No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.Pass
- We read all 3 captured tool definition(s), and no name or description among them implies an irreversible operation.Pass
- An AI judge read all 4 captured unit(s) of tool text and found none that tries to manipulate the model reading it.Pass
Capabilities100
- Implements a supported MCP spec version (2025-11-25); the latest is 2026-07-28.Pass
Every change we have recorded for this component, newest first. Security-relevant changes are always shown. ▲ marks a change for the better, ▼ a change for the worse; unmarked changes are neutral.
- 5 Nov 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 57 to 60. That category is still filling its 30-day observation window: 17 days of observed history at the previous scan, 18 at this one. The score rises as the window fills, whether or not the server changes.
- 3 Nov 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 50 to 53. That category is still filling its 30-day observation window: 15 days of observed history at the previous scan, 16 at this one. The score rises as the window fills, whether or not the server changes.
- 1 Nov 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 43 to 47. That category is still filling its 30-day observation window: 13 days of observed history at the previous scan, 14 at this one. The score rises as the window fills, whether or not the server changes.
- 30 Oct 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 37 to 40. That category is still filling its 30-day observation window: 11 days of observed history at the previous scan, 12 at this one. The score rises as the window fills, whether or not the server changes.
- 28 Oct 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 30 to 33. That category is still filling its 30-day observation window: 9 days of observed history at the previous scan, 10 at this one. The score rises as the window fills, whether or not the server changes.
- 26 Oct 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 23 to 27. That category is still filling its 30-day observation window: 7 days of observed history at the previous scan, 8 at this one. The score rises as the window fills, whether or not the server changes.
- 23 Oct 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 13 to 17. That category is still filling its 30-day observation window: 4 days of observed history at the previous scan, 5 at this one. The score rises as the window fills, whether or not the server changes.
- 21 Oct 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 7 to 10. That category is still filling its 30-day observation window: 2 days of observed history at the previous scan, 3 at this one. The score rises as the window fills, whether or not the server changes.
Diagnostic detail from the automated scan of this channel: what the scanner observed at each step, so you can see exactly where a check passed or failed. It is informational only and never changes the trust score.
Captured 5 Nov 2026 · Probed https://robots-check.quietworklabs.workers.dev/mcp
TLS valid
Negotiated TLS 1.3 with TLS_AES_128_GCM_SHA256 .
| Subject | Issuer | Valid from | Valid until | Key | Signature | Serial |
|---|---|---|---|---|---|---|
| CN=quietworklabs.workers.dev | CN=YE1,O=Let's Encrypt,C=US | 18 Oct 2026 | 16 Jan 2027 | ECDSA 256 | ECDSA-SHA384 | 6203662d048034ad6156b34972b93b5ae9a |
| SANs: *.quietworklabs.workers.dev, quietworklabs.workers.dev | ||||||
| CN=YE1,O=Let's Encrypt,C=US (CA) | CN=Root YE,O=ISRG,C=US | 3 Sept 2025 | 2 Sept 2028 | ECDSA 384 | ECDSA-SHA384 | 5ddd70dd31f801c85c186a7a04b80afe |
| CN=Root YE,O=ISRG,C=US (CA) | CN=ISRG Root X2,O=Internet Security Research Group,C=US | 13 May 2026 | 2 Sept 2032 | ECDSA 384 | ECDSA-SHA384 | 872165fc34b6e5fba8add5b3705fb53a |
| CN=ISRG Root X2,O=Internet Security Research Group,C=US (CA) | CN=ISRG Root X1,O=Internet Security Research Group,C=US | 13 May 2026 | 2 Sept 2032 | ECDSA 384 | SHA256-RSA | 6c8f1dc727c7117f7baf853ac980f9cd |
Background: What to check on a remote MCP endpoint →
DNSSEC insecure
Validation of robots-check.quietworklabs.workers.dev. — Not signed
| Zone | DS | Keys | Algorithms | Outcome |
|---|---|---|---|---|
| . | trust_anchor | 20326, 38696 | 8, 8 | Verified |
| dev. | present | 60074 | 8 | Verified |
| workers.dev. | absent | Unsigned (proven) parent-signed NSEC/NSEC3 proves an unsigned delegation |
Authentication No authorisation required
The endpoint answered without asking for a token. Anyone who knows the URL can reach it.
| Result | No authorisation required |
|---|---|
| HTTP status | 200 |
Background: How OAuth 2.1 works in the 2026 MCP spec →
Transports 2 probes
| Transport | URL | Outcome | Status | Location |
|---|---|---|---|---|
| streamable-http | https://robots-check.quietworklabs.workers.dev/mcp | Verified | 200 | |
| http (plaintext) | http://robots-check.quietworklabs.workers.dev/mcp | Inconclusive | 405 |
The tools this component advertises to a client, with an estimated token cost for each. Expand a tool to see its parameters and schema. The per-tool counts are indicative and are not scored directly; the schema's total context footprint is one signal in Schema Quality & AI Usability. A tool's description is untrusted text the model reads on every call, which is what makes this list a security surface and not just an inventory: how tool poisoning works →
ai_crawler_access Which AI crawlers a site allows or blocks ~160
AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
| Name | Type | Req | Description |
|---|---|---|---|
| url | string | yes | Site or page URL |
No output schema declared.
No examples provided.
robots_check May this user-agent crawl this URL? (RFC 9309) ~159
Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.
| Name | Type | Req | Description |
|---|---|---|---|
| url | string | yes | Page URL to test |
| user_agent | string | – | Crawler product token, e.g. GPTBot or MyAgent. Default * |
No output schema declared.
No examples provided.
sitemap_check Find and validate an XML sitemap ~130
Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
| Name | Type | Req | Description |
|---|---|---|---|
| url | string | yes | Site URL (discovers the sitemap) or a sitemap URL |
No output schema declared.
No examples provided.
How do I install the Robots.txt & AI Crawler Checker MCP server?
Robots.txt & AI Crawler Checker is a hosted endpoint at https://robots-check.quietworklabs.workers.dev/mcp, so there is nothing to install locally. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.
remote · robots-check.quietworklabs.workers.dev
claude mcp add --transport http dev-workers-quietworklabs-robots-check-robots-ch 'https://robots-check.quietworklabs.workers.dev/mcp'
What is the Robots.txt & AI Crawler Checker MCP server?
Robots.txt & AI Crawler Checker is an MCP server listed in the public MCP registry as dev.workers.quietworklabs.robots-check/robots-check. robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check. This page covers its hosted endpoint (https://robots-check.quietworklabs.workers.dev/mcp).
Is the Robots.txt & AI Crawler Checker MCP server safe to use?
Robots.txt & AI Crawler Checker scores 74 out of 100 on VerifyMCP. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.
What tools does the Robots.txt & AI Crawler Checker MCP server expose?
Robots.txt & AI Crawler Checker exposes 3 tools: robots_check, ai_crawler_access, sitemap_check. Their descriptions and schemas cost roughly 449 tokens of context every time the server is loaded.
Does the Robots.txt & AI Crawler Checker MCP server require authentication?
No. We connected to Robots.txt & AI Crawler Checker without credentials and it answered, so anything it exposes is reachable by anyone who knows the address.
Is the Robots.txt & AI Crawler Checker MCP server still maintained?
Robots.txt & AI Crawler Checker is still listed as active in the MCP registry. We last reached this channel on 5 November 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.