Crawl Census
REMOTE · CRAWLCENSUS.COM · SCANNED SEP 20
Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?
Available components
How this component scores in each security and reliability category. Every signal is checked automatically against the live server, and we only credit what we can confirm. How we score → Why this is hard to score →
Endpoint Security46
- The endpoint's TLS certificate is valid, in date, and uses a strong key. View diagnostics → Pass
- Authorisation not fully verified: no authorisation is required to call this server, and 7 tool(s) never declared a destructiveHint. The MCP spec treats an absent hint as destructive by default, so we cannot call this surface safe. See how to fix → View diagnostics → Unverified
- HTTPS check failed: the endpoint is reachable over plaintext HTTP. See how to fix → View diagnostics → Fail
- HSTS check failed: the Strict-Transport-Security header is absent. See how to fix → View diagnostics → Fail
- DNSSEC check failed: this domain isn't protected by DNSSEC. See how to fix → View diagnostics → Fail
Transport & Reachability100
- Verified streamable-http transport via a live MCP handshake. View diagnostics → Pass
Schema Quality & AI Usability81
- AI-judged instruction clarity (excellent).Pass
- Context-footprint check failed: tool/resource definitions use about 824 tokens (~117/item across 7 items; 7 tools + 0 resources), over budget; trim descriptions and params. See how to fix → Fail
- Usage-examples check failed: none of the tools include examples. See how to fix → Fail
Stability & Change Management93
- Stability observed for 28 of 30 days with no destabilising changes; credit accrues until the full window elapses.Partial
Tool Coverage100
- 100% of tools have a non-trivial description (not blank, and not just the tool's name).Pass
- 100% of tool parameters carry a description.Pass
Tool Safety100
- No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.Pass
- We read all 7 captured tool definition(s), and no name or description among them implies an irreversible operation.Pass
- An AI judge read all 8 captured unit(s) of tool text and found none that tries to manipulate the model reading it.Pass
Capabilities60
- Spec-recency check failed: implements MCP spec 2025-06-18; the latest is 2026-07-28. See how to fix → Fail
How do I install the Crawl Census MCP server?
Crawl Census is a hosted endpoint at https://crawlcensus.com/mcp, so there is nothing to install locally. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.
remote · crawlcensus.com
claude mcp add --transport http taylorsmithgg-crawl-census 'https://crawlcensus.com/mcp'
{
"mcpServers": {
"taylorsmithgg-crawl-census": {
"url": "https://crawlcensus.com/mcp"
}
}
} {
"servers": {
"taylorsmithgg-crawl-census": {
"type": "http",
"url": "https://crawlcensus.com/mcp"
}
}
} [mcp_servers.taylorsmithgg-crawl-census] url = "https://crawlcensus.com/mcp"
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"taylorsmithgg-crawl-census": {
"type": "remote",
"url": "https://crawlcensus.com/mcp",
"enabled": true
}
}
} openclaw mcp add taylorsmithgg-crawl-census --url 'https://crawlcensus.com/mcp' --transport streamable-http
mcp_servers:
taylorsmithgg-crawl-census:
url: "https://crawlcensus.com/mcp" {
"McpServers": {
"taylorsmithgg-crawl-census": {
"Transport": "http",
"Url": "https://crawlcensus.com/mcp"
}
}
} assistant mcp add taylorsmithgg-crawl-census -t streamable-http -u 'https://crawlcensus.com/mcp'
{
"mcpServers": {
"taylorsmithgg-crawl-census": {
"type": "http",
"url": "https://crawlcensus.com/mcp"
}
}
} The mcpServers block is a cross-client convention. Remote transports vary, so check your client's docs.
Every change we have recorded for this component, newest first. Security-relevant changes are always shown. ▲ marks a change for the better, ▼ a change for the worse; unmarked changes are neutral.
- 18 Sept 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 83 to 87. That category is still filling its 30-day observation window: 25 days of observed history at the previous scan, 26 at this one. The score rises as the window fills, whether or not the server changes.
- 16 Sept 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 77 to 80. That category is still filling its 30-day observation window: 23 days of observed history at the previous scan, 24 at this one. The score rises as the window fills, whether or not the server changes.
- 14 Sept 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 70 to 73. That category is still filling its 30-day observation window: 21 days of observed history at the previous scan, 22 at this one. The score rises as the window fills, whether or not the server changes.
- 12 Sept 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 63 to 67. That category is still filling its 30-day observation window: 19 days of observed history at the previous scan, 20 at this one. The score rises as the window fills, whether or not the server changes.
- 10 Sept 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 57 to 60. That category is still filling its 30-day observation window: 17 days of observed history at the previous scan, 18 at this one. The score rises as the window fills, whether or not the server changes.
- 8 Sept 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 50 to 53. That category is still filling its 30-day observation window: 15 days of observed history at the previous scan, 16 at this one. The score rises as the window fills, whether or not the server changes.
- 6 Sept 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 43 to 47. That category is still filling its 30-day observation window: 13 days of observed history at the previous scan, 14 at this one. The score rises as the window fills, whether or not the server changes.
- 3 Sept 26 +1
No change was recorded against any check on this day. Stability & Change Management went from 33 to 37. That category is still filling its 30-day observation window: 10 days of observed history at the previous scan, 11 at this one. The score rises as the window fills, whether or not the server changes.
Diagnostic detail from the automated scan of this channel: what the scanner observed at each step, so you can see exactly where a check passed or failed. It is informational only and never changes the trust score.
Captured 20 Sept 2026 · Probed https://crawlcensus.com/mcp
TLS valid
Negotiated TLS 1.3 with TLS_AES_128_GCM_SHA256 .
| Subject | Issuer | Valid from | Valid until | Key | Signature | Serial |
|---|---|---|---|---|---|---|
| CN=crawlcensus.com | CN=WE1,O=Google Trust Services,C=US | 22 Aug 2026 | 20 Nov 2026 | ECDSA 256 | ECDSA-SHA256 | 7323c363e3752f00134db73e6622d439 |
| SANs: crawlcensus.com, www.crawlcensus.com, *.www.crawlcensus.com | ||||||
| CN=WE1,O=Google Trust Services,C=US (CA) | CN=GTS Root R4,O=Google Trust Services LLC,C=US | 13 Dec 2023 | 20 Feb 2029 | ECDSA 256 | ECDSA-SHA384 | 7ff31977972c224a76155d13b6d685e3 |
| CN=GTS Root R4,O=Google Trust Services LLC,C=US (CA) | CN=GlobalSign Root CA,OU=Root CA,O=GlobalSign nv-sa,C=BE | 15 Nov 2023 | 28 Jan 2028 | ECDSA 384 | SHA256-RSA | 7fe530bf331343bedd821610493d8a1b |
Background: What to check on a remote MCP endpoint →
DNSSEC insecure
Validation of crawlcensus.com. — Not signed
| Zone | DS | Keys | Algorithms | Outcome |
|---|---|---|---|---|
| . | trust_anchor | 20326, 38696 | 8, 8 | Verified |
| com. | present | 19718 | 13 | Verified |
| crawlcensus.com. | absent | Unsigned (proven) parent-signed NSEC/NSEC3 proves an unsigned delegation |
Authentication No authorisation required
The endpoint answered without asking for a token. Anyone who knows the URL can reach it.
| Result | No authorisation required |
|---|---|
| HTTP status | 200 |
Background: How OAuth 2.1 works in the 2026 MCP spec →
Transports 2 probes
| Transport | URL | Outcome | Status | Location |
|---|---|---|---|---|
| streamable-http | https://crawlcensus.com/mcp | Verified | 200 | |
| http (plaintext) | http://crawlcensus.com/mcp | Served over HTTP | 200 |
The tools this component advertises to a client, with an estimated token cost for each. Expand a tool to see its parameters and schema. The per-tool counts are indicative and are not scored directly; the schema's total context footprint is one signal in Schema Quality & AI Usability. A tool's description is untrusted text the model reads on every call, which is what makes this list a security surface and not just an inventory: how tool poisoning works →
agent_profile ~129
What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.
| Name | Type | Req | Description |
|---|---|---|---|
| agent | string | yes | Crawler token, e.g. gptbot, claudebot, ccbot, google-extended. |
No output schema declared.
No examples provided.
census_facts ~103
Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.
Input schema present but exposes no named parameters.
No output schema declared.
No examples provided.
census_stats ~30
Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.
Input schema present but exposes no named parameters.
No output schema declared.
No examples provided.
crawl_preflight ~277
Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.
| Name | Type | Req | Description |
|---|---|---|---|
| agent | string | yes | Crawler token, e.g. gptbot, claudebot, perplexitybot, oai-searchbot, ccbot. |
| domains | array | yes | Domains to check. Up to 25 per call anonymously; send an Authorization: Bearer key for more. An over-large batch is refused outright rather than partly answered. |
No output schema declared.
No examples provided.
scan_site ~71
Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.
| Name | Type | Req | Description |
|---|---|---|---|
| domain | string | yes | Bare hostname, for example example.com |
No output schema declared.
No examples provided.
site_report ~42
Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.
| Name | Type | Req | Description |
|---|---|---|---|
| domain | string | yes | Bare hostname |
No output schema declared.
No examples provided.
submit_domains ~146
Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.
| Name | Type | Req | Description |
|---|---|---|---|
| domains | array | yes | Hostnames to queue. Up to 50 per call anonymously; an over-large batch is refused outright rather than partly queued. |
No output schema declared.
No examples provided.
What is the Crawl Census MCP server?
Crawl Census is an MCP server listed in the public MCP registry as io.github.taylorsmithgg/crawl-census. Ask before you fetch: will this domain serve your crawler, refuse it, or charge it? This page covers its hosted endpoint (https://crawlcensus.com/mcp).
Is the Crawl Census MCP server safe to use?
Crawl Census scores 72 out of 100 on VerifyMCP. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.
What tools does the Crawl Census MCP server expose?
Crawl Census exposes 7 tools: scan_site, site_report, census_stats, crawl_preflight, agent_profile, and 2 more. Their descriptions and schemas cost roughly 798 tokens of context every time the server is loaded.
Does the Crawl Census MCP server require authentication?
No. We connected to Crawl Census without credentials and it answered, so anything it exposes is reachable by anyone who knows the address.
Is the Crawl Census MCP server still maintained?
Crawl Census is still listed as active in the MCP registry. We last reached this channel on 20 September 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.