# Crawl Census (remote · crawlcensus.com)

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

- Trust score: 72/100 (medium)
- Change this week: +3
- Registry status: active
- Liveness: live
- Owner verified: no
- Last scored: 2026-09-20

## Components

- remote · `crawlcensus.com`: 72/100 (this document), [markdown](https://verifymcp.io/servers/taylorsmithgg-crawl-census/crawlcensus.md), [page](https://verifymcp.io/servers/taylorsmithgg-crawl-census/crawlcensus)

## Channel facts

- Endpoint: `https://crawlcensus.com/mcp`
- Transports: `streamable-http`
- Auth: `none`
- Version: `1.0.0`

## Trust breakdown

How this component scores in each security and reliability category. Every signal is checked automatically against the live server, and we only credit what we can confirm. Scores are 0–100 per category. Scoring method: https://verifymcp.io/docs/scoring (what has changed: https://verifymcp.io/docs/scoring/changelog)

Scored 2026-09-20.

- **Endpoint Security**: 46/100
  - The endpoint's TLS certificate is valid, in date, and uses a strong key.
  - Authorisation not fully verified: no authorisation is required to call this server, and 7 tool(s) never declared a destructiveHint. The MCP spec treats an absent hint as destructive by default, so we cannot call this surface safe.
  - HTTPS check failed: the endpoint is reachable over plaintext HTTP.
  - HSTS check failed: the Strict-Transport-Security header is absent.
  - DNSSEC check failed: this domain isn't protected by DNSSEC.
- **Transport & Reachability**: 100/100
  - Verified streamable-http transport via a live MCP handshake.
- **Schema Quality & AI Usability**: 81/100
  - AI-judged instruction clarity (excellent).
  - Context-footprint check failed: tool/resource definitions use about 824 tokens (~117/item across 7 items; 7 tools + 0 resources), over budget; trim descriptions and params.
  - Usage-examples check failed: none of the tools include examples.
- **Stability & Change Management**: 93/100
  - Stability observed for 28 of 30 days with no destabilising changes; credit accrues until the full window elapses.
- **Tool Coverage**: 100/100
  - 100% of tools have a non-trivial description (not blank, and not just the tool's name).
  - 100% of tool parameters carry a description.
- **Tool Safety**: 100/100
  - No prompt-injection markers were found in the server instructions, tool names or descriptions we captured.
  - We read all 7 captured tool definition(s), and no name or description among them implies an irreversible operation.
  - An AI judge read all 8 captured unit(s) of tool text and found none that tries to manipulate the model reading it.
- **Capabilities**: 60/100
  - Spec-recency check failed: implements MCP spec 2025-06-18; the latest is 2026-07-28.

## Install

### How do I install the Crawl Census MCP server?

Crawl Census is a hosted endpoint at https://crawlcensus.com/mcp, so there is nothing to install locally. Ready-made configuration for Claude, Cursor, VS Code, Codex and 5 more is on this page, copied from each client's own documentation.

### Claude

```bash
claude mcp add --transport http taylorsmithgg-crawl-census 'https://crawlcensus.com/mcp'
```

### Cursor

```json
{
  "mcpServers": {
    "taylorsmithgg-crawl-census": {
      "url": "https://crawlcensus.com/mcp"
    }
  }
}
```

### VS Code

```json
{
  "servers": {
    "taylorsmithgg-crawl-census": {
      "type": "http",
      "url": "https://crawlcensus.com/mcp"
    }
  }
}
```

### Codex

```toml
[mcp_servers.taylorsmithgg-crawl-census]
url = "https://crawlcensus.com/mcp"
```

### opencode

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "taylorsmithgg-crawl-census": {
      "type": "remote",
      "url": "https://crawlcensus.com/mcp",
      "enabled": true
    }
  }
}
```

### OpenClaw

```bash
openclaw mcp add taylorsmithgg-crawl-census --url 'https://crawlcensus.com/mcp' --transport streamable-http
```

### Hermes

```yaml
mcp_servers:
  taylorsmithgg-crawl-census:
    url: "https://crawlcensus.com/mcp"
```

### Netclaw

```json
{
  "McpServers": {
    "taylorsmithgg-crawl-census": {
      "Transport": "http",
      "Url": "https://crawlcensus.com/mcp"
    }
  }
}
```

### Vellum

```bash
assistant mcp add taylorsmithgg-crawl-census -t streamable-http -u 'https://crawlcensus.com/mcp'
```

### Other

```json
{
  "mcpServers": {
    "taylorsmithgg-crawl-census": {
      "type": "http",
      "url": "https://crawlcensus.com/mcp"
    }
  }
}
```

The mcpServers block is a cross-client convention. Remote transports vary, so check your client's docs.

## Changelog

Every change recorded for this component, newest first. Days that predate change tracking, or that we cannot explain, say so: "we were watching and nothing happened" and "we were not watching" are different claims.

### 2026-09-18 (score 72, +1)

No change was recorded against any check on this day. Stability & Change Management went from 83 to 87. That category is still filling its 30-day observation window: 25 days of observed history at the previous scan, 26 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-09-16 (score 71, +1)

No change was recorded against any check on this day. Stability & Change Management went from 77 to 80. That category is still filling its 30-day observation window: 23 days of observed history at the previous scan, 24 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-09-14 (score 70, +1)

No change was recorded against any check on this day. Stability & Change Management went from 70 to 73. That category is still filling its 30-day observation window: 21 days of observed history at the previous scan, 22 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-09-12 (score 69, +1)

No change was recorded against any check on this day. Stability & Change Management went from 63 to 67. That category is still filling its 30-day observation window: 19 days of observed history at the previous scan, 20 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-09-10 (score 68, +1)

No change was recorded against any check on this day. Stability & Change Management went from 57 to 60. That category is still filling its 30-day observation window: 17 days of observed history at the previous scan, 18 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-09-08 (score 67, +1)

No change was recorded against any check on this day. Stability & Change Management went from 50 to 53. That category is still filling its 30-day observation window: 15 days of observed history at the previous scan, 16 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-09-06 (score 66, +1)

No change was recorded against any check on this day. Stability & Change Management went from 43 to 47. That category is still filling its 30-day observation window: 13 days of observed history at the previous scan, 14 at this one. The score rises as the window fills, whether or not the server changes.

### 2026-09-03 (score 65, +1)

No change was recorded against any check on this day. Stability & Change Management went from 33 to 37. That category is still filling its 30-day observation window: 10 days of observed history at the previous scan, 11 at this one. The score rises as the window fills, whether or not the server changes.

## MCP tools (7)

### `scan_site` (~71 tokens)

Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.

Input parameters:

- `domain` (string, required): Bare hostname, for example example.com

### `site_report` (~42 tokens)

Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.

Input parameters:

- `domain` (string, required): Bare hostname

### `census_stats` (~30 tokens)

Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.

### `crawl_preflight` (~277 tokens)

Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.

Input parameters:

- `agent` (string, required): Crawler token, e.g. gptbot, claudebot, perplexitybot, oai-searchbot, ccbot.
- `domains` (array, required): Domains to check. Up to 25 per call anonymously; send an Authorization: Bearer key for more. An over-large batch is refused outright rather than partly answered.

### `agent_profile` (~129 tokens)

What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.

Input parameters:

- `agent` (string, required): Crawler token, e.g. gptbot, claudebot, ccbot, google-extended.

### `census_facts` (~103 tokens)

Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.

### `submit_domains` (~146 tokens)

Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.

Input parameters:

- `domains` (array, required): Hostnames to queue. Up to 50 per call anonymously; an over-large batch is refused outright rather than partly queued.

## Diagnostics

Captured diagnostic sections: TLS, DNSSEC, Authorisation, Transports. The full working is on the page: https://verifymcp.io/servers/taylorsmithgg-crawl-census/crawlcensus#diagnostics

## Score history

- 2026-09-20: 72
- 2026-09-19: 72
- 2026-09-18: 72
- 2026-09-17: 71
- 2026-09-16: 71
- 2026-09-15: 70
- 2026-09-14: 70
- 2026-09-13: 69
- 2026-09-12: 69
- 2026-09-11: 68
- 2026-09-10: 68
- 2026-09-09: 67
- 2026-09-08: 67
- 2026-09-07: 66
- 2026-09-06: 66
- 2026-09-05: 65
- 2026-09-04: 65
- 2026-09-03: 65
- 2026-09-02: 64
- 2026-09-01: 64
- 2026-08-31: 63
- 2026-08-30: 63
- 2026-08-29: 62
- 2026-08-28: 62
- 2026-08-27: 61
- 2026-08-26: 61
- 2026-08-25: 59
- 2026-08-24: 58
- 2026-08-23: 58

## Common questions

### What is the Crawl Census MCP server?

Crawl Census is an MCP server listed in the public MCP registry as io.github.taylorsmithgg/crawl-census. Ask before you fetch: will this domain serve your crawler, refuse it, or charge it? This page covers its hosted endpoint (https://crawlcensus.com/mcp).

### Is the Crawl Census MCP server safe to use?

Crawl Census scores 72 out of 100 on VerifyMCP. That is a record of what we were able to check automatically, not an endorsement. The category breakdown on this page shows every signal behind the number, including the ones we could not confirm.

### What tools does the Crawl Census MCP server expose?

Crawl Census exposes 7 tools: scan_site, site_report, census_stats, crawl_preflight, agent_profile, and 2 more. Their descriptions and schemas cost roughly 798 tokens of context every time the server is loaded.

### Does the Crawl Census MCP server require authentication?

No. We connected to Crawl Census without credentials and it answered, so anything it exposes is reachable by anyone who knows the address.

### Is the Crawl Census MCP server still maintained?

Crawl Census is still listed as active in the MCP registry. We last reached this channel on 20 September 2026. Those dates come from our own scans of the registry and the channel itself, not from anything the publisher announced.

## Links

- Remote endpoint: https://crawlcensus.com/mcp
- Repository: https://github.com/taylorsmithgg/crawl-census-client
- Website: https://crawlcensus.com/for-crawlers
- Changelog RSS feed: https://verifymcp.io/servers/taylorsmithgg-crawl-census/crawlcensus.xml
- Changelog JSON feed: https://verifymcp.io/servers/taylorsmithgg-crawl-census/crawlcensus.json
- HTML version of this page: https://verifymcp.io/servers/taylorsmithgg-crawl-census/crawlcensus