In the Crane Index™ scan of 1 October 2026, 25 of 2,295 readable websites (1%) blocked an AI answer crawler in robots.txt. The edge told a different story: 191 of the 2,207 sites probed (9%) refused at least one AI agent's request while serving a browser the same page, and 606 of the 3,368 sites approached (18%) refused the scanner outright.
The robots.txt figures are each site's own published policy. The edge figures come from requests sent from the Crane scanner's network carrying each AI agent's published name, compared with a browser request on the same pass.
A site that checks a crawler's network address may refuse that request and still admit the real crawler, so these counts show how sites treat an unverified request in an AI agent's name. How the real crawlers are treated is unmeasured. Probes with no browser baseline (88) and robots.txt files that could not be read (16) are counted as inconclusive, not as either answer.
Key statistics, 2026
- 2525 of 2,295 readable websites (1%) blocked an AI answer crawler in robots.txt. Link
- 191191 of 2,207 sites probed (9%) refused at least one AI agent's request at the edge while serving a browser. Link
- 1717 refused every one of the 5 AI agents probed. Link
- 111GPTBot was the agent most often refused, by 111 sites (5%). Link
- 44 answered an AI agent with HTTP 402, Payment Required, which the scanner reads as a pay-per-crawl response. Link
- 606606 of 3,368 sites approached (18%) refused the scanner outright, 354 of them (58%) from an edge network the scanner identified as Cloudflare. Link
robots.txt: the published policy
Most AI companies run separate crawlers for training models and for answering live questions, and robots.txt can treat them differently. OpenAI, for example, documents GPTBot as its training crawler and OAI-SearchBot as the crawler that decides whether a site can appear in ChatGPT search answers, with each setting independent of the other.
2,254 of 2,295 readable websites allowed the major AI answer crawlers. These are the answer crawlers the 25 blocking sites named:
| AI answer crawler blocked | Sites |
|---|---|
| PerplexityBot | 22 |
| Claude-SearchBot | 13 |
| OAI-SearchBot | 13 |
Among the sites that allowed the answer crawlers, these other AI crawlers were blocked, most of them crawlers that gather training data:
| Other AI crawler blocked | Sites |
|---|---|
| Bytespider | 45 |
| CCBot | 37 |
| GPTBot | 30 |
| Meta-ExternalAgent | 30 |
| Applebot-Extended | 20 |
| ClaudeBot | 19 |
| Google-Extended | 17 |
| cohere-ai | 16 |
| Diffbot | 14 |
| omgili | 13 |
| ImagesiftBot | 11 |
| OAI-AdsBot | 5 |
| Perplexity-User | 3 |
| ChatGPT-User | 1 |
OAI-AdsBot is neither a training nor an answer crawler: OpenAI says it reviews landing pages for ChatGPT Ads, so blocking it matters only to a site that advertises there.
At the edge: what an AI agent's request gets back
A firewall or bot-protection service answers a request before the site's robots.txt is ever consulted, so a site can publish an open policy and still turn AI agents away. Each readable site was sent a browser request and a request in the name of each of 5 AI agents on the same pass.
| AI agent | Sites refusing it | Share of 2,207 |
|---|---|---|
| GPTBot | 111 | 5% |
| ClaudeBot | 109 | 5% |
| PerplexityBot | 68 | 3% |
| ChatGPT-User | 59 | 3% |
| OAI-SearchBot | 35 | 2% |
2,016 sites gave every agent the page a browser got. 174 refused some agents, and 17 refused all of them.
The sites the scan could not read
1,073 of the 3,368 sites approached (32%) returned nothing the scanner could read. The Crane Index reports each of them as unmeasured, not failed.
| What happened | Sites |
|---|---|
| Refused the scanner at the edge | 606 |
| Did not respond, or the connection failed | 357 |
| Responded with no readable content (often a page built entirely in the browser) | 108 |
| Responded with something other than a web page | 2 |
Of the 606 refusals, this is the edge network the scanner identified in each response:
| Edge network named in the refusal | Sites | Share |
|---|---|---|
| Cloudflare | 354 | 58% |
| Not identified | 114 | 19% |
| Akamai | 89 | 15% |
| AWS CloudFront or AWS WAF | 35 | 6% |
| DataDome | 7 | 1% |
| Imperva (Incapsula) | 4 | 1% |
| Kasada | 2 | under 1% |
| Fastly | 1 | under 1% |
A refusal of the scanner is not proof that AI engines are refused too, for the network-address reason above. It does show how often a site's front door is set to turn away a reader it does not recognise, which is worth checking against what the business intends.