Newshot

AI Access Index: how 684 of the top thousand hostnames treat AI crawlers

Original text(via HN AI)

What robots.txt tells AI crawlers

题图
题图

Each site's policy is read the way its own report reads it: from what its robots.txt says to the 40 crawlers that fetch pages for AI search and answers, and the 9 that gather training data.

  • Open to AI79.7% (545)AI search: yes · AI training: yes
  • AI search, no AI training8.8% (60)AI search: yes · AI training: no
  • Selective3.8% (26)Some AI companies in, others kept out
  • Closed to AI7.7% (53)AI search: no · AI training: no

Shares of the 684 sites that answered. Open includes 119 sites with no robots.txt at all, which allows everything. Closed includes 13 hosts that close themselves to every crawler, search engines included: mostly API and CDN hostnames rather than a decision about AI.

Does the server do what robots.txt says?

robots.txt is a request, and the server answers first. After the check's own visit, each homepage was asked for again four times, each time with one AI crawler's user-agent string, and the answer was set beside what the site's robots.txt says about that crawler.

CrawlerBlocked in robots.txtRefused by the serverRefused while robots.txt allows itServed while robots.txt blocks it
GPTBot OpenAI training15.6%8.8%5.9%75% of 80
OAI-SearchBot ChatGPT search8.2%3.8%2.3%76.1% of 46
ClaudeBot Anthropic training15.5%8.7%5.5%74.7% of 75
PerplexityBot Perplexity search12.3%8.5%6.3%71.7% of 60

Blocked in robots.txt: share of the 684 sites that answered. The server columns: share of the sites whose homepage opened for the check's own request (about 525). Served while blocked: share of those that block that crawler. A refusal is a 401, 403, 429 or 503.

What changed since last month

This is the first edition. From the December 2026 edition on, this section names every site that changed its AI policy, started or stopped blocking one of the four crawlers, or whose server started or stopped refusing a crawler its robots.txt allows.

Look up a site

Every site in the index, as it answered on November 1, 2026. A site that is not in the sample can be checked the same way, free.

The most-blocked AI crawlers

CrawlerCompanyWhat it doesBlocked by
BytespiderBytedanceTraining only17.5% ( 120 )
CCBotCommon CrawlTraining only17.3% ( 118 )
GPTBotOpenAITraining only15.6% ( 107 )
ClaudeBotAnthropicTraining only15.5% ( 106 )
DiffbotDiffbotAI answers14% ( 96 )
meta-externalagentMeta AIAI answers14% ( 96 )
Google-ExtendedGoogle AIAI answers13.5% ( 92 )
cohere-aiCohereAI answers13.2% ( 90 )
Applebot-ExtendedApple IntelligenceTraining only12.9% ( 88 )
PetalBotHuaweiAI search index12.9% ( 88 )
AmazonbotAmazonAI answers12.3% ( 84 )
PerplexityBotPerplexityAI search index12.3% ( 84 )

Share of the 684 sites that answered whose robots.txt disallows / for the crawler, by name or through a group it falls under. Every crawler the check knows is in summary.json; what each one is, in the crawler reference.

The files and signals written for AI

SignalWhat countsSites
llms.txta real one: a text file that starts with a Markdown heading15.8% ( 108 )
Content-Signala line in robots.txt saying what the content may be used for2.5% ( 17 )
TDM reservationthe EU text-and-data-mining opt-out, as a header or /.well-known/tdmrep.json0.9% ( 6 )
A markdown copy for agentsthe homepage answers Accept: text/markdown with markdown3.7% ( 25 )
agents.jsonat /.well-known/agents.json0.3% ( 2 )
A noai directivein the homepage's meta robots or X-Robots-Tag0% ( 0 )

Shares of the 684 sites that answered.

Edition by edition

EditionAnsweredOpenAI search, no trainingSelectiveClosedRefuse an allowed crawlerServe a blocked one
November 2026684 of 100079.7 %8.8 %3.8 %7.7 %11.3 %63.7 %

Method

  • The same sites every month. The top 1,000 hostnames of Tranco list 8PXYV, the research-standard ranking, downloaded on October 11, 2026 and frozen, so a change in the index is a change in the sites, not in the list. Some are infrastructure hostnames with no website behind them; they count as not answering.
  • The same day and the same rules. On the 1st of each month, from a GitHub Actions cloud runner (this edition ran 13:27 to 13:37 UTC): each site once, six at a time, nothing retried. It is the check this site runs for anyone, scoring rules v3: robots.txt, llms.txt, the homepage and its markdown copy, agents.json and the TDM file as CrawlReadinessBot, then the homepage as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, keeping only the status. A site's own report says what the index says about it.
  • The four policies. Read from robots.txt's verdicts for the 40 search and answer crawlers and the 9 training crawlers. Open to AI: at most a fifth of the search crawlers blocked and training left mostly open. AI search, no AI training: search open, training blocked (or every training crawler the file names blocked). Closed to AI: every major AI company kept out. Selective: anything else, such as one company left open while the rest are blocked.
  • Refused and tested. A refusal is a 401, 403, 429 or 503. A crawler's refusal counts only when the homepage opened for the check's own request: when the server refused us too, it could be a wall against every bot. A 2xx to a crawler robots.txt blocks counts either way, so the share of sites that serve a blocked crawler includes some whose homepage did not open for us; the second figure on that tile, and the per-crawler table, leave them out. A site counts as tested when at least one crawler's answer can be read against its robots.txt. Each share names what it is a share of.
  • Limits. One pass, homepages only. The test sends a crawler's user-agent string, which is what firewall rules match; it does not come from the vendors' IP addresses, so a site that verifies crawlers by address could refuse us and admit the real crawler. Firewalls answer by network and by hour, and a 429 can be a rate limit, so the stricter figure leaves 429-only refusals out. The October 2026 study measured the same sites from a different network with a browser as its baseline, so its numbers are not part of this series.

Download the data

Quote the numbers anywhere, the press included, with a credit. The files are licensed CC BY-NC 4.0: free to share and adapt for non-commercial use, with credit. Commercial use of the files, such as in a product or a dataset you sell, needs our permission: ask at [email protected]. Cite as “Crawl Readiness, AI Access Index, November 2026, www.crawlreadiness.com/ai-access-index/2026-11.”

  • results.csv: one row per site in the sample, with its policy, the server's answer to each of the four crawlers and every signal.
  • changes.csv: every change since the edition before, site by site.
  • summary.json: every number on this page, with what each is a share of.

If you run a site in the sample and want its row out of the published files, write to [email protected] from an address at that domain.

Where does your site stand?

The same check, free, with the fixes: run it on your site. Sites people have checked with the free tool are listed, unranked, in Checked sites.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by HN AI
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗