A structured benchmark of 808 publicly identifiable US business websites (commercial operating entities) across 10 industry groups, measuring which AI search crawlers can — and cannot — access them.
Of 619 domains whose crawler policy could be directly observed, 261 — 42.2% — block at least one crawler used by AI search systems to discover and cite content.
We separate observable AI-crawler policy from infrastructure non-response. A block declared in robots.txt is a policy decision; a 403, timeout or unscannable response is an access outcome, not evidence of crawler policy. These are reported separately below.
Of the blocked sites, 89.7% are broad access restrictions catching AI crawlers incidentally rather than AI-specific decisions.
Policy-layer figures are based on 619 domains whose robots.txt was successfully retrieved and parsed. A further 189 domains returned no observable policy (146 access-denied, 43 unscannable) and are reported separately in the Infrastructure layer section. They are not counted as open or blocked.
The central finding: Most blocked businesses are not actively choosing to exclude AI search. They have broad access restrictions set years ago that are now inadvertently catching AI crawlers. This is a configuration problem, not a strategic decision.
We separate observable AI-crawler policy from infrastructure non-response. A block in robots.txt is a policy decision. A 403, timeout or unscannable response is an access outcome, not evidence of crawler policy. These 189 domains are reported here and excluded from every policy-layer figure.
Why this matters: a domain that denies the crawler at the infrastructure layer has not expressed an AI-crawler policy — it has prevented one from being read. Treating such a response as “open” would overstate access; treating it as “blocked” would overstate restriction. Reporting it separately keeps the policy-layer figures based only on directly observed robots.txt behaviour.
The US signature: active edge denial. The United States stands apart from the other markets in the series. Its non-response is dominated by access denial (146 domains, 18.1%, the highest measured) rather than passive non-response, and that denial carries a managed-WAF signature: of the access-denied domains, Cloudflare (41) and Akamai (26) account for the largest identifiable share. This is the fingerprint of deliberate bot mitigation: enterprise edge platforms returning 403 to automated requests, not infrastructure that simply failed to respond. By contrast, the other markets show the inverse pattern: more passive timeouts, fewer active denials, and no managed-CDN concentration. How access is restricted, not only how much, differs by market.
Of the 261 sites blocking AI retrieval crawlers, the source of the block was classified into three categories.
The infrastructure-imposed subset is the most commercially significant finding: these site owners may be blocking AI search discovery without ever having made that decision. The indeterminate category — 80 Cloudflare-hosted sites — most likely represents explicit blocks, but the configuration path cannot be confirmed by automated means alone.
Block rates vary across the 10 sectors. Retail & Ecommerce and Hospitality & Tourism are highest; Building & Trades lowest. Rates are computed on policy-observed domains per sector (readable robots.txt only).
Retail & Ecommerce at 56.2% and Hospitality & Tourism at 53.8% are the highest-blocking US sectors. Both are consumer-facing industries where content protection and bot-mitigation are common. Technology & SaaS at 44.9% remains a counterintuitive result: the sector most aware of AI is among the more likely to be invisible to it. Building & Trades at 23.2% is the most open — consistent across every market in the series.
Group A (retrieval/citation crawlers) drives the headline finding. Group B (training crawlers) is reported separately, because blocking training crawlers is often a deliberate and legitimate content-protection decision.
The Googlebot parity finding holds in the US. Googlebot is blocked at 37.8%, right alongside the AI retrieval crawlers (GPTBot 39.6%, ClaudeBot 39.3%). Most US AI-crawler blocks are broad restrictions, not targeted AI decisions. The 14 retrieval crawlers cluster tightly (37.8%–39.6%), indicating that where AI is blocked, it is typically blocked uniformly across operators rather than selectively.
Block rates by content management system among policy-observed domains. WordPress and Drupal both have substantial bases in the US sample; the others rest on small samples and should be read with caution.
Most sites return no identifiable CMS signature, so platform-level rates are based on the minority that do. WordPress at 36.9% (n=84) tracks close to the overall sample average, the broadest cross-section of US business websites. Drupal at 14.8% (n=54) is markedly lower, and with 54 sites this is a reliable signal: Drupal's heavy enterprise and government skew means robots.txt is more deliberately and conservatively managed, producing fewer broad AI blocks. Webflow (n=4), Wix, Joomla and Shopify have too few domains in this sample to report a meaningful rate.
Roles. This study is published by the Periodic Table of Digital Authority (PTODA), the methodology owner. It was conducted using the PTODA C01 Crawler v1.2, a deterministic robots.txt reference instrument, under PTODA C01 Crawler Methodology v1.2. AUTHORITY44 provided technical infrastructure and execution support as commercial operator. Douglas Lord is the founder of both PTODA and AUTHORITY44; this relationship is disclosed in full. The sample was constructed from named public directories with no reference to commercial relationships. The methodology is fully documented and reproducible. This study publishes aggregate, anonymised findings only. No named individual site results are published.
Attribution chain: Douglas Lord (researcher, author) · Periodic Table of Digital Authority (publisher & methodology owner) · PTODA C01 Crawler v1.2 (research instrument) · AUTHORITY44™ (commercial operator) · Digital Dominator Pty Ltd ABN 28 616 931 116 (operating entity).
Intellectual property notice: This study, its methodology, findings, data, and all associated content are the original work of Douglas Lord and the property of Digital Dominator Pty Ltd (ABN 28 616 931 116). The Periodic Table of Digital Authority™ is a coined framework and trade mark pending (TM 2644497). AUTHORITY44™ is a trade mark pending (TM 2643932). All rights reserved.
You may cite findings from this study with appropriate attribution identifying the author (Douglas Lord), the publisher (Periodic Table of Digital Authority — periodictableofdigitalauthority.com), and the research instrument (PTODA C01 Crawler v1.2). You may not reproduce this study in full, present these findings as your own research, or use the framework name or trade marks without prior written consent. Use of this research is subject to the Terms of Use.