A structured benchmark of 500 publicly identifiable Australian business websites across 10 industry groups, measuring which AI search crawlers can — and cannot — access them.
Across 440 scannable sites, 169 — 38.4% — block at least one crawler used by AI search systems to discover and cite content.
Of those, 91.7% are broad access restrictions catching AI crawlers incidentally. Most businesses likely do not know this is happening.
All figures based on 440 scannable domains. 60 excluded as unscannable — 23 confirmed bot-protection on major enterprise sites, 37 inaccessible.
The central finding: Most blocked businesses are not actively choosing to exclude AI search. They have broad access restrictions set years ago that are now inadvertently catching AI crawlers. This is a configuration problem, not a strategic decision.
Of the 169 sites blocking AI retrieval crawlers, the source of the block was classified into three categories.
The infrastructure-imposed subset is the most commercially significant finding: these site owners may be blocking AI search discovery without ever having made that decision. The indeterminate category — 56 Cloudflare-hosted sites — most likely represents explicit blocks, but the configuration path cannot be confirmed by automated means alone.
Block rates vary significantly across the 10 sectors. Accounting & Finance and Technology & SaaS are highest; Professional Services and Legal are lowest.
Accounting & Finance at 52.6% is the highest-blocking sector — notable given AI search is reshaping how Australians find financial advice and compare products. Technology & SaaS at 45.8% is the study's most counterintuitive finding: the sector most aware of AI is among the most likely to be invisible to it.
Group A (retrieval/citation crawlers) drives the headline finding. Group B (training crawlers) is reported separately — blocking training crawlers is often a deliberate and legitimate content-protection decision.
The Googlebot parity finding is the most important number in the dataset. Googlebot is blocked at 35.2% — nearly identical to the AI retrieval crawlers. This confirms that the vast majority of AI crawler blocks are broad restrictions, not targeted AI decisions. Businesses blocking AI crawlers are mostly also blocking Google.
Block rates differ significantly by content management system — reflecting both platform defaults and operator technical sophistication.
Drupal at 6.2% is the standout. Drupal is predominantly enterprise and government — organisations with active IT governance making deliberate robots.txt decisions. Webflow at 0% reflects a newer generation of sites by operators who are more AI-search aware. WordPress at 38.6% tracks the overall sample average — the broadest cross-section of Australian business websites.
Douglas Lord created the Periodic Table of Digital Authority methodology and founded AUTHORITY44, the platform used to conduct this study. The sample was constructed from named public directories with no reference to commercial relationships. The methodology is fully documented and reproducible. This study publishes aggregate, anonymised findings only. No named individual site results are published.
Attribution chain: Douglas Lord (researcher, author) · Periodic Table of Digital Authority (methodology) · AUTHORITY44™ (instrument) · Digital Dominator Pty Ltd ABN 28 616 931 116 (operating entity).
Intellectual property notice: This study, its methodology, findings, data, and all associated content are the original work of Douglas Lord and the property of Digital Dominator Pty Ltd (ABN 28 616 931 116). The Periodic Table of Digital Authority™ is a coined framework and trade mark pending (TM 2644497). AUTHORITY44™ is a trade mark pending (TM 2643932). All rights reserved.
You may cite findings from this study with appropriate attribution identifying the author (Douglas Lord), the publisher (Periodic Table of Digital Authority — periodictableofdigitalauthority.com), and the measurement instrument (AUTHORITY44). You may not reproduce this study in full, present these findings as your own research, or use the framework name or trade marks without prior written consent. Use of this research is subject to the Terms of Use.