AI Crawler Access: Why 22% of Top Sites Block GPTBot and PerplexityBot Without Knowing

The Block Nobody Remembers Installing

A scan of the Tranco top domains found that 22.1% block at least one of five major AI crawlers, with GPTBot blocked by 17.8% and ClaudeBot by 17.0% (aioscan.com study, August 2026). The overwhelming majority of those site owners never wrote a single line of that rule. This is the central problem with ai crawler access: it is a binary state that decays silently over time, and the decay is rarely deliberate.

Two Failure Modes, One Symptom

Deliberate blocking is a publisher protecting content from training. Accidental blocking is a marketing site that wants citations but lost access years ago through staff turnover, template inheritance, or a platform default nobody reviewed. The two look identical in a robots.txt file and produce the same outcome: exclusion from AI answers. The distinction between them is the subject of this analysis, and it matters because the SEO impact of AI crawlers depends entirely on which one a site is experiencing.

Alef operates as an AI visibility engine that audits crawl and indexability alongside answer readiness, which places it in the gap between what a site intends and what its robots.txt and edge layer actually permit. Accidental blocking is a diagnosable technical defect with a repeatable verification routine, not a philosophical debate about AI training.

What Is Happening: Access Is Being Lost Faster Than It Is Being Granted

The direction of travel is unambiguous. Across roughly 140 million analyzed sites, GPTBot is disallowed by 5.89% of them, and ClaudeBot block rates rose 32.67% year over year (Ahrefs). The block rate is rising fastest among the crawlers that feed the answer engines buyers actually use.

5.89% of roughly 140 million analyzed sites disallow GPTBot, and ClaudeBot blocking grew 32.67% year over year. The web is closing its doors to AI crawlers precisely as answer engines become a real acquisition channel.

That web-wide figure understates the problem at the head of the distribution. Only about 2.5% of live validated domains block at least one AI crawler, but 22.1% of top-ranked domains do. Blocking concentrates exactly where citation competition is fiercest.

The asymmetry explains why blocking feels rational to the people who choose it deliberately. Cloudflare's 2025 data puts Anthropic's crawl-to-refer ratio between 25,000:1 and 100,000:1, OpenAI peaking near 3,700:1, and Perplexity under 400:1, while Google sits near 3:1 to 30:1 (Cloudflare). On raw bandwidth economics, the trade looks lopsided.

Scale is what changed the defaults. AI bots excluding Googlebot accounted for about 4.2% of HTML request traffic in Cloudflare's 2025 year in review, versus 4.5% for Googlebot alone (Cloudflare Radar). AI crawling is now a first-order share of server load, which is why edge toggles moved.

Here is the paradox. The sites with the most to gain from AI citations are the ones most likely to have inherited a block, because they are the sites most likely to have migrated platforms and adopted a CDN. A broader set of AI search statistics shows how quickly that exposure compounds.

Why It Happens: Eight Ways AI Crawler Access Disappears

The striking part of accidental blocking is how little intent it requires. A single line in a text file, a toggle flipped during a migration, a security rule tuned against scrapers — each is enough to remove a site from AI answer eligibility, and none of them announces itself. What follows is a taxonomy of the eight mechanisms that account for most unintended AI crawler access loss, ordered roughly from the most common to the most invisible.

1. Inherited wildcard groups that predate the crawlers they block

A legacy User-agent: * group with Disallow: / is the single most consequential line in many robots.txt files, and it is frequently a fossil. It was written for a staging environment, copied into a production template during a replatform, or left behind by a migration where nobody owned the file afterward.

The reason this blocks GPTBot and PerplexityBot is structural. Under RFC 9309, the Robots Exclusion Protocol standard, a crawler evaluates every group whose user-agent token matches its own and merges the rules; the wildcard group applies only when no specific token matches. GPTBot and PerplexityBot did not exist when the wildcard rule was authored, so no named group was ever created for them — which means the wildcard catches them by default. The rule was not written about AI crawlers. It simply applies to them.

The diagnostic tell is a robots.txt where the only substantive group is the wildcard and every named group is either absent or permissive. If the wildcard disallows anything beyond administrative paths, every AI crawler without a named exception inherits that restriction.

2. Conflating training crawlers with search crawlers

OpenAI operates at least three distinct user-agent tokens with different functions: GPTBot for training data collection, OAI-SearchBot for the ChatGPT search index, and ChatGPT-User for fetches initiated directly by a user in a session. They are separate tokens precisely so that site owners can make separate decisions.

Many teams do not. A content-protection policy aimed at training data results in a blanket block on all OpenAI tokens, which removes the site from the search index that generates citations and AI-referred traffic. The distinction matters because the two outcomes are not symmetric: blocking training forfeits a hypothetical future benefit, while blocking the search index forfeits measurable present-day visibility.

Cloudflare's analysis of its own network data frames this as the crawl-to-click gap — the divergence between how aggressively AI crawlers fetch content and how little referral traffic they return — and argues that site owners can stay discoverable in search while disallowing AI training by treating the two use cases as separate policy decisions. The failure mode is treating them as one.

3. Treating PerplexityBot and Perplexity-User as a single agent

Perplexity's crawler architecture mirrors the same split. PerplexityBot builds and refreshes the index and honors robots.txt directives. Perplexity-User performs live fetches when a person asks a question, and it behaves differently — a robots.txt disallow aimed at PerplexityBot does not necessarily stop Perplexity-User from retrieving a page in response to a query.

The operational consequence is confusing log data. A site that blocks PerplexityBot sees Perplexity-originated requests in its server logs and concludes the block failed, when in fact the block succeeded at its stated purpose (removing index eligibility) while live fetches continued through a different token. Teams then escalate the block, tighten firewall rules, and inadvertently suppress the live-fetch path as well — the one path that produces citations.

Distinguishing the two requires reading user-agent strings in raw logs rather than aggregated bot-traffic dashboards, most of which collapse all Perplexity traffic into one bucket.

4. CDN and edge defaults that outlive the feature that created them

Cloudflare shipped a one-click "Block AI Bots" toggle and a managed robots.txt feature that injected directives such as ai-train=no into customer files. Both were deprecated in September 2026 in favor of granular categories covering search, training, and agents separately.

Any site that enabled the original toggle and never revisited it is still blocking at the edge, regardless of what its origin robots.txt now says. This is the defining characteristic of edge-level blocking: the directive lives in a CDN configuration panel, not in a file the SEO team reads, and it overrides or supplements origin rules in ways that are invisible to anyone auditing the site's own text files.

Cloudflare's 2025 Radar Year in Review documents how quickly AI crawler traffic composition shifted across its network, which is precisely why defaults set two years ago no longer reflect the current mix of agents a site needs to admit.

5. WAF and bot-management rules tuned against scrapers

Web application firewalls and managed bot-fight modes score incoming requests on user-agent reputation, ASN heuristics, request rate, and behavioral signals. These systems were tuned to repel scrapers, and legitimate AI crawlers frequently trip the same heuristics: they arrive from cloud ASNs, request pages at volume, and often do not execute JavaScript.

The result is JavaScript challenges, rate-limit throttling, or outright 403 responses served to GPTBot and PerplexityBot. From the crawler's perspective, the site is unreachable. From the site owner's perspective, nothing in the robots.txt file explains it — because the block is not in robots.txt at all. It lives in a security console that the SEO team typically has no login for and no reason to open.

6. Platform and plugin defaults that were never reversed

WordPress ships a "Discourage search engines from indexing this site" setting that writes a site-wide noindex directive and disables the platform's built-in sitemap generation. It is a standard item on launch checklists for staging environments — and a standard item that goes un-reversed after launch.

SEO plugins that replace the virtual robots.txt introduce a parallel risk. When a plugin takes over robots.txt output, directives added manually to the physical file can be silently dropped, and plugin-level toggles can override rules the team believes are in place. The failure pattern is identical to the wildcard case: a decision made for one context persists into another where it is actively harmful.

Sites recovering from this class of defect typically need to re-verify indexation end to end; the guide to diagnosing and fixing indexation issues covers the sequence for confirming that a corrected directive actually propagated.

7. Ownership gaps between teams

Robots.txt is edited by developers. The CDN configuration is owned by platform engineering. The WAF is owned by security. Content strategy and AI visibility measurement sit with marketing.

No single role owns AI crawler management, which means a rule added by one team is structurally invisible to the team measuring AI visibility. Security tightens a bot rule to reduce load; marketing notices that AI-referred traffic has declined but has no mechanism to connect the two events. The gap is organizational before it is technical, and it explains why the same block can persist for multiple quarters across otherwise well-run sites.

8. Reporting blindness in standard SEO dashboards

Google Search Console's Crawl Stats report reflects Googlebot. It does not show GPTBot, PerplexityBot, OAI-SearchBot, or ClaudeBot activity. The same is true of most rank-tracking and site-audit platforms, which model a search-engine world in which Google is the only crawler that matters.

This is the mechanism that makes the other seven durable. The block is not concealed — it is simply outside every report that marketing reviews on a weekly cadence. Ahrefs' analysis of AI bot block rates across roughly 140 million websites found blocking widespread enough to be a default condition rather than an outlier, which suggests a large share of those blocks are inherited rather than chosen. Without a report that surfaces AI bot access alongside crawl and indexability status, the condition persists indefinitely.

Alef's Site Health audit is built around exactly this gap: crawl access, indexability, and answer-readiness blockers appear in one ranked view, so a rule added in a security console surfaces next to the content issues the marketing team already tracks. For teams that want the full sequence, the technical SEO audit sprint framework shows where AI crawler verification fits into a one-day diagnostic pass.

The pattern across all eight mechanisms is the same: each was a reasonable decision in its original context, and none was revisited when the context changed. AI crawler access is lost not through opposition to AI, but through the ordinary accumulation of technical debt in systems nobody assigned an owner.

The Impact: Who Loses What When Access Is Blocked

The damage varies by mechanism, because each one fails differently and leaves a different trace. The table below maps all eight access-loss mechanisms from the previous section, plus the deliberate-blocking counterpoint, against who owns them, where they live, and how long diagnosis takes.

Why the Loss Never Looks Like a Loss

An answer engine can only cite what its retrieval layer can read. A blocked site is not ranked lower — it is absent from the candidate set entirely. That distinction matters: ranking declines appear in reporting, while absence does not. Cloudflare's crawl-to-click analysis shows AI referral traffic remains a small share of total visits, which makes a zero even easier to misread as "AI search does not work here" (Cloudflare Blog).

The effect compounds. Where retrieval draws on training-derived representations, a block sustained through a model refresh cycle can keep a brand out of answers longer than the block itself lasted. Ahrefs found AI bot blocking concentrated among a large share of the web's most-visited domains (Ahrefs), so the competitive field is already uneven.

The evidence asymmetry is the cruelest part: the site sees zero AI-referred traffic and concludes the channel is irrelevant, when the cause is a 200-line text file. Tracking how AI citations and mentions accrue without links only works once access exists — and a site health audit that surfaces crawl and indexability blockers is where that access gets verified.

What It Means for You: A Verification Routine

Access problems are only invisible until someone looks. A five-step routine surfaces most accidental blocks in under an hour, and it belongs in every site migration and platform upgrade checklist.

Step 1 — Read the live file, not the repository

Fetch /robots.txt from production and read what crawlers actually receive. Check for wildcard Disallow rules, named groups for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, and Google-Extended, and confirm the sitemap.xml directive survives. A staging copy proves nothing about the edge.

Step 2 — Test as the bot

Request a key URL using the GPTBot and PerplexityBot user-agent strings and confirm a 200 response rather than a 403 or a JavaScript challenge. A 200 from a browser is meaningless; the crawler never sees the browser experience.

Step 3 — Verify at the edge

Review CDN bot-management settings and WAF rules for AI-category blocks. Confirm whether a legacy one-click AI toggle remains active after the September 2026 Cloudflare control changes, which split search, agent, and training permissions into separate switches.

Step 4 — Confirm in logs

Filter server or CDN access logs for GPTBot, OAI-SearchBot, PerplexityBot, and ClaudeBot over 30 days, then cross-check source IPs against published ranges — OpenAI publishes a machine-readable gptbot.json — to separate genuine crawler hits from spoofed user agents.

Step 5 — Record the decision

Document which crawlers are allowed and why, so the next migration does not silently reinstate a block. Re-run the check after every platform, CDN, or DNS change.

Closing the loop

Logs confirm access, not citation. Alef's Prompt Tracker runs structured buyer prompts through ChatGPT, Gemini, and Perplexity to verify the brand appears in answers once access is restored — the measurement layer that pairs with a complete website health audit and the broader AI visibility workflow.

Takeaway

AI crawler access sits at the foundation of AI visibility, not beside it. A robots.txt directive or WAF rule that no one remembers writing determines whether a page can ever be cited in a generated answer — and the effect surfaces only as silence.

Key takeaways - Blocking concentrates at the top of the distribution: 22.1% of top domains block GPTBot, according to Ahrefs' analysis of AI bot block rates. - Most blocks are inherited from templates, CDN defaults, or legacy rules — not deliberate decisions. - Google Search Console cannot see AI crawlers, so server logs and edge configuration are the only evidence. - A five-step verification routine turns an invisible defect into a fixable ticket.

When an answer engine skips a brand, the cause is often upstream of content quality. Alef's guide to why a brand stays invisible in AI answers traces those causes through crawl access, entity clarity, and citation readiness.

Frequently Asked Questions

Does blocking GPTBot remove my site from ChatGPT answers?

Yes, with a qualification that matters. Blocking GPTBot removes the site from OpenAI's training-derived retrieval pipeline, and depending on which OpenAI token is blocked, it can also remove the site from ChatGPT search results. GPTBot, OAI-SearchBot, and ChatGPT-User are separate user-agent tokens that must be evaluated separately in robots.txt. A wildcard Disallow: / that catches one catches all three, which is why inherited rules are so consequential for earning citations in ChatGPT answers.

How do I check if PerplexityBot is blocked in robots.txt?

Fetch the live robots.txt from the production domain, not a staging copy, and scan for a named User-agent: PerplexityBot group or a wildcard Disallow that would apply to it. Then request a representative page with the PerplexityBot user agent and confirm the response is a 200 rather than a 403. Perplexity-User behaves differently from PerplexityBot, so both tokens need separate verification when assessing how to rank in Perplexity answers.

Will allowing AI crawlers hurt my Google rankings?

No. Googlebot access is governed by its own robots.txt group, and Google-Extended controls AI training use of content without affecting Search indexing or ranking. Allowing GPTBot or PerplexityBot does not change Googlebot's crawl behavior, and Google has documented that AI training preferences are independent of Search inclusion. The two decisions can be made separately.

Why does Google Search Console show no AI crawler data?

Because Crawl Stats reports Googlebot only. GPTBot, PerplexityBot, and ClaudeBot activity never appears there, which is precisely why accidental blocks stay invisible in standard SEO reporting. Third-party crawler activity must be read from server or CDN access logs filtered by user agent. Ahrefs' analysis of AI bot block rates across roughly 140 million sites illustrates how widespread the resulting blind spot is (Ahrefs).

Is it ever correct to block AI crawlers?

Yes. Publishers monetizing content, sites with genuine crawl-cost exposure, and properties serving regulated data have legitimate reasons to restrict AI bot access. The defect is not blocking itself but blocking without a recorded decision — a rule nobody can trace to an owner, a date, or a rationale. Cloudflare's work on accountable mixed-use crawlers shows that discoverability and training restrictions can coexist (Cloudflare).

How often should AI crawler access be re-verified?

After every migration, CDN configuration change, WAF update, or CMS plugin change, plus a quarterly baseline check. Edge defaults and platform settings change without notice, and a single toggle can silently reverse months of intended access.

Make Your Site Citation-Ready

An inherited Disallow rule or a forgotten CDN toggle will not appear in rank tracking, yet it can remove a site from AI answers entirely. The fastest way to convert that invisible block into a ranked, fixable issue list is a site health audit that checks crawl and indexability alongside answer readiness, verifying AI crawler access rather than assuming it. The Alef visibility solutions pair that audit with ongoing AI-referred traffic tracking, so a single next step at alef.ink turns an unexplained absence into a measurable baseline.

Sources

• Ahrefs — The AI Bots That ~140 Million Websites Block the Most
• Cloudflare Blog — The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals
• Cloudflare Blog — The 2025 Cloudflare Radar Year in Review
• Cloudflare Blog — Have it both ways: stay discoverable in search while disallowing AI training
• OpenAI Developers — Overview of OpenAI Crawlers