Should You Block AI Crawlers? A Practical Decision Framework

“Should I block AI crawlers?” doesn’t have one right answer, and I’ve stopped pretending it does. It depends on which bot, what it’s actually used for, and what you’re trying to protect — and the honest starting point is that the major AI companies now publish distinct, separately controllable crawlers for genuinely different purposes, which means “block all AI bots” is usually a blunter decision than the situation calls for.

The bots, and what each one is actually for

OpenAI’s own documentation lists four separate, independently controllable bots: GPTBot (crawls content for model training), OAI-SearchBot (powers ChatGPT’s search-and-cite feature specifically), ChatGPT-User (live fetches triggered directly by a user’s request within a chat), and OAI-AdsBot (ad-quality validation). Each has its own robots.txt user-agent string, meaning you can allow ChatGPT to cite your page in live search responses while still opting out of your content being used for model training — they’re not a package deal.

Anthropic’s documentation similarly defines three separate bots: ClaudeBot (training), Claude-User (live fetches for a user’s in-conversation request), and Claude-SearchBot (search indexing) — replacing older, now-deprecated user-agents. Perplexity’s official documentation states PerplexityBot respects robots.txt Disallow directives, though the company notes changes can take up to 24 hours to propagate, and that even a disallowed domain may still have its name, headline, or a brief summary surfaced without full content access. Google’s Google-Extended user-agent, introduced in September 2023, lets you opt out of Gemini and AI-training use specifically, entirely separate from normal Googlebot search crawling and indexing — blocking it doesn’t affect your search rankings.

The compliance question nobody has a clean answer to

Here’s where it gets genuinely uncertain: robots.txt is a voluntary standard, and studies on how well AI crawlers actually respect it disagree substantially. One 2025 dataset found roughly 49% of sites block GPTBot; a separate, smaller 1,744-site sample found only about 10% do; another report claimed 79% of top news sites specifically block AI training bots. TollBit’s own tracking found 13.26% of AI bot requests ignored robots.txt entirely in Q2 2025, up sharply from 3.3% in Q4 2024.

The honest state of AI crawler compliance in 2025-2026: it’s real for the major, reputable players, and getting less reliable over time as the ecosystem gets more crowded with bots that don’t have the same reputational incentive to behave.

I’d treat any single percentage you see quoted with real caution — the studies disagree by a factor of five depending on sample and methodology. What’s consistent across all of them is the direction: this is a voluntary system, major labeled bots from OpenAI, Anthropic, and Google generally do respect it, and the ecosystem of less accountable crawlers claiming to be AI-related is growing.

A decision framework, not a blanket rule

Given that different bots serve different purposes, the useful question isn’t “AI bots: yes or no” — it’s a short set of separate questions:

• Do you want to be cited in AI-generated search answers? If yes, you need OAI-SearchBot, Claude-SearchBot, and PerplexityBot able to reach your content — blocking these specifically removes you from the exact AI-search visibility most sites are trying to build.
• Do you have a strong opinion about your content training future models? That’s a separate, legitimate concern from search visibility — GPTBot, ClaudeBot, and Google-Extended can be blocked independently without affecting whether you show up in that same company’s AI search results.
• Is your content behind a paywall or otherwise commercially licensed? If your business model depends on gating content, allowing broad AI crawling (training or otherwise) may directly undercut that model — this is closer to a business decision than a purely technical SEO one.
• Do you have server capacity concerns? With bot traffic volume climbing and a meaningful share of it disregarding crawl-delay directives or robots.txt outright, server load from less-reputable crawlers is a legitimate, separate reason to actively monitor and block specific user-agents, distinct from any content-licensing concern.

A common WordPress-specific mistake worth checking for

Several SEO and security plugins ship with a default “block AI bots” toggle that, when enabled, blocks every recognized AI user-agent indiscriminately — training bots and search-citation bots together, with no distinction. If you’ve ever flipped one of these settings on without reading exactly which user-agents it targets, it’s worth checking your actual robots.txt output directly rather than trusting the plugin setting’s label. I’ve seen sites that enabled a blanket “block AI crawlers” toggle specifically hoping to protect content from training use, without realizing the same toggle was also blocking the search-citation bots they’d have wanted allowed — quietly opting themselves out of AI-search visibility as a side effect of a setting aimed at an entirely different concern.

What I’d actually do

For most content-driven sites genuinely trying to build AI-search visibility, blocking the search-specific bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) works directly against your own stated goal — don’t do that by accident through an overly broad robots.txt rule aimed at “AI” as a single category. Whether to block the training-specific bots (GPTBot, ClaudeBot, Google-Extended) is a legitimate, separate decision that depends on how you feel about your content being used to train models you don’t benefit from directly — reasonable people land differently on this, and it’s worth deciding deliberately rather than defaulting to whatever a plugin ships with. And given how unevenly robots.txt compliance is tracked across the wider bot ecosystem, treating robots.txt as your only line of defense against unwanted scraping — rather than one signal among several, alongside server-level rate limiting for bots that ignore it — probably overestimates what a text file can actually guarantee.