Crawl Budget in the Age of AI Bots: What’s Actually Crawling Your Site

Crawl budget used to be a concern mostly for enormous e-commerce sites with millions of URLs. In 2026 it’s relevant to almost everyone, for a reason that has nothing to do with site size: your pages aren’t just being crawled by Googlebot anymore. They’re being crawled by a growing list of AI bots, each with its own behavior, its own respect (or disregard) for your preferences, and its own load on your server.

Who’s actually crawling your site now

Beyond Googlebot and Bingbot, a real site’s server logs in 2026 typically show traffic from crawlers tied to OpenAI (GPTBot), Anthropic (ClaudeBot), Perplexity, Common Crawl (which many AI labs train on), and a growing number of smaller AI startups building their own retrieval systems. Some identify themselves honestly in their user-agent string and respect robots.txt. Some don’t.

This has two separate consequences worth treating separately: the technical load these bots add, and the strategic question of whether you want them there at all.

The technical problem: unmanaged bot traffic is real load

Every AI crawler hitting your site consumes server resources the same way a human visitor or a search bot does — and unlike Googlebot, most AI crawlers have no established, predictable crawl-rate throttling tuned to your site’s capacity. On a modest WordPress host, an aggressive AI crawler can measurably slow down real visitors, and in worse cases contribute to hosting resource limits being hit.

Practical steps that help regardless of your strategic stance on AI crawling:

• Check your server logs periodically for unfamiliar user-agents generating heavy request volume.
• Use caching aggressively, so a crawler hitting the same page repeatedly hits a cache, not your database.
• Set explicit crawl-delay or rate-limiting rules at the server level for bots that support them, separate from your indexing preferences.

The strategic problem: do you want to be trained on, or cited?

This is the decision most site owners haven’t consciously made. robots.txt lets you address different bots differently — you can allow Googlebot (for search indexing and AI Overviews, which draw from the same index) while blocking a specific AI training crawler, or vice versa. A basic pattern:

User-agent: GPTBot
Disallow: /

User-agent: Googlebot
Allow: /

There’s no universally correct answer here — it depends on your goals. Blocking AI training crawlers protects content from being used to train models with no attribution or traffic back to you. Allowing them (or allowing retrieval-specific bots that fetch content live to answer a query, which is a different behavior than bulk training scraping) can support being cited in AI-generated answers, which is the traffic AEO strategies are trying to earn. Increasingly these are separate user-agents from separate bots, so treat “AI bots” as a category you can and should differentiate, not a single on/off switch.

What I actually check on a site

1. robots.txt correctness — not just present, but intentionally configured per bot rather than left at defaults nobody reviewed.
2. Server response times under bot load — a quick log review during a traffic spike tells you fast whether crawling is a real performance problem or a non-issue.
3. XML sitemap health — a clean, current sitemap remains the single best way to direct crawl budget, human or AI, toward the pages that actually matter.
4. Orphaned and low-value pages — thin tag archives, empty category pages, and duplicate content waste crawl budget that could go toward pages worth being indexed and cited.

None of this requires exotic tooling. It requires treating “who’s crawling my site and why” as a question worth answering deliberately, instead of an invisible default you never looked at.