Cloudflare Changed Its Bot Defaults on 15 September

Most sites behind Cloudflare inherited a change to how AI traffic is treated on 15 September 2026, and inheriting it is the point — nobody had to opt in. The change rests on a three-way classification Cloudflare introduced for bots that had previously been lumped together as “AI crawlers”, and the categories are doing real work, because two of the three are now blocked by default in a common configuration.

The three categories, and why the split matters

Cloudflare’s definitions are short enough to quote in full. Search is “any behavior that collects or indexes your content, so it can answer questions about it later”. Agent is “automated behavior that is acting, usually in real time, on a person’s behalf, to get something done right now”. Training is “a crawler taking your content to train or fine-tune a model”.

The distinction that older blanket AI-bot rules missed is between Search and Agent. A training crawler takes content and gives nothing back on that visit. An agent visit has a person waiting at the other end of it, often one who is trying to buy something or book something. Blocking both with one switch treats a customer-adjacent request as though it were a scrape, which is the error most robots.txt files written in 2024 and 2025 are still making.

What changed on 15 September

The scope is narrower than the headline suggests, and it is worth reading precisely. The new defaults apply to “all new domains onboarding to Cloudflare”. For those domains, Training and Agent bots are blocked by default on the pages that display ads, while Search is allowed by default. Cloudflare’s own reasoning for the ad-page carve-out is stated directly: “An ad is a signal that a website owner meant for a person to land there and see it — something monetizable that fuels the business.”

The controls are not enterprise-only. The announcement is explicit that customers “can more finely tune how they manage AI bot traffic — including customers on our Free tier”. Existing domains are not silently re-defaulted by this change, but any new property onboarded since 15 September starts from the new position, which matters if you are launching sites or migrating a portfolio.

The trap sitting inside multi-purpose crawlers

This is the part that will catch people. Cloudflare notes that “many popular crawlers on the web fall into one of the classifications above; some fall into multiple”, and that bots with several purposes “should be tracked with all purposes, not just one of them”. The consequence is stated flatly: multi-purpose crawlers, “specifically those that combine Search with Training”, will be allowed or blocked “according to all of their behaviors”.

Read that as: blocking Training can block a crawler you wanted for Search, if the same crawler does both. That is not a bug in the classification, it is an honest reflection of the fact that some operators do not separate the two. It does mean that a decision framed as “we are not letting anyone train on us” can quietly cost visibility in a surface you cared about, and you will find out from a traffic chart rather than from a log line.

Preferences, blocks, and the difference

Cloudflare operates two mechanisms that are easy to conflate. One is an actual block at the edge. The other is Content Signals in a managed robots.txt, where the post is clear that “the values of content use signal a website owner’s preference, rather than issuing blocks directly”. The managed file now carries an additional use= signal alongside the older ones, so a robots.txt that previously read Content-Signal: search=yes,ai-train=no may now read Content-Signal: search=yes,ai-train=no,use=reference.

A preference is a request that a well-behaved operator may honour. A block is enforcement. Conflating them produces the worst version of this decision: a site that believes it is protected, is not, and has also lost the traffic from the operators that do comply.

The audit worth running this week

Four checks, none of which take long. First, open your Cloudflare dashboard and record which of Search, Agent and Training are currently allowed, rather than assuming. Second, check whether managed robots.txt is enabled and read what it is actually serving at /robots.txt, because a managed file can differ from the one in your repository. Third, if you run ads, note that the ad-page distinction now exists and decide whether you agree with it for your own site. Fourth, cross-reference against your server logs to see which of these bots were reaching you in the first place — crawl budget in the age of AI bots covers what that log analysis usually turns up.

The underlying decision has not changed, only the default you are deciding against. A decision framework for blocking AI crawlers still applies, and so does the commercial reasoning in the AI content licensing deals. The Cloudflare bot management documentation is where the per-plan capabilities are set out, and is worth reading before changing anything, because the tiers differ in what they let you express.