ai crawlers - Rakibuzzaman Siam

Three robots.txt Extensions, None of Them a Standard

October 7, 2026 by rakibuzzaman

A robots.txt file with three competing directive syntaxes layered over one another, none marked as final

Content-Signal, Content-Usage and License are three incompatible robots.txt directives for AI preferences. No RFC exists. What to put in the file today.

OAI-AdsBot Is Crawling You for Ad Placement

October 4, 2026 by rakibuzzaman

Four distinct automated clients requesting the same page for four different purposes

OpenAI documents four crawlers, and one of them exists to judge ad landing pages. What each does, which ignores robots.txt, and how to verify rather than guess.

Google-Extended Does Not Do What You Think

September 28, 2026 by rakibuzzaman

A robots.txt file with a Google-Extended disallow rule beside a list of Google fetcher user agents

Blocking Google-Extended in robots.txt does not remove you from AI Overviews. What the token covers, which fetchers ignore robots.txt, and the control that works.

Cloudflare Changed Its Bot Defaults on 15 September

September 27, 2026 by rakibuzzaman

Three streams of automated traffic arriving at a site edge, two of them stopped and one allowed through

Cloudflare now splits AI bots into Search, Agent and Training. From 15 September 2026, two of those three are blocked by default on ad-monetised pages.