OAI-AdsBot Is Crawling You for Ad Placement

Server logs on a site with any volume now show four distinct OpenAI user agents, and the robots.txt files written to deal with them usually treat all four as the same decision. They are not. OpenAI’s own bots documentation lists four crawlers doing four different jobs, and one of them — OAI-AdsBot — exists to evaluate pages submitted as advertisements, which is not a content licensing question at all.

The four bots and what each is for

OpenAI’s crawler documentation names each with a version string and a purpose:

• OAI-SearchBot/1.4 — “used to surface websites in search results in ChatGPT’s search features”. The consequence of blocking it is stated: sites opted out “will not be shown in ChatGPT search answers, though can still appear as navigational links”.
• GPTBot/1.4 — “used to make our generative AI foundation models more useful and safe”. This is the training crawler, and the one most existing blocks were written for.
• OAI-AdsBot/1.0 — “used to validate the safety of web pages submitted as ads on ChatGPT”.
• ChatGPT-User/1.0 — “used for certain user actions in ChatGPT and Custom GPTs”.

Blocking all four with one wildcard rule collapses four separate commercial decisions into one, and the decision it lands on is rarely the one anybody would make deliberately.

Why OAI-AdsBot is a different category

The documentation is specific about the scope: “OAI-AdsBot only visits pages submitted as ads”, and “the data collected by OAI-AdsBot is not used to train generative AI foundation models”. It also does more than check policy compliance — OpenAI states it “may also use content from the landing page to determine when it’s most relevant to show the ad to users”.

Two implications follow. If you do not advertise on ChatGPT, this crawler has no reason to visit you and blocking it changes nothing. If you do, blocking it is an advertising decision that interferes with the evaluation of your own landing pages, taken by whoever last edited robots.txt — usually without the paid media team knowing the rule exists.

The one that may ignore your rules

ChatGPT-User is the exception that makes robots.txt an incomplete control. OpenAI’s own wording: “Because these actions are initiated by a user, robots.txt rules may not apply.” It adds that “ChatGPT-User is not used to determine whether content may appear in Search”, and that it “is not used for crawling the web in an automatic fashion”.

This is not OpenAI misbehaving, it is the same position Google takes on user-triggered fetchers, and it follows from the protocol rather than from policy. RFC 9309, the Robots Exclusion Protocol, has been a Standards Track document since September 2022 and is explicit that these rules “are not a form of access authorization”. Robots.txt has always been a request that cooperating crawlers honour, and a fetch made because a person asked for it sits outside the case the protocol was written for.

The practical consequence is that any plan depending on robots.txt to keep content away from an assistant is depending on voluntary compliance from automated crawling only. The reasoning that should sit underneath that plan is in should you block AI crawlers.

There is no agent-specific user agent

Agentic browsing — a user asking an assistant to go and do something on a site — arrives under ChatGPT-User, because no separate token exists for it. That is a real gap for anyone trying to distinguish a person reading a page through an assistant from a person whose assistant is filling in a form on their behalf. The two have different value to a publisher and different risk to a store, and the logs cannot tell them apart.

Anyone building reporting on assistant traffic should record that limitation in the report rather than in a footnote, because the number will be read as though the distinction had been made. What the logs do show reliably is covered in what’s actually crawling your site, and the publisher-side consequences in what ChatGPT Search means for publishers.

Verify rather than match on a string

A user-agent string is self-reported and trivially forged, and imposters are a meaningful share of the traffic claiming to be any well-known bot. OpenAI publishes IP range files for each crawler — openai.com/searchbot.json, openai.com/adsbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json — and verification against those ranges is the only way to know that a request claiming to be OAI-SearchBot came from OpenAI.

Fetch the files rather than copying addresses out of them into a config, because published ranges change and a hardcoded literal becomes wrong silently. The rule that survives is one that resolves the current range at evaluation time; the rule that breaks is the one written once against a snapshot and never revisited.