robots.txt Allows the AI Bot, but the Firewall Says 403

A site can publish a robots.txt that explicitly allows an AI search crawler and still never be fetched by it. The file is a statement of permission. Whether the request reaches the server at all is decided earlier, by whatever sits in front of it: the host’s firewall, a CDN rule, a security plugin or a bot-management setting. When that layer answers with a 403, the crawler receives a refusal that robots.txt never gets a chance to override.

Two layers that answer different questions

Robots.txt answers whether a well-behaved crawler is permitted to fetch a URL. A firewall answers whether the connection is accepted. OpenAI’s page on its crawlers and user agents describes the permission side clearly. OAI-SearchBot surfaces websites in ChatGPT search features, GPTBot crawls content that may be used to train OpenAI’s models, and ChatGPT-User fetches pages when users ask ChatGPT a question. Each bot’s robots.txt setting is independent of the others, a search change can take about 24 hours to take effect, and sites opted out of OAI-SearchBot will not appear in ChatGPT search answers but can still show up as navigational links.

The page also says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply to them. A site that wants to control that agent has to do it at the network layer, which is exactly where an overzealous rule can also catch the search bot.

What a 403 on robots.txt itself does

The fetch of the robots.txt file is also a request, and it can be refused. Google’s robots.txt documentation, last updated 2026-08-31 UTC, says Google’s crawlers treat all 4xx errors except 429 as if a valid robots.txt file did not exist, which means no crawl restrictions apply. A firewall that returns 403 for the robots.txt URL therefore does not block Google, it removes the rules. The same page warns against using 401 or 403 to limit crawl rate, and says that when an existing robots.txt cannot be fetched because of a server error, Google stops crawling the site for 12 hours while it retries and uses its last good copy for up to 30 days. This describes Google’s behaviour only. Other crawlers are not documented to follow the same rules.

Testing each layer

Four checks separate the two problems, in order of effort:

1. Request /robots.txt and confirm a 200 with the intended rules, from outside your own network.
2. Search the CDN or firewall log for requests whose user agent contains OAI-SearchBot, GPTBot or ChatGPT-User, and read the status codes returned. This is the only direct evidence of what the crawler experienced.
3. Run curl -sI -A "OAI-SearchBot" https://example.com/ against a normal page. A 403 where a browser gets a 200 points to a user-agent rule.
4. Check what each rule is based on. Many bot rules are keyed on verified IP ranges as well as the user agent string.

The curl test has a limit that is easy to forget: it sends a claimed user agent from your own address, which a firewall that verifies identity will treat as an impostor. A pass from curl does not prove the real crawler gets through, and a block from curl may be correct behaviour. The method for verifying crawler identity properly is in the post on fake Googlebot and verification that works, and the same logic applies to any vendor that publishes verification data.

Deciding what you want before you test

The test is only meaningful against a stated policy. A site that wants citations in ChatGPT search needs OAI-SearchBot to be allowed at both layers, and may still block GPTBot, because the settings are independent. The reasoning for that split is in the post on whether to block AI crawlers. Write the policy as a table with one row per bot and two columns, one for the robots.txt rule and one for the network rule, and the contradictions are visible immediately.

The costs of getting it wrong are asymmetric. A mistaken allow is visible in the logs and can be reversed in a day. A mistaken network block produces no error on the site, only an absence, and absence is the hardest result in this field to notice. A monthly look at the status codes returned to the search crawlers is a short job that catches it.