One robots.txt Line to Opt Out of AI Training

If you want to keep your content out of AI training today, you have to list AI companies one by one in robots.txt, each with its own crawler name, and keep the list up to date as new ones appear. On 30 October 2026, Google announced plans to support a single shared signal instead. The Search Central blog post describes it as a way for websites “to express their AI training preferences across multiple AI providers in a single step”. It is not live yet, and some details may change, but it is worth understanding now.

What Google announced

The new control is a shared “policy token” used inside robots.txt. Website owners will be able to use it to set one broad preference that applies across participating AI providers. The work comes from the AIPREF working group at the Internet Engineering Task Force (IETF), the body that writes many of the internet’s technical standards.

Three points from Google’s post matter most:

• How Google will read it. If your robots.txt uses the shared token to opt out of training, and you have not mentioned Google-Extended specifically, Google plans to treat it as if you had disallowed Google-Extended.
• Existing controls keep working. Company-specific controls such as Google-Extended continue alongside the shared signal, so you can set a broad rule and still make exceptions for individual companies.
• Timing. Google says it aims to support the control, along with other AI providers, in 2027.

What the draft actually proposes

The proposal itself is an Internet-Draft called draft-kelley-aipref-training-preference-signal, version 00, dated 30 October 2026. It does not add any new robots.txt directives. Instead, it uses a special user-agent name. The draft’s own example looks like this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

According to the draft, this means “no AI training on anything, except the /press-kit/ folder”. Because it reuses the familiar User-Agent, Disallow and Allow lines, any existing robots.txt parser can read it.

The example also relies on a rule that already exists. Under the robots.txt standard, RFC 9309, when an Allow and a Disallow rule both match a URL, the most specific rule (the one with the longest matching path) wins. So Allow: /press-kit/ beats Disallow: / for anything inside that folder. If you already use exceptions like this for other crawlers, the new token works the same way.

Two caveats are important. First, the datatracker lists it as an individual draft, not yet a working group document, and the draft itself says it has no formal IETF standing. Second, the draft defines “AI training” narrowly, using the AIPREF vocabulary: using content “to modify the learned parameters of a generative AI model”. It is about training models, not about search or about AI answers that quote your page.

What this does not change

A common mistake is to think an AI opt-out removes your site from AI search features. It does not, and Google keeps these separate:

• Google Search and AI Overviews are controlled by Googlebot rules and by snippet controls such as nosnippet, plus the Search generative AI setting in Search Console. Google’s page on AI features and your website explains this.
• Google-Extended covers AI training and grounding in some of Google’s other systems, such as Gemini Apps. It does not affect ranking in Search. The earlier post on what Google-Extended does and doesn’t do covers the details.
• The new shared token is about training, and Google maps it onto Google-Extended.

Google’s post also adds one more point: sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. So opting out of training through Google-Extended has a cost in Gemini, and each site has to weigh that.

Also remember what robots.txt is: a request, not a lock. Well-behaved crawlers follow it. Crawlers that ignore it need to be blocked at the server or firewall, which is a different job. The earlier post on robots.txt rules versus firewall blocks shows how the two can disagree.

What to do now

Nothing needs to change today, but a little preparation makes 2027 easier:

1. Decide your position. Do you want your content used for AI training? Many small businesses want visibility in AI answers but not training. Others are happy with both. Write the decision down so the robots.txt file reflects a choice, not an accident.
2. Read your current robots.txt. Open yourdomain.com/robots.txt and list which AI crawlers you already mention.
3. Don’t add the new token yet. Google says support is planned for 2027, and the draft is version 00. Adding it now does no harm, but it also does nothing with Google until support starts.
4. Keep company-specific rules. When the shared signal arrives, you will still be able to make exceptions per company, so your existing rules remain useful.
5. Edit robots.txt carefully. On WordPress, Rank Math includes a robots.txt editor, which saves editing files on the server. Whatever tool you use, test the file afterwards in Search Console’s robots.txt report.

The direction is clear even if the details are not final: one shared signal for the broad choice, and company-specific tokens for exceptions. That is simpler for site owners than chasing a new crawler name every few months, and it is worth following as the IETF work moves on. The datatracker page linked above lists each new version of the draft, so it is the easiest place to check progress.