Three robots.txt Extensions, None of Them a Standard

Three groups are currently proposing that publishers add a new line to robots.txt to express what AI systems may do with their content, and the three proposals are mutually incompatible in syntax, vocabulary and legal theory. None of them has been published as an RFC. A publisher being told to “add the AI directive” is being told to pick a side in an unresolved argument, usually by someone who has not said that is what they are asking.

What each proposal adds to the file

The three differ in what they are trying to express, not only in how they spell it:

• Content-Signal, Cloudflare’s. A comma-separated set of named signals on one line, for example Content-Signal: search=yes,ai-train=no,use=reference. It expresses permission per behaviour, and Cloudflare serves it in managed robots.txt files.
• Content-Usage, the IETF’s. Defined in an Internet-Draft with an optional path prefix, as in Content-Usage: train-ai=n or, scoped to a directory, Content-Usage: /ai-ok/ train-ai=y.
• License, RSL’s. A single directive pointing at an external document rather than encoding the preference inline: the ABNF is license-directive = "License" ":" OWS absolute-URI OWS, and the URI resolves to an XML licence.

The third is doing something categorically different from the first two. Content-Signal and Content-Usage encode a yes or no against a named behaviour in the file itself. RSL points at a licence with terms, including payment terms, which is an attempt to make this a commercial instrument rather than a robots convention.

The IETF vocabulary, and the disclaimer on it

The IETF’s aipref working group splits the work in two: a vocabulary of preferences and a mechanism for attaching them to content. The attachment draft, draft-ietf-aipref-attach, reached revision 05 on 19 August 2026 and remains an active Internet-Draft intended for Proposed Standard status. The vocabulary draft reached revision 08 on 14 September 2026 and defines three categories: train-ai, “using an asset to modify the learned parameters of a generative AI model”; ai-use, “using an asset as input to a generative AI model, where the asset is not directly provided by the user”; and search, “using an asset in an application where the primary purpose of the application is to select assets and direct users to the location of those assets”.

The vocabulary draft carries a disclaimer that should end most discussion of adopting it today: “This is a working document. Its contents DO NOT REFLECT CONSENSUS of the Working Group either in whole or part.” Several sections, including the vocabulary definition itself, carry their own no-consensus notes. Both of the group’s chartered milestones were targeted for 31 August 2026 and neither has produced an RFC.

RSL is the one that is trying to be enforceable

RSL 1.0 was published on 10 December 2025 as a Recommendation, describing itself as a machine-readable vocabulary for “usage, licensing, payment, and legal terms”. Its editor list is unusually heavyweight for a proposal at this stage, drawing from the RSL Collective, Condé Nast, Ziff Davis, Schema.org, Yahoo, Automattic, O’Reilly Media and Fastly. The RSL specification states that any crawler or agent covered by an RSL licence “MUST” locate and parse it before accessing the asset.

That MUST is a specification’s MUST, not a court’s. It binds implementations that choose to conform, which is the same standing RFC 9309 gives ordinary robots.txt rules — that document is explicit that “these rules are not a form of access authorization”. The legal weight, if any, comes from the licence the URI points at and from contract or copyright law, not from the directive.

Nobody can tell you which one wins

The adoption question has no good answer right now. No statement of support for any of the three from Google or Bing was found while checking this, and the absence of a search-engine implementation is the gap that matters, because a preference nobody reads is a comment. The Content Signals Policy has Cloudflare’s distribution behind it, which makes it the most widely present of the three by volume of files, but presence in a file and compliance by a crawler are different measurements and only the first has been counted.

What to do in the meantime is narrower than any of the three camps would like. Keep the conventional robots.txt correct, because the mechanism that is standardised and honoured is still User-agent and Disallow — robots.txt versus meta robots versus X-Robots-Tag covers where each one actually takes effect. If your edge provider serves a managed file with Content-Signal in it, read what it is emitting rather than assuming. Add a second directive only if you have a commercial reason that survives the question “what changes if no crawler honours this”, and answer that question before, not after. The economics behind that reason are in publisher licensing economics, and the broader call in whether to block AI crawlers.