Agent Readiness: 4% of Top Sites Declare Anything

Most advice about making a site ready for AI agents is written as though the field were competitive. Cloudflare measured it on 17 April 2026 across the 200,000 most-visited domains, and the results say the opposite: almost nobody is doing any of it, including on the cheap parts. That makes the audit worth running, but for different reasons than the framing usually implies.

The four dimensions being scored

Cloudflare’s agent readiness score assesses Discoverability, Content, Bot Access Control and Capabilities. Discoverability asks whether an agent can find out what exists on the site and where — robots.txt, sitemaps, and the HTTP-level pointers that let a client locate structured information without crawling for it. Content asks whether the substance of a page is retrievable in a form a machine can use. Bot Access Control asks whether the site expresses what automated clients may do. Capabilities asks whether the site exposes anything an agent can call rather than read.

The four are not equally expensive, and that turns out to be the finding.

What the numbers say about the field

The spread across the dataset is stark:

• 78% of sites have a robots.txt file at all.
• 4% have declared their AI usage preferences in robots.txt.
• 3.9% pass the Markdown content negotiation check.
• Fewer than 15 sites in the entire dataset publish an MCP Server Card or an API Catalog.

Fifteen sites out of two hundred thousand is not a low adoption rate. It is an absence. Any recommendation that treats publishing a server card as table stakes is describing a practice that effectively does not exist, and the gap between that and the volume of writing on the subject is itself worth noticing.

The 4% figure is the one that should give pause to anyone who has read a great deal about robots.txt AI directives this year. Ninety-six per cent of the most-visited sites on the web have expressed no AI preference at all, which is the practical context for every argument about which directive syntax to adopt.

The cheapest wins are all in Discoverability

Twenty-two per cent of the most-visited domains have no robots.txt. That is a file, not a project, and its absence costs more than the AI question — it removes the only conventional mechanism for saying anything at all to an automated client. Sitemaps are in the same category of effort, and the details that actually matter at volume are covered in sitemap best practices.

The honest case for doing the Discoverability work is not that agents are waiting on it. It is that these are the same artefacts search crawlers have used for twenty years, they are already cheap, and a site missing them is missing them for conventional reasons too. Anything justified only by hypothetical agent traffic is a worse investment than something justified twice.

Markdown content negotiation, and the uncertainty in it

Markdown content negotiation means serving a Markdown representation of a page when a client asks for one through the Accept header, rather than making it parse HTML. It passes on 3.9% of sites. The mechanism is sound and the implementation cost on a CMS is moderate — a content type, a template and a route.

What cannot be stated is who consumes it. No published figure was found for how often clients request a Markdown representation, or what they do differently when they get one. Treating that as a settled win would repeat the error catalogued in what llms.txt actually does, where a plausible mechanism was widely adopted and then measured to be almost entirely unfetched. The difference here is that content negotiation is a standard HTTP behaviour rather than a new convention, so the downside is smaller — but plausible is not the same as demonstrated, and saying so is the difference between advice and enthusiasm.

Which dimension to ignore for now

Capabilities, for almost everyone. Publishing an MCP Server Card advertises callable tools, which means building and maintaining tools, authenticating them, and accepting that the MCP specification itself describes tools as arbitrary code execution paths that must be treated with caution. For a brochure site, a blog or a services business, that is a security surface added for a demand signal that fifteen domains in the world have judged worth answering.

The useful sequencing runs the other way round from most checklists: get the conventional discoverability artefacts correct, make a deliberate decision on bot access control rather than inheriting a default, treat Markdown negotiation as optional with an honest uncertainty attached, and leave capabilities alone unless you are a platform. That order also happens to match the priorities in an AI search visibility strategy, which is a reasonable consistency check on any framework that claims otherwise.