robots.txt vs Meta Robots vs X-Robots-Tag: When to Use Which

One of the most common tickets I see in Rank Math support reads some version of: “I blocked this page in robots.txt, but Google still has it indexed.” It’s not a bug. It’s what robots.txt was always designed to do — and it’s the clearest sign that a lot of site owners are using three different tools for one job without knowing they can cancel each other out.

robots.txt Controls Crawling, Not Indexing

A Disallow rule in robots.txt tells well-behaved crawlers not to fetch a URL. It says nothing about indexing. If other pages — on your site or elsewhere — link to that disallowed URL, Google can still index it based on that external signal, typically showing it in results with no title or description, something like “no information is available for this page.” That’s exactly the scenario behind that recurring ticket: the block worked exactly as designed, and the outcome still surprised the site owner because they expected “don’t crawl” to mean “don’t index.”

Meta Robots Tags Control Indexing — But Only If They’re Seen

A meta robots noindex tag placed in the page’s HTML head is the actual tool for keeping a specific page out of the index. The catch: it only works if a crawler fetches the page and reads that tag. If the same URL is also disallowed in robots.txt, Googlebot never gets far enough to see the noindex directive, so it can’t act on it — the two directives directly contradict each other, and robots.txt wins by making the noindex invisible. Google’s own developer documentation flags this explicitly: if you need a noindex tag to actually take effect, the URL has to remain crawlable.

X-Robots-Tag Handles What Meta Tags Can’t

The X-Robots-Tag carries the same set of directives — noindex, nofollow, noarchive, unavailable_after, and so on — delivered as an HTTP response header instead of an HTML tag. That matters because a meta tag only exists inside an HTML head, and plenty of indexable content isn’t HTML at all: PDFs, images, video files, other downloadable documents. You can’t embed a meta tag inside a PDF, but you can set an X-Robots-Tag header on the server response for that file type, applied once at the server or CMS level rather than per file.

Practical Decision Guide

• Want to stop crawlers from wasting time on low-value paths, like internal search results or filtered URLs, without necessarily worrying about indexing? Use robots.txt.
• Want a specific HTML page kept out of the index while it stays crawlable? Use a meta robots noindex tag, and make sure robots.txt isn’t also blocking it.
• Need to de-index a PDF, image, or other non-HTML file? Use X-Robots-Tag, since there’s no HTML head to put a meta tag in.
• Need the same rule applied across a whole file type or directory at once? X-Robots-Tag set at the server level is usually more practical than tagging files individually.

The single most common technical SEO mistake I see is layering these three tools like redundant safety nets, when they can actually cancel each other out — disallow a URL in robots.txt and its noindex tag becomes invisible to Google, which is the opposite of what most site owners think they’ve configured.

Pick the tool that matches the actual goal — crawl efficiency versus index removal versus non-HTML file handling — rather than assuming they’re interchangeable. And if you’re not sure which directive is actually winning on a given URL, check it directly through the URL Inspection tool in Search Console rather than trusting what your plugin settings say should be happening.