Fake Googlebot: Verification That Actually Works

Every access log with a public IP address contains requests claiming to be Googlebot, and a large share of them are not. The string costs nothing to copy, which makes it the default disguise for scrapers, vulnerability scanners and anything that expects a crawler to be waved through a rate limit. Blocking on the strength of that string, or trusting it, both fail for the same reason.

The user agent string proves nothing

Google’s documentation of its common crawlers, last updated 2026-07-14 UTC, says it directly: the HTTP user agent string can be spoofed, and it points readers at a separate verification procedure rather than asking them to trust the header. That is the whole of the problem in one line. Any rule that allows or denies traffic on a substring match against Googlebot is a rule an attacker can satisfy in one line of configuration, and a rule that a well-behaved crawler can fail as soon as the string changes.

The consequence is not only security. Log-based analysis built on the raw user agent overstates Google’s crawl volume, sometimes considerably, which distorts every conclusion drawn from it. Anyone doing serious log file analysis has to verify before counting, not after.

The two methods Google documents

The page on verifying Googlebot and other Google crawlers, last updated 2026-03-20 UTC, gives two routes, and they suit different scales.

The manual route is a reverse DNS lookup on the accessing IP address from your logs. A genuine Google crawler resolves to a hostname under googlebot.com, google.com or googleusercontent.com, with names in the shape crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. The step people skip is the confirmation: resolve that hostname forward again and check it returns the same IP address you started with. Reverse DNS alone can be claimed by whoever controls the address’s PTR record; the round trip cannot.

The automatic route is matching against Google’s published IP range files, which are served as JSON and updated by Google. There are five of them, split by purpose: common-crawlers.json for Googlebot and its relatives, special-crawlers.json, user-triggered-fetchers.json, user-triggered-fetchers-google.json and user-triggered-agents.json, with a broader goog.json covering other Google services. For anything running at volume — a firewall rule, a bot-management layer, a nightly log job — the JSON files are the practical choice, because a DNS round trip per request is not something you want in a hot path.

What a claimed location is worth

On 14 August 2026 it was reported that Google’s crawlers self-declare a location in California while Google operates data centres worldwide, so the declared origin need not be where the request physically came from. That matters for anyone who has built geography into their bot handling — serving different content, applying different rate limits, or excluding a region from analysis — on the assumption that a stated location is a measured one. It is not. The IP ranges and the DNS round trip are verifiable; a location claim is not, and it should not be load-bearing in any rule.

The eleven crawlers worth being able to name

The common crawlers page currently lists eleven, and knowing which is which changes what you do about them:

• Googlebot, plus Googlebot Image, Googlebot Video and Googlebot News — the Search crawlers proper.
• Google StoreBot — shopping and product experiences.
• Google-InspectionTool — the agent behind Search Console’s testing tools, triggered by you.
• GoogleOther, GoogleOther-Image and GoogleOther-Video — general-purpose fetching for internal use, not Search indexing.
• Google-CloudVertexBot — fetches for Vertex AI customers building their own agents.
• Google-Extended — the control token for Gemini and grounding, which is not a crawler in the ordinary sense and does not affect Search ranking or inclusion.

The last four are where most confusion sits, and they are the reason a robots.txt decision cannot be taken on the word “Google” alone. Working out what’s actually crawling your site means separating the Search crawlers from the rest before deciding anything, and the mechanics of that decision — robots.txt versus meta robots — behave differently for crawling and for inclusion.

Rules that survive the next change

Allowlisting by literal IP address breaks the first time Google adds a range, and allowlisting by user agent string never worked. Both failure modes have the same fix: verify at request time or at ingest time, then act on the verified result. Fetch the JSON range files on a schedule rather than pasting their contents into a configuration file, treat an unverified request claiming to be Googlebot as ordinary traffic subject to your ordinary limits, and log the verification outcome alongside the request so the next person analysing the file does not have to repeat the work.