How to Verify Googlebot
Anyone can claim to be Googlebot. This guide shows the two methods Google documents to prove a visitor is real: a manual reverse and forward DNS check, and matching the IP against Google’s published ranges. Includes exact commands, worked examples, a decision table and a log-reading routine.
- 2 official methods
- Exact commands
- 3 worked examples
- Decision table
$ host 66.249.66.1 crawl-66-249-66-1.googlebot.com $ host crawl-66-249-66-1.googlebot.com 66.249.66.1 => names match: real Googlebot
Google documents two ways. Method 1 (manual): run a reverse DNS lookup on the IP from your logs, confirm the domain is googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that name and confirm it returns the same IP. Method 2 (automatic): match the IP against Google’s published JSON range lists. Google says the manual method is enough for most uses.
Why you need to verify
Every request carries a user agent string, and anyone can type “Googlebot” into it. Scrapers, vulnerability scanners and bad bots do exactly that to slip past rules that trust Google. Verifying means checking something the visitor cannot fake: the DNS records tied to the IP address.
Trust the user agent
It is only a label.
Check IP and DNS
Google controls the DNS for its crawler addresses.
For the full crawler list, read Google crawlers and user agents.
Method 1: manual reverse and forward DNS
Look up the IP
Take the IP from your access log and run host 66.249.66.1. This returns the hostname.
Confirm the domain
The hostname must end in googlebot.com, google.com or googleusercontent.com.
Look up the hostname
Run host crawl-66-249-66-1.googlebot.com.
Does it return the same IP?
If the forward lookup returns the original IP, the visitor is verified. If not, it is not Google.
The forward step matters. A fake can set a reverse record that looks like Google, but only Google can make that Google hostname resolve back to the same IP.
Three worked examples from Google’s page
| IP address | Reverse lookup result | Forward result |
|---|---|---|
| 66.249.66.1 | crawl-66-249-66-1.googlebot.com | 66.249.66.1 (match) |
| 35.247.243.240 | geo-crawl-35-247-243-240.geo.googlebot.com | 35.247.243.240 (match) |
| 66.249.90.77 | rate-limited-proxy-66-249-90-77.google.com | 66.249.90.77 (match) |
These are Google’s own examples. Real addresses change over time, so always run the live check rather than memorising addresses.
Method 2: match the IP against Google’s lists
Google publishes lists of crawler IP ranges as JSON files, written in CIDR notation. A script can compare a visitor’s IP with the right list.
| File | Covers | Reverse DNS domain |
|---|---|---|
| common-crawlers.json | Googlebot and other common crawlers | see Google’s page |
| special-crawlers.json | Special-case crawlers such as AdsBot | see Google’s page |
| user-triggered-fetchers.json | User-triggered fetchers | gae.googleusercontent.com |
| user-triggered-fetchers-google.json | Fetchers controlled by Google | google.com |
| user-triggered-agents.json | User-triggered agents | see Google’s page |
For other Google IP addresses, such as those used by Apps Script, Google refers to a general list called goog.json on gstatic.com. Download the live files from Google’s documentation page rather than copying a saved version, because the ranges change.
Which method should you use?
Robots.txt and verification
- Common crawlers always respect robots.txt for automatic crawls.
- Special-case crawlers may or may not respect it.
- User-triggered fetchers ignore robots.txt because a person requested the fetch.
So a verified request from a user-triggered fetcher that ignores your Disallow rule is normal behaviour, not an attack. Learn the rules in robots.txt and sitemaps.
Check your logs step by step
Get the log lines
Pull lines whose user agent contains Googlebot. Example: grep -i googlebot access.log.
List the unique IPs
Extract the IP column and remove duplicates, for example with awk '{print $1}' | sort -u (my own tip).
Run the DNS check on each
Use Method 1 on every IP, or load the JSON ranges and compare.
Separate real from fake
Keep a list of verified and unverified IPs.
Block only the fakes
Block unverified IPs at the server or firewall, not with robots.txt. Never block verified Google ranges by mistake.
Cross-check Search Console
See the Crawl Stats report. If your logs show far more “Googlebot” hits than Google reports, many are fakes.
Notes for WordPress sites
- Get raw logs from your host. Many hosts offer them in the control panel.
- Security plugins may flag or block unverified bots. Make sure real Googlebot is not blocked; test with the URL Inspection tool.
- Cloudflare can identify verified bots for you. See Cloudflare SEO settings.
- Do not allow-list by user agent. That opens a hole.
Troubleshooting
| Symptom | Likely cause (my own diagnosis) | Fix |
|---|---|---|
| Reverse lookup returns nothing | Not Google, or DNS issue | Treat as unverified; retry once |
| Hostname not on a Google domain | Fake crawler | Block at the firewall |
| Forward lookup returns a different IP | Spoofed reverse record | Unverified |
| Real Googlebot is blocked | Firewall rule too strict | Allow verified ranges |
| Many unverified hits | Scrapers | Rate limit or block them |
Checklist
Method 1
- Reverse lookup done
- Domain is one of the 3
- Forward lookup done
- IP matches
Method 2
- Latest JSON files downloaded
- Right file for the crawler
- CIDR matching used
- Lists refreshed regularly
Actions
- Fakes blocked at server
- Verified Google allowed
- No user agent allow-lists
- Rules tested
Follow-up
- Crawl Stats compared
- URL Inspection live test passes
- Logs reviewed monthly
- Alerts for spikes
What Google’s page does not say
Google does not promise that its IP ranges stay the same, does not describe how to build your own blocking rules and does not say how often fake Googlebots appear. The share of fake traffic on your site is something only your own logs can show.
Key terms
- Reverse DNS
- Finding the hostname for an IP address.
- Forward DNS
- Finding the IP address for a hostname.
- CIDR
- A compact way to write an IP range, such as 66.249.64.0/27.
- Spoofing
- Faking an identity, such as a user agent.
Common mistakes
Skipping the forward lookup
A reverse record alone can be faked.
Trusting the user agent
It is trivial to fake.
Using an old IP list
Ranges change.
Blocking via robots.txt
Bad bots ignore it.
Blocking all Google IPs
You lose crawling.
Using the wrong JSON file
Different crawlers, different lists.
Related guides
Verify Googlebot questions
What are the two ways to verify Googlebot?
Manual reverse and forward DNS lookup, or matching the IP against Google’s published JSON range lists.
Which domains should the reverse lookup return?
googlebot.com, google.com or googleusercontent.com.
Is the user agent enough?
No. It can be faked.
Which method does Google recommend?
Google says the manual method is sufficient for most use cases; the automatic method suits large-scale lookups.
Do all Google crawlers obey robots.txt?
Common crawlers always do for automatic crawls; special-case crawlers may not; user-triggered fetchers ignore it.
Where are the IP lists?
Google publishes JSON files such as common-crawlers.json on its documentation page.
What if verification fails?
Treat the visitor as not Google. Block or rate-limit at the server if it is abusive.
Will blocking fake Googlebots hurt SEO?
No, if they are truly fake. Just avoid blocking verified Google ranges.
- Google Search Central: Verifying Googlebot and other Google crawlers
- Google Search Central: Google’s common crawlers
Both methods, the three domains, the three worked examples, the JSON file names and the robots.txt notes come from Google’s page. The log commands, step-by-step routine, WordPress notes, troubleshooting table and checklists are my own.