Google Crawlers and User Agents
Not every Google visitor is Googlebot. This guide explains Google’s three crawler types, the common user agent tokens, how to tell a real Google request from a fake one, how robots.txt rules apply to each, and how to read your server logs with confidence.
- 3 crawler types
- Token reference table
- Verify real Googlebot
- robots.txt examples
66.249.x.x GET /page/ 200 UA: Googlebot step 1 match the user agent step 2 check the source IP step 3 reverse DNS lookup => real Google, or not
Google uses three kinds of crawlers: common crawlers (such as Googlebot) that respect robots.txt, special-case crawlers for specific products (such as AdsBot) and user-triggered fetchers that run when a person asks for a fetch. Identify a request by its user agent, source IP address and reverse DNS. A user agent string alone can be faked, so verify before you trust or block it.
What a Google crawler is
A crawler is an automated program that requests pages from your server. Google’s documentation describes crawlers as the programs that fetch content for its products. Each one announces itself with a user agent, a label sent in every request. Googlebot is the best known, but it is only one of several.
Different bots, different rules
A robots.txt rule written for Googlebot does not always apply to another Google crawler.
Fakes exist
Anyone can send “Googlebot” as a user agent. Real checks need more than the label.
For the bigger picture, read how Google Search works.
The three types of Google crawler
| Type | What it does | robots.txt |
|---|---|---|
| Common crawlers | Build Google’s search index and power other products. Googlebot is the main one | Always respect robots.txt rules |
| Special-case crawlers | Serve specific products, such as ads checks, and may have an agreement with the site | May or may not follow the general rules |
| User-triggered fetchers | Fetch a URL because a person asked, for example a site tool | Rules usually do not apply, because a user initiated the fetch |
The three categories and their robots.txt behaviour follow Google’s crawler overview. Check the live page for the exact wording per crawler.
Common user agent tokens
The token is the name you use in robots.txt. These are tokens named in Google’s crawler documentation; the descriptions are short summaries.
| Token | Used for | Type |
|---|---|---|
| Googlebot | Search crawling (smartphone and desktop) | Common |
| Storebot-Google | Shopping and product pages | Common |
| AdsBot | Checks ad landing page quality | Special-case |
| APIs-Google | Delivers push notification messages | Special-case |
| Feedfetcher | Fetches RSS and Atom feeds | User-triggered |
| Google Read Aloud | Text-to-speech reading of pages | User-triggered |
Google also documents image, video and news crawlers. See the live page for the full, current list, because it changes.
How to identify a real Google request
Google says you can recognise its crawlers by three things. Use all three when it matters.
For a hands-on method, follow the guide to verify Googlebot. If you only match the user agent, you may be counting scrapers as Google.
How robots.txt treats Google crawlers
- Rules are grouped by user agent. Googlebot follows the most specific group that matches it.
- A
*group applies to crawlers that do not have their own group, except AdsBot, which ignores the*group and must be named explicitly. - Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. See indexed though blocked by robots.txt.
- To keep a page out of results, use noindex, and keep the page crawlable so Google can see the tag.
Robots.txt examples
# Allow everyone except one folder User-agent: * Disallow: /private/ # A separate rule only for Googlebot User-agent: Googlebot Disallow: /search/ # AdsBot ignores * so name it if you need a rule User-agent: AdsBot-Google Disallow: /ad-test/
Check who is crawling you, step by step
Get your server logs
Ask your host for raw access logs, or open them in your control panel (my own tip). Each line shows an IP, a time, a URL, a status code and a user agent.
Search for Google tokens
Filter lines containing Googlebot, AdsBot or another token from the table.
Check the IP
Compare the IP with Google’s published ranges and run a reverse DNS lookup, as explained in verify Googlebot.
Cross-check Search Console
Open the Crawl Stats report and compare Google’s numbers with your logs.
Fix or block deliberately
Fix server errors for real Googlebot. Block only fakes you have verified, at the server or firewall, not in robots.txt.
Notes for WordPress sites
- Security plugins. Firewalls and bot-protection can block real Googlebot by mistake. After enabling one, test with the URL Inspection tool.
- Cloudflare. Bot settings can interfere with crawling. See Cloudflare SEO settings.
- Virtual robots.txt. WordPress creates one automatically if no file exists; edit it from your SEO plugin.
- Server load. If crawling slows your site, read crawl budget before blocking anything.
Troubleshooting
| Symptom | Likely cause (my own diagnosis) | Fix |
|---|---|---|
| Lots of “Googlebot” hits, but Search Console shows few | Many are fake bots | Verify by IP and reverse DNS |
| AdsBot ignores your * rule | AdsBot needs its own group | Add a named AdsBot group |
| Real Googlebot gets 403 or 5xx | Firewall or server overload | Allow Google ranges, fix the server |
| Blocked page still appears | Robots.txt does not remove URLs | Use noindex or removal tools |
| Tool fetch ignored robots.txt | User-triggered fetcher | Expected behaviour for that type |
Server errors seen by Googlebot are covered in server error 5xx in Search Console.
Checklist
Know the bots
- Common, special-case, user-triggered
- Tokens you use in robots.txt
- AdsBot named separately
- Image and video bots as needed
Verify
- User agent matched
- IP in Google’s ranges
- Reverse DNS checked
- Forward DNS confirms the IP
Rules
- Robots.txt tested
- No accidental site-wide Disallow
- Noindex used for removal
- Sitemap line included
Monitor
- Crawl Stats checked monthly
- Logs scanned for errors
- Firewall rules reviewed
- Fakes blocked at server level
What Google’s page does not say
Google’s overview does not promise how often any crawler will visit, and it does not let you request a faster crawl rate through robots.txt. It lists tokens and IP ranges, but those lists change, so rely on the live page rather than a copy. It also does not tell you to block unknown bots; that is your own decision.
Key terms
- User agent
- The label a program sends to say who it is.
- Token
- The short product name used in robots.txt, such as Googlebot.
- Reverse DNS
- Looking up the hostname that belongs to an IP address.
- Fetcher
- A tool that retrieves a page when a person asks.
Common mistakes
Trusting the user agent alone
It is easy to fake.
Assuming * covers AdsBot
AdsBot ignores the * group.
Using robots.txt to deindex
It controls crawling only.
Blocking Google at the firewall
Real Googlebot then gets errors.
Copying an old IP list
Ranges change.
Noindex plus Disallow
Google cannot see noindex on a blocked page.
Related guides
Google crawler questions
What are the three types of Google crawler?
Common crawlers, special-case crawlers and user-triggered fetchers.
How can I tell if a request is really Google?
Check the user agent, the source IP against Google’s published ranges, and reverse DNS. A user agent alone is not proof.
Does AdsBot follow the * rule in robots.txt?
No. AdsBot ignores the * group, so it needs its own named group.
Does robots.txt stop a page appearing in Google?
No. It controls crawling. Use noindex or the removal tools to keep a page out of results.
Do user-triggered fetchers obey robots.txt?
Generally no, because a person started the fetch. Check the specific crawler’s entry in Google’s documentation.
Where do I see Google’s crawl activity?
In the Crawl Stats report in Search Console, and in your own server logs.
Should I block unknown bots?
Only after verifying them. Block at the server or firewall level rather than relying on robots.txt (my own advice).
Can I make Googlebot crawl faster?
Google does not offer a robots.txt setting for that. Improve server speed and site quality instead.
The crawler types, tokens, identification methods and the AdsBot note come from those pages. The log-checking steps, WordPress notes, troubleshooting table and checklists are my own.