// SEO plans from €35/hour or €395/month// from €35/hr · €395/mo Free SEO checker[email protected]+31 6 4555 9353
Free SEO audit
technical seo

Google Crawlers and User Agents

Not every Google visitor is Googlebot. This guide explains Google’s three crawler types, the common user agent tokens, how to tell a real Google request from a fake one, how robots.txt rules apply to each, and how to read your server logs with confidence.

  • 3 crawler types
  • Token reference table
  • Verify real Googlebot
  • robots.txt examples
access.loguser agent
66.249.x.x GET /page/ 200
UA: Googlebot

step 1  match the user agent
step 2  check the source IP
step 3  reverse DNS lookup
=> real Google, or not
// quick answer
Google uses three kinds of crawlers: common crawlers (such as Googlebot) that respect robots.txt, special-case crawlers for specific products (such as AdsBot) and user-triggered fetchers that run when a person asks for a fetch. Identify a request by its user agent, source IP address and reverse DNS. A user agent string alone can be faked, so verify before you trust or block it.

What a Google crawler is

A crawler is an automated program that requests pages from your server. Google’s documentation describes crawlers as the programs that fetch content for its products. Each one announces itself with a user agent, a label sent in every request. Googlebot is the best known, but it is only one of several.

why it matters

Different bots, different rules

A robots.txt rule written for Googlebot does not always apply to another Google crawler.

why it matters

Fakes exist

Anyone can send “Googlebot” as a user agent. Real checks need more than the label.

For the bigger picture, read how Google Search works.

The three types of Google crawler

TypeWhat it doesrobots.txt
Common crawlersBuild Google’s search index and power other products. Googlebot is the main oneAlways respect robots.txt rules
Special-case crawlersServe specific products, such as ads checks, and may have an agreement with the siteMay or may not follow the general rules
User-triggered fetchersFetch a URL because a person asked, for example a site toolRules usually do not apply, because a user initiated the fetch

The three categories and their robots.txt behaviour follow Google’s crawler overview. Check the live page for the exact wording per crawler.

Common user agent tokens

The token is the name you use in robots.txt. These are tokens named in Google’s crawler documentation; the descriptions are short summaries.

TokenUsed forType
GooglebotSearch crawling (smartphone and desktop)Common
Storebot-GoogleShopping and product pagesCommon
AdsBotChecks ad landing page qualitySpecial-case
APIs-GoogleDelivers push notification messagesSpecial-case
FeedfetcherFetches RSS and Atom feedsUser-triggered
Google Read AloudText-to-speech reading of pagesUser-triggered

Google also documents image, video and news crawlers. See the live page for the full, current list, because it changes.

How to identify a real Google request

Google says you can recognise its crawlers by three things. Use all three when it matters.

user agentThe text in the request header, such as Googlebot. Easy to fake
source IPGoogle publishes lists of IP ranges used by its crawlers
reverse DNSThe IP should resolve to a Google domain, and that name should resolve back to the same IP

For a hands-on method, follow the guide to verify Googlebot. If you only match the user agent, you may be counting scrapers as Google.

How robots.txt treats Google crawlers

  • Rules are grouped by user agent. Googlebot follows the most specific group that matches it.
  • A * group applies to crawlers that do not have their own group, except AdsBot, which ignores the * group and must be named explicitly.
  • Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. See indexed though blocked by robots.txt.
  • To keep a page out of results, use noindex, and keep the page crawlable so Google can see the tag.

Robots.txt examples

# Allow everyone except one folder
User-agent: *
Disallow: /private/

# A separate rule only for Googlebot
User-agent: Googlebot
Disallow: /search/

# AdsBot ignores * so name it if you need a rule
User-agent: AdsBot-Google
Disallow: /ad-test/
Test every change. A single wrong Disallow line can hide a whole section. Read robots.txt and sitemaps before editing.

Check who is crawling you, step by step

1
find the data

Get your server logs

Ask your host for raw access logs, or open them in your control panel (my own tip). Each line shows an IP, a time, a URL, a status code and a user agent.

access loglast 7 days
2
filter

Search for Google tokens

Filter lines containing Googlebot, AdsBot or another token from the table.

grep Googlebot
3
verify

Check the IP

Compare the IP with Google’s published ranges and run a reverse DNS lookup, as explained in verify Googlebot.

IP rangereverse DNS
4
compare

Cross-check Search Console

Open the Crawl Stats report and compare Google’s numbers with your logs.

requestsresponse codes
5
act

Fix or block deliberately

Fix server errors for real Googlebot. Block only fakes you have verified, at the server or firewall, not in robots.txt.

fix 5xxblock fakes

Notes for WordPress sites

  • Security plugins. Firewalls and bot-protection can block real Googlebot by mistake. After enabling one, test with the URL Inspection tool.
  • Cloudflare. Bot settings can interfere with crawling. See Cloudflare SEO settings.
  • Virtual robots.txt. WordPress creates one automatically if no file exists; edit it from your SEO plugin.
  • Server load. If crawling slows your site, read crawl budget before blocking anything.

Troubleshooting

SymptomLikely cause (my own diagnosis)Fix
Lots of “Googlebot” hits, but Search Console shows fewMany are fake botsVerify by IP and reverse DNS
AdsBot ignores your * ruleAdsBot needs its own groupAdd a named AdsBot group
Real Googlebot gets 403 or 5xxFirewall or server overloadAllow Google ranges, fix the server
Blocked page still appearsRobots.txt does not remove URLsUse noindex or removal tools
Tool fetch ignored robots.txtUser-triggered fetcherExpected behaviour for that type

Server errors seen by Googlebot are covered in server error 5xx in Search Console.

Checklist

Know the bots

  • Common, special-case, user-triggered
  • Tokens you use in robots.txt
  • AdsBot named separately
  • Image and video bots as needed

Verify

  • User agent matched
  • IP in Google’s ranges
  • Reverse DNS checked
  • Forward DNS confirms the IP

Rules

  • Robots.txt tested
  • No accidental site-wide Disallow
  • Noindex used for removal
  • Sitemap line included

Monitor

  • Crawl Stats checked monthly
  • Logs scanned for errors
  • Firewall rules reviewed
  • Fakes blocked at server level

What Google’s page does not say

Google’s overview does not promise how often any crawler will visit, and it does not let you request a faster crawl rate through robots.txt. It lists tokens and IP ranges, but those lists change, so rely on the live page rather than a copy. It also does not tell you to block unknown bots; that is your own decision.

Key terms

User agent
The label a program sends to say who it is.
Token
The short product name used in robots.txt, such as Googlebot.
Reverse DNS
Looking up the hostname that belongs to an IP address.
Fetcher
A tool that retrieves a page when a person asks.

Common mistakes

mistake 1

Trusting the user agent alone

It is easy to fake.

Verify IP and reverse DNS.
mistake 2

Assuming * covers AdsBot

AdsBot ignores the * group.

Name it in its own group.
mistake 3

Using robots.txt to deindex

It controls crawling only.

Use noindex or a removal request.
mistake 4

Blocking Google at the firewall

Real Googlebot then gets errors.

Allow verified ranges.
mistake 5

Copying an old IP list

Ranges change.

Use Google’s current published list.
mistake 6

Noindex plus Disallow

Google cannot see noindex on a blocked page.

Allow crawling for noindex pages.

Related guides

faq

Google crawler questions

What are the three types of Google crawler?

Common crawlers, special-case crawlers and user-triggered fetchers.

How can I tell if a request is really Google?

Check the user agent, the source IP against Google’s published ranges, and reverse DNS. A user agent alone is not proof.

Does AdsBot follow the * rule in robots.txt?

No. AdsBot ignores the * group, so it needs its own named group.

Does robots.txt stop a page appearing in Google?

No. It controls crawling. Use noindex or the removal tools to keep a page out of results.

Do user-triggered fetchers obey robots.txt?

Generally no, because a person started the fetch. Check the specific crawler’s entry in Google’s documentation.

Where do I see Google’s crawl activity?

In the Crawl Stats report in Search Console, and in your own server logs.

Should I block unknown bots?

Only after verifying them. Block at the server or firewall level rather than relying on robots.txt (my own advice).

Can I make Googlebot crawl faster?

Google does not offer a robots.txt setting for that. Improve server speed and site quality instead.

sources

The crawler types, tokens, identification methods and the AdsBot note come from those pages. The log-checking steps, WordPress notes, troubleshooting table and checklists are my own.

Scroll to Top