Cloudflare SEO Settings: Avoid Blocking Googlebot
Cloudflare can speed up and protect a site, but security and caching features can also challenge or block search crawlers if misconfigured. This guide explains how to check that Googlebot reaches your pages, what Cloudflare documents about verified bots, and which settings to review. Cloudflare’s own page does not describe Googlebot handling, so the checks here are practical and labelled.
- Verify crawler access
- Review security rules
- Redirect and SSL checks
- Cache safely
URL Inspection live test -> available to Google? Cloudflare events -> any challenge/block?
After enabling Cloudflare, run a live test in URL Inspection and check Cloudflare’s security events for blocked or challenged Google requests. Cloudflare describes Verified Bots as bots that identify themselves reliably and obey robots.txt, with customers able to set block or allow behaviour through policies. Its page does not say how Googlebot is treated or how Bot Fight Mode affects verified bots, so verify, do not assume.
What Cloudflare says about verified bots
- A verified bot is open about its identity and does not misuse that access; examples include search crawlers and monitoring services.
- It must identify itself deterministically, using a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS.
- It must obey robots.txt and crawl directives and keep request rates reasonable.
- Customers can set block or allow behaviour through AI Crawler and Agent access policies; legacy categories such as Search Engine Crawler still work in WAF custom rules.
Googlebot appears only as part of legacy categories in that page, with no specific handling described. See verify Googlebot.
Test that Google can reach you
Security rules to review (my own checklist)
- Bot fight settings and managed challenges.
- WAF custom rules and rate limiting.
- Country or IP blocks.
- Browser integrity checks.
- Under Attack mode, which should be temporary.
Cloudflare’s verified bots page does not discuss how Bot Fight Mode or Super Bot Fight Mode interact with verified bots, so test rather than assume.
SSL and redirect checks
A mismatch between Cloudflare’s SSL mode and your origin settings can create redirect loops (my own note). See redirect chains and loops and HTTPS and SEO. Avoid stacking Cloudflare redirects on top of server and plugin redirects.
Caching caution
- Do not cache admin or logged-in pages.
- Purge the cache after major content changes so crawlers see current pages (my own advice).
- Check that cached error pages are not served to crawlers.
Set up safely step by step
Record baseline
Crawl stats before changes.
Enable Cloudflare
With default settings.
Live test key pages
Home, post, robots.txt.
Read security events
Look for Google blocks.
Tune rules
Allow verified crawlers.
Monitor weekly
Crawl stats and errors.
Allowing genuine Googlebot only
Do not allowlist by user agent alone, since it can be spoofed. Verify using Google’s methods (verify Googlebot). Cloudflare’s page gives no step-by-step allowlisting process.
Notes for WordPress sites
- Plugin conflicts: a cache plugin plus Cloudflare caching can double up; configure one primary layer.
- wp-admin and login: protect, but do not break feeds or REST used by tools.
- Sitemaps: ensure they are not challenged (sitemap guide).
- Origin errors: see server errors.
Troubleshooting
| Symptom | Likely cause (my own diagnosis) | Action |
|---|---|---|
| Live test fails | Challenge or block | Review security events |
| Too many redirects | SSL mode mismatch | Align SSL settings |
| Stale content in search | Old cache served | Purge cache |
| Crawl errors spike | Rate limiting rules | Relax for verified crawlers |
Checklist
Access
- Live test passes
- robots.txt reachable
- Sitemap reachable
- No challenges for Google
Security
- WAF rules reviewed
- Rate limits sensible
- No broad country blocks
- Under Attack off
Redirects
- SSL mode correct
- No loops
- Single redirect layer
- HTTPS enforced
Cache
- Admin excluded
- Purge on changes
- No cached errors
- One cache layer
What Cloudflare’s page does not say
The verified bots page does not describe how Googlebot is verified or treated, how Bot Fight Mode or Super Bot Fight Mode affects verified bots, step-by-step allowlisting, review timelines, numeric definitions of reasonable request rates, or how AI crawler policies interact with verified status. All settings advice here is practical and should be tested on your own site.
Key terms
- Verified bot
- Bot Cloudflare recognises as legitimate.
- WAF
- Web application firewall.
- Challenge
- Check that blocks simple bots.
- Purge
- Clear cached copies.
Common mistakes
Assuming Googlebot is allowed
Not documented.
Allowlisting by user agent
Can be spoofed.
Wrong SSL mode
Redirect loops.
Caching everything
Stale or private pages.
Harsh rate limits
Crawl errors.
No monitoring
Blocks unnoticed.
Related guides
Cloudflare SEO questions
Does Cloudflare block Googlebot?
Not by default per its page, which gives no Googlebot detail; test your own site.
What is a verified bot?
A bot that identifies itself reliably and follows robots.txt and reasonable rates.
How do I test?
URL Inspection live test and Cloudflare security events.
Does Bot Fight Mode affect verified bots?
Cloudflare’s page does not say.
Can SSL settings hurt SEO?
A wrong mode can cause redirect loops.
Should I cache HTML?
Carefully; exclude admin and purge on updates.
How do I allow Google?
Cloudflare gives no step-by-step process; verify crawlers and tune rules.
Do I need Cloudflare for SEO?
No; it is optional.
The verified bot definitions come from Cloudflare’s page. The testing method, settings checklist, steps, WordPress notes and troubleshooting are my own.