Robots Meta Tag and X-Robots-Tag
The robots meta tag and the X-Robots-Tag header control how Google indexes a page and shows it in results. This guide covers the syntax for both, every rule Google supports, how conflicts are resolved, how to add them in WordPress, Apache and NGINX, and the mistakes that quietly deindex pages.
- Meta tag and header syntax
- Every supported rule
- Conflicts explained
- Server examples
<meta name="robots"
content="noindex, nofollow">
X-Robots-Tag: noindexPut <meta name=”robots” content=”…”> in the page head for HTML pages, or send an X-Robots-Tag HTTP header for any file type, including PDFs, images and video. Key rules: noindex, nofollow, nosnippet, max-snippet, max-image-preview, max-video-preview, unavailable_after. These only work if the URL is not blocked in robots.txt, because Google must crawl the page to see them.
Meta tag syntax
Place a meta tag with name=”robots” and a content value in the page head. Google also reads these tags if they appear in the body. Both the name and content attributes are case-insensitive.
<meta name="robots" content="noindex"> <meta name="googlebot" content="nofollow">
| name value | Applies to |
|---|---|
| robots | All crawlers that read the tag |
| googlebot | All Google text web results |
| googlebot-news | Google News results |
Google recognises only those two crawler names; other names are ignored. Use separate meta tags for different crawlers. For basics see the noindex tag guide.
X-Robots-Tag HTTP header
For non-HTML files such as PDFs, videos and images, use the X-Robots-Tag header. It accepts any rule a meta tag accepts, as several headers or one comma-separated list. You can prefix a rule with a user agent and a colon, such as googlebot: nofollow. Without a prefix it applies to all crawlers. The header name, user agent name and values are case-insensitive.
HTTP/1.1 200 OK X-Robots-Tag: noindex, nofollow X-Robots-Tag: googlebot: nosnippet
Every rule Google supports
| Rule | Effect |
|---|---|
| all | No restrictions; the default |
| noindex | Keeps the page, media or resource out of results |
| nofollow | Google is told not to follow links on the page |
| none | Shorthand for noindex plus nofollow |
| nosnippet | Blocks text snippets and video previews, including in AI Overviews and AI Mode; a static image thumbnail may still appear |
| indexifembedded | Lets Google index content embedded in another page, even with noindex; works only alongside noindex |
| max-snippet:[number] | Caps the text snippet in characters; 0 equals nosnippet; -1 lets Google choose |
| max-image-preview:[setting] | none, standard, or large (up to viewport width) |
| max-video-preview:[number] | Maximum video snippet length in seconds; 0 allows at most a static image; -1 removes the limit |
| notranslate | No translated title link and snippet for other-language queries |
| noimageindex | Stops Google indexing images on the page |
| unavailable_after:[date] | Removes the page from results after that date; accepts RFC 822, RFC 850 and ISO 8601; Googlebot then reduces crawling of the URL |
Google lists noarchive, nocache and nositelinkssearchbox as historical rules that its search ignores, because the features they controlled no longer exist. Detail on snippet controls is in nosnippet, max-snippet and data-nosnippet.
Combining rules and conflicts
The page must be crawlable
These settings are only read if crawlers can access the page. If robots.txt disallows the URL, the crawler never sees its meta tags or headers, so the rules are ignored. This is the most common reason noindex seems not to work. See noindex vs robots.txt and Blocked by robots.txt.
Add a rule step by step
Pick the goal
Keep a page out of results, limit snippets, or expire a page.
Meta tag or header?
HTML pages can use either; files like PDFs need the header.
Add the rule
Use the syntax above with a comma-separated list.
Make sure crawling is allowed
Check robots.txt does not block the URL.
Server examples
Google notes Apache (.htaccess or httpd.conf) and NGINX (location blocks) can apply headers site-wide by file pattern, or to a single file.
# Apache: noindex for PDFs (illustrative)
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>
# NGINX (illustrative)
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}Notes for WordPress sites
- Use your SEO plugin. Per-page and per-type noindex settings usually exist. See Yoast SEO settings (my own note).
- Site-wide discourage setting. WordPress has a visibility option that can noindex everything.
- Archives. Decide per type; see archives: index or noindex.
- Headers. Adding an X-Robots-Tag for PDFs needs server config or a plugin.
Troubleshooting
| Symptom | Likely cause (my own diagnosis) | Fix |
|---|---|---|
| Noindex ignored | URL blocked in robots.txt | Allow crawling |
| Rule not in page source | Sent as a header | Check response headers |
| Snippet still long | A different rule or not yet recrawled | Check conflicts and wait |
| Whole site hidden | Site-wide setting | Reverse it |
| PDF still indexed | No header set | Add X-Robots-Tag |
Checklist
Syntax
- name=”robots” or googlebot
- Valid values
- Comma-separated list
- Header for non-HTML
Access
- Not blocked in robots.txt
- Page returns 200
- Cache purged
- Live test passes
Logic
- No conflicting rules
- noindex only where intended
- indexifembedded paired with noindex
- Dates valid
Review
- Audit after launches
- Check staging copies
- Document intentional rules
- Recheck monthly
What Google’s page does not say
Google does not say how long rules take to take effect, only that they are read when a URL is crawled. It also does not recognise crawler names beyond googlebot and googlebot-news in meta tags.
Key terms
- Directive
- An instruction such as noindex.
- X-Robots-Tag
- HTTP header carrying robots rules.
- Snippet
- The text description under a result.
- User agent token
- The name that identifies a crawler.
Common mistakes
Blocking and noindexing
Google cannot read the tag.
Meta tag on a PDF
Not possible.
Using noarchive
Ignored by Google Search.
Conflicting rules
The strictest wins.
Forgetting staging noindex
Hides the live site.
Invalid values
Ignored silently.
Related guides
Robots meta tag questions
Meta tag or X-Robots-Tag?
Use the meta tag for HTML pages, and the header for non-HTML files such as PDFs, videos and images.
Which crawler names work in meta tags?
Only googlebot and googlebot-news are recognised.
What does none mean?
noindex plus nofollow.
What happens when rules conflict?
The more restrictive rule wins.
Why is noindex ignored?
If robots.txt blocks the URL, Google never sees the tag.
Does nosnippet affect AI Overviews?
Google says nosnippet blocks text snippets across its surfaces, including AI Overviews and AI Mode.
Is noarchive supported?
No. Google lists it as ignored.
How long until rules apply?
Google does not say; they are read when the URL is next crawled.
Syntax, supported rules, conflicts and the crawlability requirement come from Google’s page. The server snippets are my own illustrations; the step-by-step routine, WordPress notes, troubleshooting table and checklists are my own.