// SEO plans from €35/hour or €395/month// from €35/hr · €395/mo Free SEO checker[email protected]+31 6 4555 9353
Free SEO audit
technical seo

Robots Meta Tag and X-Robots-Tag

The robots meta tag and the X-Robots-Tag header control how Google indexes a page and shows it in results. This guide covers the syntax for both, every rule Google supports, how conflicts are resolved, how to add them in WordPress, Apache and NGINX, and the mistakes that quietly deindex pages.

  • Meta tag and header syntax
  • Every supported rule
  • Conflicts explained
  • Server examples
headrobots
<meta name="robots"
      content="noindex, nofollow">

X-Robots-Tag: noindex
// quick answer
Put <meta name=”robots” content=”…”> in the page head for HTML pages, or send an X-Robots-Tag HTTP header for any file type, including PDFs, images and video. Key rules: noindex, nofollow, nosnippet, max-snippet, max-image-preview, max-video-preview, unavailable_after. These only work if the URL is not blocked in robots.txt, because Google must crawl the page to see them.

Meta tag syntax

Place a meta tag with name=”robots” and a content value in the page head. Google also reads these tags if they appear in the body. Both the name and content attributes are case-insensitive.

<meta name="robots" content="noindex">
<meta name="googlebot" content="nofollow">
name valueApplies to
robotsAll crawlers that read the tag
googlebotAll Google text web results
googlebot-newsGoogle News results

Google recognises only those two crawler names; other names are ignored. Use separate meta tags for different crawlers. For basics see the noindex tag guide.

X-Robots-Tag HTTP header

For non-HTML files such as PDFs, videos and images, use the X-Robots-Tag header. It accepts any rule a meta tag accepts, as several headers or one comma-separated list. You can prefix a rule with a user agent and a colon, such as googlebot: nofollow. Without a prefix it applies to all crawlers. The header name, user agent name and values are case-insensitive.

HTTP/1.1 200 OK
X-Robots-Tag: noindex, nofollow
X-Robots-Tag: googlebot: nosnippet

Every rule Google supports

RuleEffect
allNo restrictions; the default
noindexKeeps the page, media or resource out of results
nofollowGoogle is told not to follow links on the page
noneShorthand for noindex plus nofollow
nosnippetBlocks text snippets and video previews, including in AI Overviews and AI Mode; a static image thumbnail may still appear
indexifembeddedLets Google index content embedded in another page, even with noindex; works only alongside noindex
max-snippet:[number]Caps the text snippet in characters; 0 equals nosnippet; -1 lets Google choose
max-image-preview:[setting]none, standard, or large (up to viewport width)
max-video-preview:[number]Maximum video snippet length in seconds; 0 allows at most a static image; -1 removes the limit
notranslateNo translated title link and snippet for other-language queries
noimageindexStops Google indexing images on the page
unavailable_after:[date]Removes the page from results after that date; accepts RFC 822, RFC 850 and ISO 8601; Googlebot then reduces crawling of the URL

Google lists noarchive, nocache and nositelinkssearchbox as historical rules that its search ignores, because the features they controlled no longer exist. Detail on snippet controls is in nosnippet, max-snippet and data-nosnippet.

Combining rules and conflicts

combineUse commas in one tag, or several tags
conflictThe more restrictive rule wins; nosnippet overrides max-snippet:50
generic+specificGoogle combines the restrictive rules from both
exampleGeneric nofollow plus googlebot noindex gives noindex and nofollow for Googlebot

The page must be crawlable

These settings are only read if crawlers can access the page. If robots.txt disallows the URL, the crawler never sees its meta tags or headers, so the rules are ignored. This is the most common reason noindex seems not to work. See noindex vs robots.txt and Blocked by robots.txt.

Google does not say how long rules take to apply. They are discovered when a URL is crawled, so timing depends on the next crawl.

Add a rule step by step

1
choose

Pick the goal

Keep a page out of results, limit snippets, or expire a page.

noindexnosnippetunavailable_after
2
choose

Meta tag or header?

HTML pages can use either; files like PDFs need the header.

metaheader
3
write

Add the rule

Use the syntax above with a comma-separated list.

content=”…”
4
allow

Make sure crawling is allowed

Check robots.txt does not block the URL.

robots.txt
5
test

Inspect the live page

Use the URL Inspection tool live test.

live test
6
wait

Let Google recrawl

Request indexing if urgent.

recrawl

Server examples

Google notes Apache (.htaccess or httpd.conf) and NGINX (location blocks) can apply headers site-wide by file pattern, or to a single file.

# Apache: noindex for PDFs (illustrative)
<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

# NGINX (illustrative)
location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}
These snippets are my own illustrations of the pattern Google describes. Test on a staging copy first.

Notes for WordPress sites

  • Use your SEO plugin. Per-page and per-type noindex settings usually exist. See Yoast SEO settings (my own note).
  • Site-wide discourage setting. WordPress has a visibility option that can noindex everything.
  • Archives. Decide per type; see archives: index or noindex.
  • Headers. Adding an X-Robots-Tag for PDFs needs server config or a plugin.

Troubleshooting

SymptomLikely cause (my own diagnosis)Fix
Noindex ignoredURL blocked in robots.txtAllow crawling
Rule not in page sourceSent as a headerCheck response headers
Snippet still longA different rule or not yet recrawledCheck conflicts and wait
Whole site hiddenSite-wide settingReverse it
PDF still indexedNo header setAdd X-Robots-Tag

Checklist

Syntax

  • name=”robots” or googlebot
  • Valid values
  • Comma-separated list
  • Header for non-HTML

Access

  • Not blocked in robots.txt
  • Page returns 200
  • Cache purged
  • Live test passes

Logic

  • No conflicting rules
  • noindex only where intended
  • indexifembedded paired with noindex
  • Dates valid

Review

  • Audit after launches
  • Check staging copies
  • Document intentional rules
  • Recheck monthly

What Google’s page does not say

Google does not say how long rules take to take effect, only that they are read when a URL is crawled. It also does not recognise crawler names beyond googlebot and googlebot-news in meta tags.

Key terms

Directive
An instruction such as noindex.
X-Robots-Tag
HTTP header carrying robots rules.
Snippet
The text description under a result.
User agent token
The name that identifies a crawler.

Common mistakes

mistake 1

Blocking and noindexing

Google cannot read the tag.

Allow crawling.
mistake 2

Meta tag on a PDF

Not possible.

Use the header.
mistake 3

Using noarchive

Ignored by Google Search.

Drop it.
mistake 4

Conflicting rules

The strictest wins.

Keep rules simple.
mistake 5

Forgetting staging noindex

Hides the live site.

Audit at launch.
mistake 6

Invalid values

Ignored silently.

Check formats.

Related guides

faq

Robots meta tag questions

Meta tag or X-Robots-Tag?

Use the meta tag for HTML pages, and the header for non-HTML files such as PDFs, videos and images.

Which crawler names work in meta tags?

Only googlebot and googlebot-news are recognised.

What does none mean?

noindex plus nofollow.

What happens when rules conflict?

The more restrictive rule wins.

Why is noindex ignored?

If robots.txt blocks the URL, Google never sees the tag.

Does nosnippet affect AI Overviews?

Google says nosnippet blocks text snippets across its surfaces, including AI Overviews and AI Mode.

Is noarchive supported?

No. Google lists it as ignored.

How long until rules apply?

Google does not say; they are read when the URL is next crawled.

sources

Syntax, supported rules, conflicts and the crawlability requirement come from Google’s page. The server snippets are my own illustrations; the step-by-step routine, WordPress notes, troubleshooting table and checklists are my own.

Scroll to Top