// SEO plans from €35/hour or €395/month// from €35/hr · €395/mo Free SEO checker[email protected]+31 6 4555 9353
Free SEO audit
technical seo

robots.txt Syntax and Rules

A robots.txt file is short, but a small syntax mistake can block your whole site. This guide explains where the file goes, every rule Google supports, how groups are processed, wildcards, copy-paste examples, and how to test before you upload.

  • File location rules
  • User-agent, Disallow, Allow, Sitemap
  • Wildcards and groups
  • Test and submit
/robots.txtexample
User-agent: *
Disallow: /private/
Allow: /private/public/

Sitemap: https://example.com/sitemap.xml
// quick answer
Name the file robots.txt, put it at the root of the host, save it as UTF-8 plain text, and write groups that start with User-agent followed by Disallow and Allow lines, one per line. Add Sitemap lines with full URLs. Anything you do not disallow is crawlable by default. Test it in Search Console before relying on it.

File basics

RuleWhat Google says
Namerobots.txt, exactly
LocationThe root of the host it governs
ScopeApplies only to the protocol, host and port where it is posted; subdomains and non-standard ports can have their own
QuantityOne file per site
EncodingUTF-8; use a plain text editor because word processors can add odd characters such as curly quotes

For the overall picture of how robots.txt and sitemaps fit together, see robots.txt and sitemaps.

The four rules Google supports

RuleWhat it doesFormat notes
User-agentNames the crawler a group applies to; * means all crawlers except AdsBot crawlers, which must be namedStarts each group
DisallowA path the crawler should not crawlPath starts with /; directory paths end with /
AllowOverrides a Disallow to permit a subfolder or pageSame path format
SitemapFully qualified URL of a sitemap; zero or more allowedGoogle does not assume http/https or www variants

Other details: paths are case-sensitive, a line starting with # is a comment, and unrecognised lines are ignored.

How groups are processed

groupOne or more User-agent lines followed by their rules
readingCrawlers read groups top to bottom
matchA crawler uses only the first, most specific group matching its user agent
mergeSeveral groups for the same agent are merged first
Specific beats general. If you have a Googlebot group, Googlebot uses it and ignores the * group (my reading of Google’s description).

Wildcards

Google says all rules except Sitemap support the * wildcard, for prefixes, suffixes or whole strings.

Disallow: /*.pdf    # paths containing .pdf (illustrative)
Disallow: /search*  # paths starting with /search

The page I used does not describe the $ end-of-string character, so I do not explain it here. Check Google’s full robots.txt specification if you need it.

Copy-paste examples

allow everything
User-agent: *
Disallow:
block one folder
User-agent: *
Disallow: /private/
block only Googlebot
User-agent: Googlebot
Disallow: /
add a sitemap
Sitemap: https://example.com/sitemap.xml

The first example (an empty Disallow, meaning nothing blocked) is my own illustration. The Googlebot-only block mirrors an example in Google’s documentation.

Create and upload it step by step

1
create

Open a plain text editor

Create a file named robots.txt, saved as UTF-8.

plain textUTF-8
2
write

Add your groups

User-agent first, then Disallow and Allow lines, one per line.

User-agentDisallow
3
sitemap

Add Sitemap lines

Use the full URL of each sitemap.

https://…
4
upload

Put it at the root

The method depends on your hosting; ask your host if unsure.

/robots.txt
5
check

Open it in a private window

This confirms it is publicly reachable.

private window
6
test

Use the Search Console report

Verify Google can read and interpret it.

robots.txt report

Testing and getting Google to refresh

Google gives two testing options: the robots.txt report in Search Console, which only works for files already live on your site, and Google’s open-source robots.txt library for local testing by developers. Google’s crawlers find and use the file automatically once it is uploaded; if you need a faster refresh of its cached copy, Google has a separate guide on submitting an updated file.

files1per site
encodingUTF-8plain text
defaultallowedunless disallowed
testreportSearch Console

Notes for WordPress sites

  • It may be virtual. WordPress and SEO plugins can generate robots.txt without a physical file (my own note). See Yoast SEO settings.
  • Do not block what pages need. Blocking theme or plugin assets can hinder rendering (my own advice).
  • Staging. A blanket Disallow on a test site must be removed at launch.
  • Remember the limits. It controls crawling, not indexing; see Blocked by robots.txt.

Troubleshooting

SymptomLikely cause (my own diagnosis)Fix
Rules ignoredFile not at the host rootMove it to /robots.txt
Odd characters break rulesWord processor or wrong encodingRe-save as UTF-8 plain text
Rule not matchingCase difference or missing slashMatch case exactly; start paths with /
Subdomain not coveredEach host needs its own fileAdd one per subdomain
Sitemap not foundRelative URL usedUse a fully qualified URL

Checklist

File

  • Named robots.txt
  • At the host root
  • UTF-8 plain text
  • Publicly reachable

Rules

  • User-agent starts each group
  • Paths start with /
  • Case matches
  • No accidental Disallow: /

Sitemaps

  • Full URLs
  • Correct protocol and host
  • Listed once each
  • Files exist

Test

  • Open in a private window
  • Check Search Console report
  • Test key URLs
  • Recheck after changes

What Google’s page does not say

The page I used does not state a file size limit, does not explain the $ character and does not spell out how conflicting Allow and Disallow rules are ranked within a group. For those details consult Google’s full robots.txt specification rather than relying on guesses.

Key terms

User-agent
The name a crawler identifies itself with.
Group
A set of rules that applies to named crawlers.
Wildcard
The * character matching any run of characters.
Root
The top level of a host, such as example.com/.

Common mistakes

mistake 1

Disallow: / by accident

Blocks the whole site.

Read the file after every edit.
mistake 2

Wrong location

Ignored outside the root.

Place at /robots.txt.
mistake 3

Ignoring case

Paths are case-sensitive.

Copy the real path.
mistake 4

Relative sitemap URLs

Must be fully qualified.

Use https://…
mistake 5

Using it to hide pages

It does not deindex.

Use noindex.
mistake 6

No testing

Errors go live.

Test first.

Related guides

faq

robots.txt questions

Where does robots.txt go?

At the root of the host it governs, named robots.txt.

Can I have more than one file?

One per site, though each subdomain or non-standard port can have its own.

Which encoding should I use?

UTF-8, saved from a plain text editor.

Are paths case-sensitive?

Yes.

What does User-agent: * mean?

All crawlers except AdsBot crawlers, which must be named explicitly.

Which group does a crawler use?

The first, most specific group matching its user agent.

What is crawlable by default?

Anything not blocked by a Disallow rule.

How do I test it?

Use the robots.txt report in Search Console or Google’s open-source library.

sources

File rules, the four supported rules, group processing, wildcard support and testing options come from Google’s page. The copy-paste examples marked as mine, the step-by-step routine, WordPress notes, troubleshooting table and checklists are my own.

Scroll to Top