robots.txt Syntax and Rules
A robots.txt file is short, but a small syntax mistake can block your whole site. This guide explains where the file goes, every rule Google supports, how groups are processed, wildcards, copy-paste examples, and how to test before you upload.
- File location rules
- User-agent, Disallow, Allow, Sitemap
- Wildcards and groups
- Test and submit
User-agent: * Disallow: /private/ Allow: /private/public/ Sitemap: https://example.com/sitemap.xml
Name the file robots.txt, put it at the root of the host, save it as UTF-8 plain text, and write groups that start with User-agent followed by Disallow and Allow lines, one per line. Add Sitemap lines with full URLs. Anything you do not disallow is crawlable by default. Test it in Search Console before relying on it.
File basics
| Rule | What Google says |
|---|---|
| Name | robots.txt, exactly |
| Location | The root of the host it governs |
| Scope | Applies only to the protocol, host and port where it is posted; subdomains and non-standard ports can have their own |
| Quantity | One file per site |
| Encoding | UTF-8; use a plain text editor because word processors can add odd characters such as curly quotes |
For the overall picture of how robots.txt and sitemaps fit together, see robots.txt and sitemaps.
The four rules Google supports
| Rule | What it does | Format notes |
|---|---|---|
| User-agent | Names the crawler a group applies to; * means all crawlers except AdsBot crawlers, which must be named | Starts each group |
| Disallow | A path the crawler should not crawl | Path starts with /; directory paths end with / |
| Allow | Overrides a Disallow to permit a subfolder or page | Same path format |
| Sitemap | Fully qualified URL of a sitemap; zero or more allowed | Google does not assume http/https or www variants |
Other details: paths are case-sensitive, a line starting with # is a comment, and unrecognised lines are ignored.
How groups are processed
Wildcards
Google says all rules except Sitemap support the * wildcard, for prefixes, suffixes or whole strings.
Disallow: /*.pdf # paths containing .pdf (illustrative) Disallow: /search* # paths starting with /search
The page I used does not describe the $ end-of-string character, so I do not explain it here. Check Google’s full robots.txt specification if you need it.
Copy-paste examples
User-agent: * Disallow:
User-agent: * Disallow: /private/
User-agent: Googlebot Disallow: /
Sitemap: https://example.com/sitemap.xml
The first example (an empty Disallow, meaning nothing blocked) is my own illustration. The Googlebot-only block mirrors an example in Google’s documentation.
Create and upload it step by step
Open a plain text editor
Create a file named robots.txt, saved as UTF-8.
Add your groups
User-agent first, then Disallow and Allow lines, one per line.
Add Sitemap lines
Use the full URL of each sitemap.
Put it at the root
The method depends on your hosting; ask your host if unsure.
Open it in a private window
This confirms it is publicly reachable.
Use the Search Console report
Verify Google can read and interpret it.
Testing and getting Google to refresh
Google gives two testing options: the robots.txt report in Search Console, which only works for files already live on your site, and Google’s open-source robots.txt library for local testing by developers. Google’s crawlers find and use the file automatically once it is uploaded; if you need a faster refresh of its cached copy, Google has a separate guide on submitting an updated file.
Notes for WordPress sites
- It may be virtual. WordPress and SEO plugins can generate robots.txt without a physical file (my own note). See Yoast SEO settings.
- Do not block what pages need. Blocking theme or plugin assets can hinder rendering (my own advice).
- Staging. A blanket Disallow on a test site must be removed at launch.
- Remember the limits. It controls crawling, not indexing; see Blocked by robots.txt.
Troubleshooting
| Symptom | Likely cause (my own diagnosis) | Fix |
|---|---|---|
| Rules ignored | File not at the host root | Move it to /robots.txt |
| Odd characters break rules | Word processor or wrong encoding | Re-save as UTF-8 plain text |
| Rule not matching | Case difference or missing slash | Match case exactly; start paths with / |
| Subdomain not covered | Each host needs its own file | Add one per subdomain |
| Sitemap not found | Relative URL used | Use a fully qualified URL |
Checklist
File
- Named robots.txt
- At the host root
- UTF-8 plain text
- Publicly reachable
Rules
- User-agent starts each group
- Paths start with /
- Case matches
- No accidental Disallow: /
Sitemaps
- Full URLs
- Correct protocol and host
- Listed once each
- Files exist
Test
- Open in a private window
- Check Search Console report
- Test key URLs
- Recheck after changes
What Google’s page does not say
The page I used does not state a file size limit, does not explain the $ character and does not spell out how conflicting Allow and Disallow rules are ranked within a group. For those details consult Google’s full robots.txt specification rather than relying on guesses.
Key terms
- User-agent
- The name a crawler identifies itself with.
- Group
- A set of rules that applies to named crawlers.
- Wildcard
- The * character matching any run of characters.
- Root
- The top level of a host, such as example.com/.
Common mistakes
Disallow: / by accident
Blocks the whole site.
Wrong location
Ignored outside the root.
Ignoring case
Paths are case-sensitive.
Relative sitemap URLs
Must be fully qualified.
No testing
Errors go live.
Related guides
robots.txt questions
Where does robots.txt go?
At the root of the host it governs, named robots.txt.
Can I have more than one file?
One per site, though each subdomain or non-standard port can have its own.
Which encoding should I use?
UTF-8, saved from a plain text editor.
Are paths case-sensitive?
Yes.
What does User-agent: * mean?
All crawlers except AdsBot crawlers, which must be named explicitly.
Which group does a crawler use?
The first, most specific group matching its user agent.
What is crawlable by default?
Anything not blocked by a Disallow rule.
How do I test it?
Use the robots.txt report in Search Console or Google’s open-source library.
File rules, the four supported rules, group processing, wildcard support and testing options come from Google’s page. The copy-paste examples marked as mine, the step-by-step routine, WordPress notes, troubleshooting table and checklists are my own.