What Is Robots.txt and How to Use It Correctly
Robots.txt tells search engine crawlers which parts of your site they can access. A small mistake in it can hide your whole site from Google. Here's how it works and how to use it safely.
Key takeaways
- Robots.txt controls crawling, not indexing. A blocked page can still appear in Google if other pages link to it.
- To keep a page out of Google, use a noindex tag and let Google crawl it.
- Never block CSS or JavaScript files that Google needs to render your pages.
- You can use robots.txt to set rules for AI crawlers such as GPTBot or Google-Extended.
Robots.txt is a plain text file at the root of your website (yourdomain.com/robots.txt) that tells search engine crawlers which URLs they may and may not request. It’s one of the simplest files on your site and one of the easiest to get dangerously wrong.
A basic example
“` User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /cart/ Disallow: /checkout/
Sitemap: https://www.example.com/sitemap.xml “`
- User-agent: which crawler the rules apply to (
*means all) - Disallow: paths crawlers shouldn’t request
- Allow: exceptions inside a disallowed path
- Sitemap: where to find your XML sitemap
Robots.txt controls crawling, not indexing
This is the most misunderstood point. Blocking a page in robots.txt stops Google crawling it, but if other pages link to it, Google can still index the URL, often showing it with no description.
| Goal | Use |
|---|---|
| Keep a page out of Google’s results | A noindex robots meta tag, with the page crawlable |
| Stop crawlers wasting time on endless URLs (filters, internal search) | Robots.txt Disallow |
| Protect private content | Password protection (robots.txt is public) |
Good uses of robots.txt
- Blocking internal search result pages and faceted filter combinations that create endless URLs (saves crawl budget)
- Blocking cart, checkout and account pages
- Pointing to your sitemap
- Setting rules for specific bots, including AI crawlers
AI crawlers
Robots.txt is also how you tell AI companies’ crawlers what they may access. For example:
“` User-agent: GPTBot Disallow: /private-research/
User-agent: Google-Extended Disallow: / “`
Google-Extended controls whether your content can be used for Google’s Gemini models; it doesn’t affect Google Search ranking. Think carefully before blocking AI crawlers: being cited in AI answers can bring visibility. See AEO explained.
Common mistakes
Disallow: /left over from a staging site, which blocks the entire site. This happens surprisingly often after launches.- Blocking CSS and JavaScript, which stops Google rendering pages properly
- Using robots.txt to hide pages from Google instead of noindex
- Blocking pages that have noindex tags: Google can’t see the noindex if it can’t crawl the page
- Typos and wrong capitalisation: paths are case-sensitive
How to test your robots.txt
- Visit
yourdomain.com/robots.txtand read it - In Search Console, check the robots.txt report (under Settings) for errors and the last fetched version
- Use URL Inspection on important pages to confirm they aren’t blocked
Robots.txt is covered in our technical SEO audit.