Robots.txt generator
Write a robots.txt without the one line that hides your whole site.
Search engines
Block AI crawlers
robots.txt
User-agent: * Disallow: /admin Disallow: /cart Disallow: /checkout Sitemap: https://example.com/sitemap.xml
Save this as robots.txt at the root of your domain. It only works there, not in a subfolder.
Questions
- What does Disallow: / do?
- It asks every crawler to ignore the entire site. That is correct for a staging server and catastrophic on a live one, and it is the most common reason a site disappears from search overnight. It usually arrives by being copied from staging to production.
- Is robots.txt a security measure?
- No, and treating it as one is a mistake. It is a polite request that well behaved crawlers honour and everything else ignores. The file is public, so listing a secret path in it advertises that path to anyone who looks.
- Does it stop a page appearing in Google?
- Not reliably. Blocking a URL stops it being crawled, but it can still be listed if other sites link to it. To keep a page out of results properly, let it be crawled and serve a noindex tag.
- Should I block AI crawlers?
- It depends what you want. Blocking them keeps your content out of training data, and also keeps you out of AI answers, which is increasingly how people find things. The tool offers both without pushing you either way.
- Where does the file go?
- At the root of the domain, so example.com/robots.txt. In a subfolder it does nothing at all.