Robots.txt Generator
Build a robots.txt file that tells search engines what to crawl and what to skip. Remember: this only asks crawlers nicely — it doesn't hide pages from people who have the link.
Where this file goes
Want a page out of Google’s results? robots.txt is the wrong tool for that: read robots.txt vs noindex: which one keeps a page out of Google?
Save the result as robots.txt and upload it to the root of your domain — it must be reachable at yourdomain.com/robots.txt, not inside a subfolder. Most CMS platforms (WordPress, Shopify, Webflow) have a settings page for this instead of raw file upload.
How to use the Robots.txt Generator: step by step
- Choose the default rule: allow all crawlers (most sites) or block all (site not ready).
- List folders to block, one per line, like
/wp-admin/. - Add your sitemap URL.
- Copy the result into a file named
robots.txtin your site’s root folder.
A real example
A WordPress shop wants Google to skip its cart and checkout pages. Allow all, block /cart/ and /checkout/, add the sitemap, and upload the file to example.com/robots.txt.
Tips for better results
- robots.txt asks crawlers not to visit pages; it doesn’t hide them from people or guarantee they stay out of Google.
- To keep a page out of search results, use a
noindextag instead. - Never block CSS or JavaScript files Google needs to render your pages.
More questions
Can I block AI crawlers?
Yes. Add a block for their user-agent by hand, for example User-agent: GPTBot followed by Disallow: /. Our guide to GPTBot explains the trade-offs.
Frequently asked questions
Does robots.txt hide a page from Google?
No. It asks crawlers not to fetch it, but a blocked URL can still appear in results if other sites link to it. Use a noindex tag to keep a page out of search.
Where does the file go?
At the root of your domain, e.g. example.com/robots.txt. It is ignored inside a subfolder.
Should I use crawl-delay?
Rarely. Google ignores it, and most sites never need it.