Robots.txt Generator
Build a robots.txt file that tells search engines what to crawl and what to skip. Remember: this only asks crawlers nicely — it doesn't hide pages from people who have the link.
Where this file goes
Save the result as robots.txt and upload it to the root of your domain — it must be reachable at yourdomain.com/robots.txt, not inside a subfolder. Most CMS platforms (WordPress, Shopify, Webflow) have a settings page for this instead of raw file upload.
How to use the Robots.txt Generator
- Choose whether crawlers are allowed by default.
- List any folders to block and add your sitemap URL.
- Upload the result as /robots.txt at the root of your domain.
Frequently asked questions
Does robots.txt hide a page from Google?
No. It asks crawlers not to fetch it, but a blocked URL can still appear in results if other sites link to it. Use a noindex tag to keep a page out of search.
Where does the file go?
At the root of your domain, e.g. example.com/robots.txt. It is ignored inside a subfolder.
Should I use crawl-delay?
Rarely. Google ignores it, and most sites never need it.