Tools

Website SEO AuditBroken Link CheckerSchema Markup ValidatorAI Crawler CheckerInternal Link CheckerKeyword Cannibalization CheckerKeyword ClusteringKeyword SuggestionsKeyword GapSEO Content CheckerReadability CheckerKeyword DensityAI HumanizerWriting Pattern IndicatorSearch Console Regex GeneratorLong-Tail Keyword IdeasSERP Snippet PreviewMeta Tag GeneratorURL Slug GeneratorRobots.txt GeneratorMeta Title GeneratorMeta Description GeneratorOpen Graph GeneratorTwitter Card GeneratorHTML Meta TagsMeta Tags for WordPressMeta Tags for ShopifyMeta Tags for BloggerMeta Tags for WebflowMeta Tags for ReactMeta Tags for Next.jsCore Web Vitals CheckerImage CompressorHTML MinifierCSS MinifierJavaScript MinifierVideo & Image SpecsQR Code GeneratorCurrency ConverterScientific CalculatorLoan / EMI CalculatorMortgage CalculatorCompound Interest CalculatorSalary to Hourly CalculatorAge CalculatorDate Difference CalculatorJSON Formatter & ValidatorWord CounterMerge PDFSplit PDFRotate PDFDelete PDF PagesRearrange PDF PagesExtract PDF PagesCrop PDFJPG to PDFPNG to PDFPDF to JPGPDF to PNGResize ImageCrop ImageImage ConverterWebP to JPGHEIC to JPGProfile Photo MakerImage to Text (OCR)MOV to MP4MKV to MP4WebM to MP4AVI to MP4MP4 to MP3Extract Audio from VideoTrim VideoMute VideoVideo to GIFResize VideoCompress VideoM4A to MP3WAV to MP3OGG to MP3AAC to MP3Split CSVSplit ExcelCSV to ExcelExcel to CSVExcel to XMLXML to ExcelXML to CSVXML to JSONJSON to CSV
On-page SEOFree, no sign-upRuns in your browser

Robots.txt Generator

Build a robots.txt file that tells search engines what to crawl and what to skip. Remember: this only asks crawlers nicely — it doesn't hide pages from people who have the link.

Where this file goes

Want a page out of Google’s results? robots.txt is the wrong tool for that: read robots.txt vs noindex: which one keeps a page out of Google?

Save the result as robots.txt and upload it to the root of your domain — it must be reachable at yourdomain.com/robots.txt, not inside a subfolder. Most CMS platforms (WordPress, Shopify, Webflow) have a settings page for this instead of raw file upload.

How to use the Robots.txt Generator: step by step

  1. Choose the default rule: allow all crawlers (most sites) or block all (site not ready).
  2. List folders to block, one per line, like /wp-admin/.
  3. Add your sitemap URL.
  4. Copy the result into a file named robots.txt in your site’s root folder.

A real example

A WordPress shop wants Google to skip its cart and checkout pages. Allow all, block /cart/ and /checkout/, add the sitemap, and upload the file to example.com/robots.txt.

Tips for better results

  • robots.txt asks crawlers not to visit pages; it doesn’t hide them from people or guarantee they stay out of Google.
  • To keep a page out of search results, use a noindex tag instead.
  • Never block CSS or JavaScript files Google needs to render your pages.

More questions

Can I block AI crawlers?

Yes. Add a block for their user-agent by hand, for example User-agent: GPTBot followed by Disallow: /. Our guide to GPTBot explains the trade-offs.

Frequently asked questions

Does robots.txt hide a page from Google?

No. It asks crawlers not to fetch it, but a blocked URL can still appear in results if other sites link to it. Use a noindex tag to keep a page out of search.

Where does the file go?

At the root of your domain, e.g. example.com/robots.txt. It is ignored inside a subfolder.

Should I use crawl-delay?

Rarely. Google ignores it, and most sites never need it.