Robots.txt Generator

Create a robots.txt file for your website. Control which crawlers can access which paths.


        

      


      

        
Robots.txt Syntax

        
The robots.txt file lives at the root of your domain (https://example.com/robots.txt). Each block starts with one or more User-agent lines, followed by Allow or Disallow rules.

        
    
          
  • User-agent: * — applies to all crawlers
    • 
          
  • Disallow: /admin/ — blocks the /admin/ path
    • 
          
  • Allow: / — allows everything (use to override Disallow)
    • 
          
  • Sitemap: — tells crawlers where your sitemap is
    •