Robots.txt Checker

Validate your robots.txt file and test if specific URLs are blocked or allowed for different crawlers.

We'll fetch /robots.txt automatically

Validation Complete!

0
User-agents
0
Allow Rules
0
Disallow Rules
0
Sitemaps
Syntax Errors
    Warnings
      Errors
        Summary
          Sitemaps Declared
            Crawl Delays
              Parsed Rules
              Test URLs
              Test Results
              URL Status Matching Rule
              Optimizing for search engines?

              Add a Boei AI chatbot trained on your site to convert more of that organic traffic.

              Create AI Chatbot
              Raw Content
              
                              

              What We Validate

              Syntax Validation
              Check for proper directive format and colon usage
              User-agent Rules
              Verify Allow and Disallow rules for each user-agent
              Sitemap Declarations
              Find and validate sitemap URLs
              Crawl-delay
              Detect crawl-delay directives and warn about high values
              Common Mistakes
              Identify duplicate user-agents and missing wildcards
              URL Testing
              Test specific URLs against rules for different bots

              Frequently Asked Questions

              What is robots.txt?

              robots.txt is a file that tells search engine crawlers which pages they can or cannot access on your site. It's placed in your website's root directory (example.com/robots.txt) and helps manage crawler traffic.

              Does robots.txt block pages from search results?

              No, robots.txt only prevents crawling, not indexing. Pages can still appear in search results if other sites link to them. To truly block pages from search results, use the noindex meta tag or X-Robots-Tag header.

              What is the wildcard user-agent (*)?

              The wildcard (*) user-agent applies to all crawlers that don't have specific rules. It's recommended to always include a * section as a fallback for unknown or new crawlers.

              How do Allow and Disallow interact?

              When both Allow and Disallow rules match a URL, the more specific (longer) rule takes precedence. If they're equal length, Allow typically wins. This varies slightly between search engines.

              Should I block AI crawlers like GPTBot?

              It depends on your preference. If you don't want your content used to train AI models, you can block bots like GPTBot (OpenAI), CCBot (Common Crawl), and others. However, this won't affect AI assistants that have already been trained on your content.

              Create your first Boei widget today

              Get 30% more conversations and effortlessly convert them into customers.
              Don't wait, experience it for free yourself!

              URL

              https://
              https://

              Trusted by 10,000+ businesses

              Quick 5-min, no code setup

              Andrew Lee David S. Vance W. Grant Nitesh Manav
              from 159 reviews