Robots.txt Checker
Validate your robots.txt file and test if specific URLs are blocked or allowed for different crawlers.
Validation Complete!
0User-agents
0Allow Rules
0Disallow Rules
0Sitemaps
Syntax Errors
Warnings
Errors
Summary
Sitemaps Declared
Crawl Delays
Parsed Rules
Test URLs
Test Results
| URL | Status | Matching Rule |
|---|
Optimizing for search engines?
Add a Boei AI chatbot trained on your site to convert more of that organic traffic.
Create AI ChatbotRaw Content
What We Validate
Syntax Validation
Check for proper directive format and colon usageUser-agent Rules
Verify Allow and Disallow rules for each user-agentSitemap Declarations
Find and validate sitemap URLsCrawl-delay
Detect crawl-delay directives and warn about high valuesCommon Mistakes
Identify duplicate user-agents and missing wildcardsURL Testing
Test specific URLs against rules for different botsFrequently Asked Questions
What is robots.txt?
+
robots.txt is a file that tells search engine crawlers which pages they can or cannot access on your site. It's placed in your website's root directory (example.com/robots.txt) and helps manage crawler traffic.
Does robots.txt block pages from search results?
+
No, robots.txt only prevents crawling, not indexing. Pages can still appear in search results if other sites link to them. To truly block pages from search results, use the noindex meta tag or X-Robots-Tag header.
What is the wildcard user-agent (*)?
+
The wildcard (*) user-agent applies to all crawlers that don't have specific rules. It's recommended to always include a * section as a fallback for unknown or new crawlers.
How do Allow and Disallow interact?
+
When both Allow and Disallow rules match a URL, the more specific (longer) rule takes precedence. If they're equal length, Allow typically wins. This varies slightly between search engines.
Should I block AI crawlers like GPTBot?
+
It depends on your preference. If you don't want your content used to train AI models, you can block bots like GPTBot (OpenAI), CCBot (Common Crawl), and others. However, this won't affect AI assistants that have already been trained on your content.
Turn more website visitors into qualified leads
Answer questions instantly, capture intent, and hand every qualified conversation to your team.
Start free, no credit card required.
Start free, no credit card required.
Trusted by 17,000+ businesses
Quick 5-min, no code setup
from 159 reviews