Robots.txt Checker
Check your robots.txt file — analyze Allow/Disallow rules, Sitemap and User-agents
Working...
What does robots.txt do for SEO?
The robots.txt file tells search engines which pages they may crawl. A misconfiguration can accidentally block your whole site from Google. The tool fetches and parses every rule.
The robots.txt file is tiny but devastating: a single misplaced "Disallow: /" line removes your entire website from Google. It is an extremely common mistake after moving a site from staging to production without removing the block. The Robots.txt Checker fetches your robots.txt and parses every rule: it lists User-agent groups, counts Allow/Disallow lines, flags which paths are blocked from crawling, and checks whether you declared a Sitemap in the file (a best practice that helps Google find your sitemap faster). The raw content is also shown for direct review. Run this after every deploy to make sure nothing is accidentally blocked.
- Counts Allow/Disallow rules and User-agents
- Detects Sitemap declarations in robots.txt
- Shows the raw content for review
How to use in 3 steps
- 1Enter your website URL (the tool finds /robots.txt automatically).
- 2Click "Check robots.txt" to fetch and parse the file.
- 3Review the Allow/Disallow rules, User-agents and Sitemap declaration.