SeoCheckup
Free Robots.txt Checker
Fetch any public robots.txt and highlight User-agent, Allow, Disallow, and Sitemap lines.
What you get
- Load robots.txt from any public site URL
- Highlight crawl directives so rules are easy to scan
- Spot Sitemap lines declared for search engines
FAQ
- Where does robots.txt live?
- It must be at the site root, for example https://example.com/robots.txt. Search engines look there first before crawling.
- What does the Sitemap directive do?
- A Sitemap line in robots.txt tells crawlers where to find your XML sitemap or sitemap index, so they can discover URLs faster.
- Does Disallow stop a page from being indexed?
- Disallow asks crawlers not to fetch a URL. Indexing can still happen from links elsewhere. Use noindex meta or headers when you need to keep a page out of results.