Robots.txt Checker and Tester
Test whether robots.txt exists, whether it allows Googlebot to crawl a given path, and whether it declares your sitemap, then read the rules it returns.
What this check catches
robots.txt tells compliant crawlers which paths they may request. Site builders and hosting platforms can ship a default file, serve an HTML error page in its place, or carry a staging rule that blocks the whole site into production.
Why it matters before launch
A disallow rule stops compliant crawlers from requesting a page at all, so the page cannot be read or evaluated. Blocking is sometimes intended for private or staging paths, so confirm the intent before removing any rule.
Common warning signs
- /robots.txt returns 404, or returns an HTML page instead of plain text
- A wildcard Disallow: / rule survived from staging
- Googlebot is blocked while other crawlers are allowed
- No Sitemap: directive points crawlers at the sitemap
How to inspect it
- Request /robots.txt on the preferred HTTPS hostname and read the raw response
- Check the user-agent groups and the order of Allow and Disallow rules
- Test the exact paths you care about, not only the homepage
- Look for a Sitemap: line with an absolute URL
How to fix it
- Serve /robots.txt as plain text from the production hostname
- Remove staging disallow rules that were not meant to ship
- Keep intentional restrictions for private, admin, or preview paths
- Add a Sitemap: line with the absolute sitemap URL
Verify after the change
Request the file directly after deploying, re-test the paths that matter, and confirm the Sitemap line resolves. Search Console reports what Google itself fetched, which is the authoritative view.
Scan the live URL again →