What Does The Test Robots.Txt Tool Do?
The Robots.txt Validator allows you to check if your robots.txt file is properly permitting or blocking web crawlers from indexing your site, or specific URLs. Enter your robot.txt file URL to get accurate results (i.e: http//:mysite.com/robots.txt).
What Is A Robots.Txt File?
Robots.txt is a file on your domain (website.com/robots.txt) that indicates whether certain web-crawling software or bots can or cannot crawl the different pages of your website. Crawl instructions are specified by “disallowing” or “allowing” pages of your site to be crawled by the different web crawlers.