🤖 Robots.txt Tester

Free online robots.txt tester. Fetch and analyze robots.txt from any website, test if a URL is allowed for Googlebot, Bingbot, and other crawlers.

Fetch and analyze robots.txt, test if a URL is allowed for crawling