๐Ÿค– Robots.txt Validator

CN EN
Updated: 2026-07-11

๐Ÿ“ Paste your robots.txt content

๐Ÿ“– Robots.txt Specification

โ€ข User-agent: Specifies which crawler the rules apply to (* = all crawlers)

โ€ข Disallow: Paths the crawler should not access

โ€ข Allow: Paths the crawler can access (can override Disallow)

โ€ข Sitemap: Declares the sitemap URL for crawlers

โ€ข Crawl-delay: Sets the request interval in seconds

โ€ข File location: Should be at the root of the website (/robots.txt)