โข User-agent: Specifies which crawler the rules apply to (* = all crawlers)
โข Disallow: Paths the crawler should not access
โข Allow: Paths the crawler can access (can override Disallow)
โข Sitemap: Declares the sitemap URL for crawlers
โข Crawl-delay: Sets the request interval in seconds
โข File location: Should be at the root of the website (/robots.txt)