Robots.txt & AI Crawler Checker
dev.workers.quietworklabs.robots-check/robots-check
Tools
Exactly what the server advertised in the scan completed . Effects and capabilities are classified by Squatch from each tool's metadata. Squatch never runs tools.
- Effect: Read 3
Effect
Capability
Showing 3 of 3 tools
| Details | ||
|---|---|---|
| ai_crawler_access AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Cla… | Effect: Read | |
| robots_check Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309… | Effect: Read | |
| sitemap_check Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entr… | Effect: Read |