RobotsGate
dev.workers.mike-tusa.robotsgate/robotsgate
Tools
Exactly what the server advertised in the scan completed . Effects and capabilities are classified by Squatch from each tool's metadata. Squatch never runs tools.
- Effect: Read 4
Showing 4 of 4 tools
| Details | ||
|---|---|---|
| check_site Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes;… | Effect: Read | |
| generate_robots Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token … | Effect: Read | |
| list_crawlers Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or othe… | Effect: Read | |
| validate_robots Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or bloc… | Effect: Read |