Crawler Policy
Which AI crawlers does a site actually allow?
Paste a domain. This reads its robots.txt and works out, crawler by crawler, what it permits — following the same precedence rules a crawler does, rather than searching the file for a name.