Browser Support • Re: Cloudflare Verification Loop issues

robots.txt

AI scrapers don't honor robots.txt at all.

UA filtering is pointless as they go out of their way to use a smattering of Chrome UAs when scraping, and do not identify as bots in the UA at all.

Example of a scraper UA:

CODE:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36