Browser Support • Re: Cloudflare Verification Loop issues

AI scrapers don't honor robots.txt at all.

I mean they should... But yeah, no shit most don't. I said as much in my post.

UA filtering is pointless as they go out of their way to use a smattering of Chrome UAs when scraping, and do not identify as bots in the UA at all.

Are we on different Internets? Or are you just making assumptions? Have you tried? Like I said it was night and day for my forum. The bot traffic problem went below the leve of noise. Please reconsider.

I think you have missidentified the source of the bulk of problematic bot traffic. The sources are "legitimate" outfits using predictable UAs. Tell them to pound sand and your woes will go away.

(The sources being "legit" is also why I give them a chance to do the right thing with robots.txt.)