General Discussion • Re: wtf? almost every forum and website is on cloudflare now?

There are real differences. If someone visits a site himself, he actually sees whatever the webmaster has written and wants to share with the world. A bot crawling the site cannot do this. It is a soulless, automated request which copies the page into a database which may or may not ever actually be read. For small servers especially, with limited bandwidth*, a webmaster might wish to reserve his traffic for real people.

A dubious argument. For example, if I'm looking for something specific, I don't read the entire page; I quickly scan it, intuitively finding what I need (sometimes just one post), and then skip the rest. If I find a book that covers a specific point I need, I read that specific point and save the entire book to my hard drive. It's not a given that I'll read it all later. I don't fully read everything I open in my browser, either. If I need to dig a hole, I grab a shovel, but there's no soul in it... Yes, I could dig by hand, but a shovel is faster and easier. So, does that mean I'm a bot? Even increased traffic is a weak argument these days. Traffic is free almost everywhere, as far as I know. Server load - yes, that's an argument. It simply means limiting the frequency of user requests, whether it's a human or a program configured by a human.

Again, different people have different purposes.

There's no doubt about it. Occasionally, very unusual situations occur.

But I think the strongest objection is that against scraping one’s site ‘to train AI’. Many people, I among them, are categorically opposed to this technology’s existence and abstain completely from it.

This is truly an unusual technology. You just have to be critical of the results AI produces (as you should be of everything you find online). But I wouldn't draw any esoteric or religious implications here. Essentially, AI is a sophisticated search engine masquerading as a sentient being.