General Discussion • Re: wtf? almost every forum and website is on cloudflare now?

There are real differences. If someone visits a site himself, he actually sees whatever the webmaster has written and wants to share with the world. A bot crawling the site cannot do this. It is a soulless, automated request which copies the page into a database which may or may not ever actually be read. For small servers especially, with limited bandwidth*, a webmaster might wish to reserve his traffic for real people.

Again, different people have different purposes. Indexing and archival are generally regarded as benign here, although there might be reasons somebody wants his site unlisted, and a general claim like ‘search engines are beneficial for websites’ does not trump these specific purposes. Privacy might be one reason (cf the right to be forgotten), but one is not required to justify one’s wish to be excluded to strangers, be they search engine proprietors or others. But I think the strongest objection is that against scraping one’s site ‘to train AI’. Many people, I among them, are categorically opposed to this technology’s existence and abstain completely from it.† We regard any use of our words or webpages for LLM or MMM as abuse, an offence upon our moral rights, and bots intending that malicious. It does not matter one jot if many people use it.

*Bandwidth shortages still happen today. Just last week, an independent TRPG site which has existed for over twenty years was down because the webmaster had run out of bandwidth for the month. †In our personal lives, at least. This statement is without prejudice to somebody who hates the technology, but whose work obligates him to use it during his shift.