General Discussion • Re: wtf? almost every forum and website is on cloudflare now?

A dubious argument. For example, if I'm looking for something specific, I don't read the entire page; I quickly scan it, intuitively finding what I need (sometimes just one post), and then skip the rest…

Some publications are not intended for readers who are merely skimming for information. Indeed, I have found site redesigns catering to such readers (I have British statistical offices in mind as my chief concrete example) quite annoying. When we go beyond reference material where nobody expects the textbook to be read straight through, the fact becomes more apparent. If I write a story and post it on my personal website, I want my readers to pay attention to it undistracted, enter the setting I am describing and fall in love with the characters the way I have.

It is true that not everybody who visits a page is certain to give it the attention it is due, but the chances are far greater when someone has clicked on this link thinking ‘I want to visit this page right now’ and has actually got it on the screen than if a spider, worm or some other vermin is accessing the server without anyone directly involved. Automatic requests might receive or bring personal attention, like when the Internet Archive mirrors a site, or a search engine helps somebody discover a page, but it is far less likely, so a website intended really to be read will naturally wish for most of its traffic to be authentic and organic.

Traffic is free almost everywhere…

In my footnote, I shared a recent counter-example. This is especially a problem for small sites who host everything personally. Moreover, I continued to live under rather strict monthly bandwidth caps well after this ceased to be common, until a few years ago. This has left me with the firm belief that the interests of people with low bandwidth ought always to be regarded, as a matter of accessibility.

It simply means limiting the frequency of user requests, whether it's a human or a program configured by a human.

Webmasters might wish for natural persons to make up all, or almost all, their traffic, regulating bot traffic so it stays within prescribed, known bounds. Many do not want their natural traffic to be crowded out by bots.

This is truly an unusual technology. You just have to be critical of the results AI produces (as you should be of everything you find online). But I wouldn't draw any esoteric or religious implications here. Essentially, AI is a sophisticated search engine masquerading as a sentient being.

I do not feel like turning this thread into yet another debate about the merits, or lack thereof, of LLM as a technology. There is much I could say, but it is beside the point. Suffice it to say that many of us have weighed it in the balance, found it wanting and wish nothing to do with it, such that scraping against our will is unauthorised, hence malicious. Many of us here, I among them, have been harmed in many ways by LLM scraper swarms. Among them are the dilution of legitimate search results with plagiarised gibberish, and most importantly for this thread, the resultant spike in JavaScript-based captchas and especially Cloudflare use.