{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiewfwnjiekdfz4z3ehgi7tltzhscr7pogft3f6us2bzrrzfbmz2kq",
    "uri": "at://did:plc:hqad6xwuzg7oqfmwylfkvqfm/app.bsky.feed.post/3mn7vpm3yoo52"
  },
  "path": "/viewtopic.php?t=33349&p=275047#p275047",
  "publishedAt": "2026-06-01T08:53:35.000Z",
  "site": "http://forum.palemoon.org",
  "textContent": "The mechanics of scraping are also considerably different and exponentially more heavy on the server because many, many copies of the same data is requested because scrapers are not smart enough to not follow links leading to the same data from a different URL.\n\ne.g. on the repo with tagged issues, a scraper will dutifully scrape the same issues under each tag for as many tags are attached to the issue. Same with versions. Same with any other dynamically generated data based on filter criteria. One single page can be crawled hundreds or thousands of times because there are as many different ways to land on it.\nAny \"AI summary\" reducing load by people being satisfied with the (potentially flawed) primer of the target page is **_insignificant_** compared to the relentless requesting of every which way one could reach specific content ballooning the total number of requests for that content.\n\n* * *",
  "title": "General Discussion • Re: wtf? almost every forum and website is on cloudflare now?",
  "updatedAt": "2026-06-01T08:53:35.000Z"
}