{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreicui2xzujgvnjgzn5kvlh76bfzkefvbqpwkqcb62ein3j7pdxeleq",
    "uri": "at://did:plc:pi6woz4d47bkuws673w2il2r/app.bsky.feed.post/3micauj2s7zx2"
  },
  "path": "/t/availability-of-hoogle-haskell-org-etc/13858#post_5",
  "publishedAt": "2026-03-30T13:58:53.000Z",
  "site": "https://discourse.haskell.org",
  "tags": [
    "hoogle.haskell.org"
  ],
  "textContent": "andreasabel:\n\n> Or do we have to accept this as the new status quo?\n\nHi. The main issue here is that the web is plagued by a sheer absurd amount of AI crawlers that are unbelievably disrespectful, hitting expensive URLs on the server that runs the hoogle.haskell.org.\n\nThere are certainly measures that would improve this but they’re usually quite drastic. We could e.g. do tarpitting, do aggressive rate limiting etc. I think the general consensus across the haskell org infrastructure team has been that these more aggressive strategies are not acceptable as they would impair fair use in the same way as the abusers.\n\nHowever, given that the user experience using haskell services has been drastically degraded since the onset of the AI crawler epidemic, I think it might be worth to reevaluate that policy. I will make sure to ask about this topic when the infrastructure team meets next.\n\nIn the meantime, I also host a hoogle under hoogle dot mangoiv dot com which employs some of the more aggressive strategies to stay fast. You are free to use it.",
  "title": "Availability of hoogle.haskell.org etc"
}