{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreihio7qe35zge3m7dujod6mhmimxfqcaazo5ubfli3mexhnjxhlfqq",
    "uri": "at://did:plc:pi6woz4d47bkuws673w2il2r/app.bsky.feed.post/3mj72w2sg63i2"
  },
  "path": "/t/incident-certs-on-most-haskell-org-domains-are-expired/13912#post_3",
  "publishedAt": "2026-04-10T09:28:26.000Z",
  "site": "https://discourse.haskell.org",
  "textContent": "The roots of this issue lay in the mists of time.\n\nFastly has sponsored CDN and TLS services for Haskell domains for a very long time. Some of our configuration predates ACME wildcard domains, if not ACME itself. Given all the legacy stuff, it’s hard to know why everything is the way it is.\n\nRecently, some changes pushed the problem over the line. ACME config on the servers, DNS zone config in Cloudflare (another sponsor), and TLS config in Fastly conflicted in a way that broke the ACME update.\n\nThe fix involved removing old conflicting certs in Fastly, removing or correcting DNS entries in Cloudflare, and rerunning the ACME update. This cleaned up other potential lurking issues, as well.\n\nIt looks like there may have been migrations done in the past that were partially incomplete. We believe—well, I believe in particular—a better focus on understanding systems, planning changes, and following up on them would have prevented this. But the real failure here was the lack of alerting. The cert tried to tell us for a whole month that something was wrong, but we never heard it.\n\nThe Infra team has been getting more organized recently. The Haskell Foundation has devoted some resources to that work. Getting a team alerting system set up is high on our wishlist.",
  "title": "Incident: Certs on most haskell.org domains are expired"
}