{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreifwjlcxbufxxabcug6jubpzrc4rzcs2cpvpbex4l4y32mhqnviqsq",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mojozje6chj2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreigmozluretegg4bgd4yu2qtmrwuwtye7hojlkm4jblbwgqoo66vnq"
    },
    "mimeType": "image/webp",
    "size": 72118
  },
  "path": "/tonywangca/14-of-the-web-is-actually-dead-but-not-how-you-think-we-scanned-10m-domains-35g9",
  "publishedAt": "2026-06-18T01:33:38.000Z",
  "site": "https://dev.to",
  "tags": [
    "webdev",
    "webscraping",
    "dns",
    "datascience",
    "crawlora.net",
    "github.com/Crawlora-org/dead-web-index-data",
    "crawlora.net/dead-web-index",
    "crawlora.net/blog/how-much-of-the-web-is-dead-2026"
  ],
  "textContent": "_Originally published at crawlora.net._\n\nWhen you hit a dead URL in production, do you know whether the domain is **gone** — or whether an anti-bot system just blocked your crawler? They look identical from a failed request, but they're completely different failures, and most tools don't tell them apart.\n\nWe scanned the **DomCop top 10 million domains** to find out how much of the popular web is actually dead. The short version: about **14%** — not the ~27% you've probably seen quoted.\n\n##  Dead and blocked are not the same failure\n\nA domain that won't load failed for one of two reasons:\n\n  * **It's gone.** No DNS record, or nothing accepts a TCP connection. Genuinely dead.\n  * **It's alive and blocking you.** A real server returning a 403 or 429 to anything that looks like a bot.\n\n\n\nMost \"dead web\" studies count both as dead. They shouldn't, because the right response to each is opposite:\n\n  * A **dead** domain never comes back. Retrying it — rotating proxies, escalating clients — is wasted compute.\n  * A **blocked** domain is live. It needs a different _client_ , not more retries.\n\n\n\n##  The numbers\n\nProbing every domain over HTTP and classifying each as alive / redirect / blocked / dead:\n\n  * **14.1% genuinely dead** — overwhelmingly vanished DNS (76% of the dead bucket). The server is gone.\n  * **8.9% blocked** — live servers returning 403/429 to automated clients.\n  * **76.6% alive** , 0.3% redirect.\n\n\n\nThe widely-cited \"~27% of the web has rotted\" figure conflates blocked-but-live servers (and 404/5xx responses — still a live server answering) with the genuinely gone. Separate them honestly and the truly-dead web is about half what people assume.\n\n##  Proof: same domains, different client\n\nTo show the 8.9% \"blocked\" really are alive, we re-probed them with a real **Chrome TLS/JA3 fingerprint** — an HTTP client that speaks Chrome's exact TLS handshake and header order (not a headless browser, no canvas/WebGL).\n\n**~72,000** of the blocked domains served content normally. Same URLs, same network — the only thing \"dead\" was the wall. That dropped the blocked rate from 8.9% to 8.2%.\n\nThe takeaway for anyone building crawlers or link-checkers: when a tool reports a dead domain, ~9% of the time it's a live server with anti-bot deployed. `NXDOMAIN`/`REFUSED` → dead, skip it. `403`/`429` → alive, recheck with a real browser TLS context before you mark it dead.\n\n##  The web rots unevenly\n\nDeath rate isn't uniform. By country-code TLD:\n\n  * China's `.cn`: **33%** dead\n  * Germany's `.de`: **7.6%** dead\n\n\n\nA 4× gap. Institutional TLDs fare badly too — `.gov` 26%, `.edu` 22% — matching Pew Research's finding that government and reference pages suffer the worst link rot.\n\n##  The famous dead\n\nThe casualties are all in the data: Grooveshark, Gfycat, del.icio.us, Yahoo Pipes, AddThis, DMOZ, OpenSolaris, GeoCities. Two decades of the social and developer web's graveyard.\n\n##  The open dataset\n\nEvery domain, both probe arms, is open under **CC BY 4.0** (one JSON row per domain per arm: `domain`, `tld`, `rank`, `mode`, `outcome`, `reason`, HTTP statuses, redirect hops, parked flag):\n\n  * **Dataset:** github.com/Crawlora-org/dead-web-index-data\n  * **Interactive explorer** (look up any domain, browse the country map): crawlora.net/dead-web-index\n  * **Full write-up + methodology:** crawlora.net/blog/how-much-of-the-web-is-dead-2026\n\n\n\n_(Disclosure: we build a web-scraping API, which is why the dead-vs-blocked distinction bites us daily.)_",
  "title": "14% of the Web Is Actually Dead — But Not How You Think (We Scanned 10M Domains)"
}