{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreigekv2uthcws2cg6vm3ke5t63z27eno4dknkizcjery2ogric3jaa",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mpi475gqppq2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreicdcsc2uo5jciqelv7bdxjkguqfxragv32h5gd56kyecd2m3blt6e"
    },
    "mimeType": "image/webp",
    "size": 432946
  },
  "path": "/mrinal_narang_13a3d00eb37/two-kubernetes-decisions-nobody-writes-about-honestly-29fb",
  "publishedAt": "2026-06-30T03:14:00.000Z",
  "site": "https://dev.to",
  "textContent": "##  1. Node Group Sizing\n\nFewer large nodes vs. many small ones. Textbooks don't cover this.\n\nWe ran 10 nodes, 32 CPU each. Seemed efficient.\n\nProblem: One node dies, 320 CPU worth of workloads need to reschedule. Cluster autoscaler couldn't handle it. Pods sat pending for 10 minutes.\n\nWe switched to 20 nodes, 16 CPU each. Same total capacity.\n\nOne node dies now? 160 CPU to reschedule. Autoscaler catches it in 90 seconds. Scheduling is tighter, but failures are isolated.\n\nCost stayed the same. Blast radius halved.\n\nWhy nobody writes about this: The tradeoff isn't obvious. Large nodes are \"more efficient.\" Smaller nodes are \"more resilient.\" Both are true. It depends on whether you'd rather have one big problem or many small ones.\n\nWe picked smaller nodes because a node failure was our actual failure mode. Not resource efficiency.\n\n##  2. Readiness vs Liveness Probes\n\nMisconfigure these and your cluster looks like it's melting.\n\nReadiness probe: \"Can this pod take traffic?\"\n\nLiveness probe: \"Is this pod alive? Restart it if not.\"\n\nOne team set readiness = liveness. Same probe checked both.\n\nProbe logic: \"If I can reach the database, I'm ready.\"\n\nDatabase gets slow. Probe fails. Pod becomes \"not ready.\" Load balancer removes it from rotation (correct).\n\nBut liveness also failed. Kubernetes killed the pod and restarted it.\n\nNew pod starts. Probe fails immediately (database still slow). Gets killed. Restarted.\n\nThis cascaded across 30 pods. 30 restarts/minute. New pods spent 100% of time restarting.\n\nLooks like an application bug for the first 20 minutes. Actually a probe configuration bug.\n\nFix: Readiness checks \"can I take traffic right now?\" Liveness checks \"am I fundamentally broken?\" Use separate probes.\n\nReadiness: Check database connection with short timeout. Fail if slow. This is reasonable - don't send traffic to slow pods.\n\nLiveness: Check if the process is responding at all. Much stricter threshold. Only kill if truly hung.\n\nSame database slowness? Pods become unready. Traffic reroutes. Cluster stabilizes. No restart loop.\n\n##  The Connection\n\nBoth decisions have hidden failure modes that surface under stress.\n\nNode sizing looks good until a node fails and you realize your blast radius is too large.\n\nProbe configuration looks good until the database gets slow and your whole cluster starts thrashing.\n\nNeither is \"wrong.\" Both have tradeoffs. The difference is understanding the failure mode you're actually optimizing for.\n\nLarge nodes are efficient until they aren't.\n\nReadiness = Liveness is simple until cascading restarts make it obvious.\n\nIf you're running EKS with 8+ large nodes, consider downsizing and multiplying. If you've never hit a restart loop from probe misconfiguration, you will eventually.\n\nWhen you do, check if readiness and liveness are the same. Usually they are.\n\n#  Kubernetes #EKS #DevOps #SRE #IncidentResponse",
  "title": "Two Kubernetes Decisions Nobody Writes About Honestly"
}