{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rednafi.com/go/request-coalescing/",
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreibz4kwi3yba6ehh7omf5irvknstbjgvvq6g72d4hmkpe55hg2mehu"
    },
    "mimeType": "image/png",
    "size": 121470
  },
  "description": "A hot cache key expires and a hundred requests issue the same query at once, saturating the database. Go's singleflight package coalesces those duplicate calls into one. How to wire it up, how to measure whether it's firing, and why per-pod coalescing is usually enough.",
  "path": "/go/request-coalescing/",
  "publishedAt": "2026-06-27T00:00:00.000Z",
  "site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "tags": [
    "Go",
    "Concurrency",
    "Distributed Systems"
  ],
  "textContent": "Say you put a cache in front of Postgres to speed up reads. A hot key expires:\n\n- the next request misses the cache, so it queries Postgres to refill the key\n- the key is popular, so while that first query runs, a hundred more pile in for it\n- they all miss too, and each fires its own query\n- Postgres ends up running a hundred identical queries at once, all for the same value\n\n\n\n{{< mermaid >}}\nsequenceDiagram\n    participant U1 as User 1\n    participant UN as User N\n    participant App as Application\n    participant DB as Database\n    U1->>App: Get Product 123\n    UN->>App: Get Product 123\n    App->>DB: SELECT  FROM products WHERE id=123\n    App->>DB: SELECT  FROM products WHERE id=123\n    DB-->>App: Product data\n    DB-->>App: Product data\n    App-->>U1: Product data\n    App-->>UN: Product data\n    Note over App,DB: N identical queries!\n{{</ mermaid >}}\n\n\n\nThat's known as a [thundering herd] or [cache stampede]. I've also heard people call it\ndog-piling. Every request wants the same value, yet each one still fires its own query. It's\nwasted work, and it pounds your database for nothing.\n\nWorse, a stampede can feed itself. The query flood slows the database. Slow queries time\nout, clients retry, and the retries heap on even more load. The overload outlives the spike\nthat set it off. Marc Brooker shows how this can lead to a metastable failure in [Caches,\nModes, and Unstable Systems].\n\nRequest coalescing aims to fix that. Another name for it is [request collapsing]. The first\ncaller runs the query; everyone else waits on it and gets the same result.\n\nYou'll want it anywhere a crowd of callers needs the same expensive value at once:\n\n- an access token expires, and every in-flight request tries to refresh it at once\n- a config value gets evicted, and every worker reloads it\n- a thousand goroutines resolve the same hostname at the same time\n\nSuppressing duplicate calls\n\nCoalescing keeps a table of the calls in flight, one entry per key. When a caller asks for a\nkey that's already running, it doesn't start a second call. It waits on the one in flight\nand takes the result.\n\nGo's [golang.org/x/sync/singleflight] package does this for you. Create a Group, then call\nDo with a key and a function:\n\nDo runs the function at most once per key at a time. Call it with a key that's already in\nflight and it won't run the function again. It blocks until that first call returns, then\nhands you the same v and err.\n\nv comes back as any, so you type-assert it to its real type. The third return value,\nshared, tells you whether the result went to more than one caller. You'll use that for\nmetrics later.\n\nOn a cache miss\n\nDo only deduplicates calls that overlap in time, so it pairs with a cache.\n\nThe usual setup is [cache-aside]. You read the cache first, and on a miss you fetch the\nvalue and store it. Wrap just the fetch in Do, and concurrent misses coalesce into one\ncall per key.\n\nHere s.fetch is the upstream call: a database query, or an RPC to another service. The\ncache itself has to be safe for concurrent use.\n\n- (1) a cache hit returns straight away, without touching Do\n- (2) a miss is the only place a herd can form, so it's the only thing Do wraps; the first\n  caller per key runs fetch while other concurrent callers wait\n- (3) the first caller stores the result in cache before it returns, so later reads hit the\n  cache and skip Do\n- (4) Do returns any, so you assert the value back to a string\n\n> [!NOTE]\n>\n> Singleflight sits on the cache's miss path. When callers pile onto a miss, it runs one\n> call for all of them, then purges the key the moment that call returns.\n\nBy the time the next wave of requests shows up, the cache is warm and they never reach the\ngroup.\n\nWith Do wrapping the fetch, the first caller runs the query and the rest wait:\n\n\n\n{{< mermaid >}}\nsequenceDiagram\n    participant U1 as User 1\n    participant UN as User N\n    participant App as Application\n    participant DB as Database\n    U1->>App: Get Product 123\n    activate App\n    Note over App: singleflight Do(key) runs the fetch once\n    App->>DB: SELECT  FROM products WHERE id=123\n    UN->>App: Get Product 123\n    Note over App: User N joins the in-flight Do(key)\n    DB-->>App: Product data\n    Note over App,DB: only 1 query to the database\n    App-->>U1: Product data\n    App-->>UN: Product data\n    deactivate App\n    Note over U1,UN: every caller gets the same result\n{{</ mermaid >}}\n\n\n\nThe only thing that changes is the call into the database: a hundred queries become one.\n\nThe [example repo] shows this. It fires 100 concurrent Get calls at a cold key and sees a\nsingle fetch. After that the cache is warm, so the next 50 reads never reach the upstream.\n\nThe cost of a shared call\n\nCoalescing isn't free. You've routed many callers through one call, so a single failure or\none slow fetch hits all of them.\n\nWhen the shared call fails, every caller waiting on it gets the same error, not just the one\nthat triggered it. They can all retry, and the next call starts fresh, because singleflight\ndrops the result as soon as it's delivered. But for that one window, a single failure hits\neveryone.\n\nThey also share the wait. A caller is stuck for as long as the shared call takes, and\nthrough Do there's no way to bail out early. This is [head-of-line blocking].\n\nCloudflare hit it. Inside a datacenter, their servers share a cache lock so only one of them\nfetches from origin. They built [concurrent streaming acceleration] so the waiters don't\nblock until that fetch finishes.\n\nYou can't make the shared call faster, but you can keep it from trapping everyone. Bound the\ncall with its own timeout. Give each caller a deadline of its own, say 200ms. Then it can\nleave instead of waiting the shared call out. DoChan gives you both: it returns a channel,\nso you select on it alongside the caller's context:\n\n- (1) detach the shared call from any single caller, then give it its own timeout.\n  WithoutCancel drops the caller's deadline along with its cancellation, so without\n  WithTimeout the shared fetch would run with no bound\n- (2) each caller can still leave on its own deadline; the shared call keeps running for the\n  others\n- (3) DoChan hands back a singleflight.Result with Val, Err, and Shared\n\n> [!WARNING]\n>\n> Passing the first caller's context into the shared call is a common mistake. When that\n> caller cancels or times out, it takes down every other caller waiting on the shared call.\n> context.WithoutCancel (Go 1.21) detaches the shared work from any single caller's\n> lifetime. But it drops the deadline too, so give the shared call its own timeout or it\n> runs with no bound. Go's resolver detaches its shared lookup from any single caller's\n> cancellation, for the same reason.\n\nctx.Done gets one caller out, but the slow call stays in flight. The next caller just\njoins it and waits all over again. A time.After case caps the wait, and Forget drops the\nkey so that next caller starts fresh. Forget doesn't cancel the in-flight fetch, which is\nstill bounded by its own fetchTimeout:\n\n> [!WARNING]\n>\n> Every waiter gets the same value the shared call returned: one pointer or slice, not a\n> per-caller copy. That's fine for an immutable cache fill, but a bug the moment a caller\n> mutates it or needs a per-caller result. Even the standard library guards against it: its\n> DNS resolver clones the address slice it returns to shared callers, so each caller can\n> mutate its copy safely.\n\nMeasuring what it coalesces\n\nAlways instrument it. It's the only way to know whether coalescing is doing anything at all.\n\nDo returns shared, set to true when the result went to more than one caller. It's true\nfor the whole coalesced group, the caller that ran the fetch included. Count those returns\nand split them by success:\n\n- (1) cache the result here, the same way the cache-aside Get does above\n- (2) shared is true when this result went to more than one caller, the one that ran the\n  fetch included, so the call was part of a coalesced group\n- (3) the shared call returned an error, so every caller in the group got that error\n- (4) the shared call succeeded, so every caller in the group got that value\n\nCounted this way, each increment is one recipient of a shared result, not one suppressed\nduplicate. A coalesced group of n callers adds n, not n - 1. That's fine as a relative\nsignal.\n\nCompare the total against your traffic to see whether coalescing is doing real work or\nsitting idle. If it stays near zero, your callers rarely hit the same key at the same time.\nEither traffic is low, or the keyspace is wide enough that they don't collide.\n\nA faster upstream lowers the count too: the in-flight window shrinks, so fewer callers land\ninside it. A drop can mean a quicker upstream, not a regression.\n\nWatch the ratio of errors to successes. A rising share of shared errors means callers keep\njoining a call that fails, then retrying. That serializes the herd instead of absorbing it.\n\n[Fastly] splits coalesced requests into two counters: request_collapse_usable_count and\nrequest_collapse_unusable_count. Their usable and unusable track whether a collapsed\nrequest produced a reusable cache object. That's a cache-policy question, not a Go error. So\ntreat the mapping as an analogy, not a copy.\n\nShould you do distributed request coalescing?\n\nSingleflight is per-process. A Group sees only the calls inside one app instance, so it\ncan't coalesce across the fleet. Run twenty pods behind a load balancer. When a hot key\nexpires, each pod that takes the miss runs its own query: up to twenty, not the whole herd.\n\n\n\n{{< mermaid >}}\nsequenceDiagram\n    participant P1 as Application (pod 1)\n    participant PN as Application (pod N)\n    participant DB as Database\n    Note over P1,DB: each pod coalesces only its own herd\n    P1->>DB: SELECT  FROM products WHERE id=123\n    PN->>DB: SELECT  FROM products WHERE id=123\n    Note over P1,DB: N queries, one per pod\n{{</ mermaid >}}\n\n\n\nFor most services that's enough. Per-pod coalescing ties your database load to how many pods\nyou run, not how much traffic you get. Go's own resolver does the same: it [coalesces DNS\nlookups] per process. As long as one miss per pod fits the downstream's budget, stop here.\n\nGoing fleet-wide takes coordination. The usual tool is a per-key distributed lock. A pod\ngrabs a short Redis lease before it fetches, so only one fetch runs for that key across the\nfleet. But every miss now waits on a second system, and the lease needs tuning. Too short\nand the herd slips through; too long and a dead holder stalls everyone. To turn a handful of\nper-pod queries into one, that's often more coordination than it earns.\n\nFor cacheable HTTP, you may not have to do any of this. The cache or CDN layer in front of\nyour service can collapse requests, once you configure it to. [Varnish] coalesces concurrent\nrequests for the same object into one upstream fetch. Nginx has [proxy_cache_lock]. A\n[CloudFront Origin Shield] does the same across regions. That only covers cacheable objects,\nthough. Token refreshes, RPCs, auth-scoped or per-tenant data, none of it collapses up\nthere. That's exactly the work singleflight handles in-process.\n\nMost services never need more than per-pod. When you do, the cache layer covers cacheable\ntraffic; a lock covers the rest. Measure before you reach for either.\n\nWhen to use it\n\nCoalescing is worth it when three things are true:\n\n- the key is hot enough that callers overlap in time\n- the work behind it is expensive or slow enough to be worth sharing\n- the key fully determines the result\n\nA hot cache key fits. So do token refreshes, config reloads, and DNS. The same goes for\nanything with a predictable hotspot: a scoreboard everyone polls, a feature-flag bundle read\non every request.\n\nWhen they aren't, coalescing has little to do, and leaving it on is cheap. A miss that\noverlaps nothing only pays for Do's lock and map lookup, lost in the noise against a slow\nupstream.\n\nSo coalescing every cache miss by default is reasonable: it does nothing until a key turns\nhot, then absorbs the herd. The exception is cheap work, where the bookkeeping can cost more\nthan the call it guards. Measure there before you assume it's free.\n\nSome calls must never be coalesced. Anything with side effects is out. Merging two\ncreate payment calls is a correctness bug: the second caller wanted its own payment, not a\ncopy of the first one's.\n\nThe key also has to capture everything that affects the result. Coalesce by URL when the\nresponse depends on the Authorization header, and you'll serve one user's data to another.\n\nOr reach for a library\n\nRolling your own with singleflight has a catch. A Group is one mutex over one map:\n\nEvery Do, DoChan, and Forget locks g.mu, so a group serializes all its keys through\none mutex. At high throughput that's a contention point. Sharding spreads it out: run\nseveral groups and hash each key to one.\n\nIf you're wiring singleflight in front of a cache like this, [sturdyc] is worth a look. It's\na cache that handles the coalescing as part of its stampede protection. You also get\nrefresh-ahead, eviction, and sharded storage.\n\n\n\n\n[golang.org/x/sync/singleflight]:\n    https://pkg.go.dev/golang.org/x/sync/singleflight\n\n[cache-aside]:\n    https://learn.microsoft.com/en-us/azure/architecture/patterns/cache-aside\n\n[sturdyc]:\n    https://github.com/viccon/sturdyc\n\n[example repo]:\n    https://github.com/rednafi/examples/tree/main/request-coalescing\n\n[request collapsing]:\n    https://www.fastly.com/blog/request-collapsing-demystified\n\n[Fastly]:\n    https://www.fastly.com/documentation/reference/changes/2024/12/request-collapsing-metrics/\n\n[coalesces DNS lookups]:\n    https://go.dev/src/net/lookup.go\n\n[CloudFront Origin Shield]:\n    https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/origin-shield.html\n\n[concurrent streaming acceleration]:\n    https://blog.cloudflare.com/introducing-concurrent-streaming-acceleration/\n\n[head-of-line blocking]:\n    https://en.wikipedia.org/wiki/Head-of-line_blocking\n\n[Varnish]:\n    https://info.varnish-software.com/blog/two-minutes-tech-tuesdays-request-coalescing\n\n[proxy_cache_lock]:\n    https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_lock\n\n[Caches, Modes, and Unstable Systems]:\n    https://brooker.co.za/blog/2021/08/27/caches.html\n\n[thundering herd]:\n    https://en.wikipedia.org/wiki/Thundering_herd_problem\n\n[cache stampede]:\n    https://en.wikipedia.org/wiki/Cache_stampede",
  "title": "Request coalescing with Go singleflight"
}