{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreia5ljlitv5ihdejco66do6cksyhqgj4uxky7kwhkibozzzmf6ykgu",
    "uri": "at://did:plc:om3y3n35sj2qfdrmdvrzmusa/app.bsky.feed.post/3mm5vkyksdjp2"
  },
  "path": "/2026/05/18/fedimeteo-haproxy-and-the-art-of-not-wasting-snac-threads/",
  "publishedAt": "2026-05-18T09:44:00.000Z",
  "site": "https://it-notes.dragas.net",
  "tags": [
    "freebsd",
    "haproxy",
    "server",
    "networking",
    "hosting",
    "fediverse",
    "snac",
    "jail",
    "ownyourdata",
    "snac2",
    "web",
    "social",
    "FediMeteo",
    "Improving snac Performance with Nginx Proxy Cache",
    "Caching snac Proxied Media with Nginx",
    "Bastille"
  ],
  "textContent": "When I wrote about FediMeteo for the first time, I told the story from the beginning: the idea born almost by chance while checking the weather for a holiday, the memory of my grandfather, who for years had been my personal meteorologist, the decision to build something small and useful, and then the surprise of seeing people actually use it. What began as a personal experiment quickly became a small global service, still running with the same philosophy: FreeBSD, jails, simple scripts, snac, text, emoji, and a lot of small pieces doing their work quietly.\n\nThat article was mostly about the birth and growth of the project. This one is about one of the less romantic parts of the same story, although I have to admit that I find a certain beauty in it too: keeping the service light as it grows.\n\nFediMeteo is still intentionally simple from the outside. A homepage, some numbers, a list of countries, and many ActivityPub accounts publishing weather forecasts. The posts are text and emoji. There is no JavaScript requirement to read the pages, no heavy frontend, no unnecessary media attached to every forecast, and no dynamic homepage recalculated at every visit just to show the same numbers. This is not accidental. It is the way I wanted the service to behave from the beginning.\n\nBut the more the service is used, the more the small details matter. A request that looks harmless when there are ten followers may become a repeated request when there are thousands of followers, remote instances, crawlers, previews, and other servers fetching the same public objects. In the Fediverse, the same small thing can be asked many times by many different places, each one with a perfectly legitimate reason. The backend doesn't care: it just needs to deal with the requests.\n\nAnd in FediMeteo, the backend is snac.\n\nI like snac very much precisely because it is small, clear, and efficient. It is not a giant application that tries to be everything. It does a focused job and does it well. But this also means that I want to respect its shape. I do not want to waste its threads on work that the reverse proxy can safely do. A snac thread serving the same public avatar again and again is not a tragedy, but it is still a waste. A snac thread answering the same public ActivityPub object several times in the same minute is doing real work, but often not necessary work.\n\nThis is the reason behind the HAProxy tuning I am currently using in front of FediMeteo.\n\nIt is not about making the configuration look clever. It is about keeping snac quiet.\n\n## A continuation of the same idea\n\nI had already explored the same problem with snac and nginx in two previous posts: Improving snac Performance with Nginx Proxy Cache and Caching snac Proxied Media with Nginx. In both cases, the idea was that the reverse proxy should absorb repeated public requests instead of letting them consume snac resources.\n\nThis is especially important because snac uses a limited number of threads. I like that. Limits are healthy. They force us to understand what the service is doing, and they prevent a small program from pretending to be an infinite resource. But limits also make waste visible. If a few threads are busy serving files that could have been served from cache, those threads are not available for something more useful.\n\nWith FediMeteo the implementation is different because the reverse proxy is HAProxy, but the reasoning is the same. I have many small snac instances, each one in its own FreeBSD (Bastille) jail, and one public entry point that has to route, terminate TLS, compress, cache, and generally remove as much repetitive work as possible from the backends.\n\nThis is, in a way, the natural continuation of the original FediMeteo design. In the first article I wrote that I wanted to manage everything according to the Unix philosophy: small pieces working together. This is another piece of that same puzzle. HAProxy does the edge work. snac does the ActivityPub work. Scripts generate forecasts. cron launches updates. ZFS gives me snapshots. FreeBSD jails keep countries separated. Nothing is particularly heroic by itself, but the whole system becomes pleasant because each part has a clear responsibility.\n\n## Why there is almost no media\n\nBefore talking about HAProxy, it is worth mentioning one of the most important optimizations, which is not in the proxy configuration at all.\n\nFediMeteo does not use media in its forecasts.\n\nNo images attached to the posts, no generated weather cards, no maps for each city, no decorative banners. The forecasts are text and emoji. This was a deliberate decision. Weather information does not become more useful just because it is put inside an image, and every media file used by the service would become something to store, serve, cache, federate, expire, back up, and occasionally debug.\n\nText and emoji are enough. They are accessible, light, readable in text browsers, friendly to timelines, and understandable even when someone does not know the local language perfectly. This was one of the original design principles of FediMeteo, and it also helps the infrastructure. Less media means less work, fewer cache entries, fewer repeated fetches, fewer surprises.\n\nThere is one exception: the avatar.\n\nAll FediMeteo accounts use the same avatar, and this is also intentional. I could have used a different avatar for each country, or for each city, or created something visually richer. It would have been nicer in some screenshots, perhaps. It would also have been operationally worse.\n\nWith one shared avatar, the reverse proxy has one very useful object to cache. It is public, identical for everyone, small, requested often, and therefore almost always hot in cache. HAProxy can serve it directly instead of asking each snac instance to return the same file. Since avatars are requested by remote instances, browsers, profile previews, and all sorts of federation-related fetches, this single decision removes a surprising amount of pointless backend traffic.\n\nSo the avatar is not only a visual identity. It is part of the architecture.\n\nThis is the kind of optimization I like most, because it starts before the software. It starts with deciding not to create a problem.\n\n## The homepage is static because it can be static\n\nThe main homepage follows the same logic.\n\nIt is a static HTML page generated from a template. Once per hour, a cron script updates the numbers and statistics. It counts the data I want to show, regenerates the page, and then the page remains static until the next run.\n\nThis is not because I cannot make a dynamic page. It is because I do not need one. Boring is good.\n\nThe homepage does not need to query all the country instances on every visit. It does not need a database request for each user who opens it. It does not need to ask snac anything in real time. The numbers are useful, but they do not need to be updated every second. Once per hour is enough, and it also fits the spirit of the whole project: do the work when it is needed, then serve the result cheaply.\n\nI have seen too many small services become heavy because the first implementation was convenient rather than appropriate. A cron job and a template are not fashionable, but they are often exactly what a page like this needs.\n\n## Many countries, one entry point\n\nFediMeteo is made of many country instances. Each one runs in its own jail and listens on its own internal address and port. From the outside, however, they all live under the same domain structure:\n\n\n    fedimeteo.com\n    www.fedimeteo.com\n    it.fedimeteo.com\n    uk.fedimeteo.com\n    jp.fedimeteo.com\n    us.fedimeteo.com\n    usa.fedimeteo.com\n    can.fedimeteo.com\n    canada.fedimeteo.com\n\n\nAnd many more.\n\nAt the beginning, it is always tempting to write one ACL after another in the HAProxy frontend. It is quick, it is explicit, and for five hostnames it is perfectly fine. But FediMeteo did not remain at five hostnames. As countries and aliases grew, a long chain of ACLs would have turned the frontend into a list of names instead of a description of how the proxy behaves.\n\nSo I moved the hostname to backend mapping into a map file:\n\n\n    fedimeteo.com        backend_fedimeteo\n    www.fedimeteo.com    backend_fedimeteo\n    it.fedimeteo.com     backend_it\n    uk.fedimeteo.com     backend_uk\n    jp.fedimeteo.com     backend_jp\n    us.fedimeteo.com     backend_us\n    usa.fedimeteo.com    backend_us\n    can.fedimeteo.com    backend_ca\n    canada.fedimeteo.com backend_ca\n\n\nThe frontend then needs only one rule:\n\n\n    use_backend %[req.hdr(host),field(1,:),lower,map(/usr/local/etc/fedimeteo.map,backend_fedimeteo)]\n\n\nThis reads the `Host` header, removes the port if present, lowercases the result, and looks it up in `/usr/local/etc/fedimeteo.map`. If nothing matches, it falls back to the main FediMeteo backend.\n\nI like this because it keeps the configuration honest. The frontend contains the policy. The map contains the data. Adding a country means adding an entry to the map and defining a backend. I do not need to make the frontend more complicated every time the service grows.\n\n## Backends as small compartments\n\nThe country backends are deliberately plain:\n\n\n    backend backend_it\n        mode http\n        http-reuse safe\n        server srv1 10.0.0.2:8001 maxconn 30\n\n    backend backend_uk\n        mode http\n        http-reuse safe\n        server srv1 10.0.0.7:8001 maxconn 30\n\n    backend backend_jp\n        mode http\n        http-reuse safe\n        server srv1 10.0.0.32:8001 maxconn 30\n\n\nOne backend, one jail, one snac instance. This is exactly the same organizational principle as the rest of the project. If I need to reason about Italy, I look at the Italian jail. If I need to reason about the United Kingdom, I look at the UK jail. If one day I need to move a country elsewhere, the separation is already there.\n\nThe `maxconn 30` value is not a magic number. It is a ceiling. I want each small backend to have a visible limit in front of it. If something starts hammering a country instance, I prefer the pressure to appear at the HAProxy layer instead of becoming unlimited concurrent work inside snac.\n\n`http-reuse safe` lets HAProxy reuse backend connections where appropriate. This is another small reduction in unnecessary work. Opening connections repeatedly is not the biggest problem in the world, but avoiding it is still better, especially when many small services sit behind the same proxy.\n\n## The front door\n\nThe HTTPS frontend listens on IPv4 and IPv6 and offers both HTTP/2 and HTTP/1.1:\n\n\n    frontend https_in\n        bind :::443 v4v6 ssl crt /usr/local/etc/certs/ alpn h2,http/1.1\n        mode http\n        option http-keep-alive\n\n\nTLS defaults are set globally:\n\n\n    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256\n    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets\n\n\nPort 80 only redirects to HTTPS, except for Let's Encrypt challenges:\n\n\n    acl letsencrypt-acl path_beg /.well-known/acme-challenge/\n    http-request redirect scheme https code 301 unless letsencrypt-acl\n    use_backend letsencrypt-backend if letsencrypt-acl\n\n\nIn the HTTPS frontend I also set the usual forwarding headers:\n\n\n    http-request set-header X-Real-IP %[src]\n    http-request set-header X-Forwarded-Proto https\n\n\nAnd I add HSTS:\n\n\n    http-response set-header Strict-Transport-Security \"max-age=31536000; includeSubDomains; preload\"\n\n\nNone of this is unusual, and that is fine. The interesting parts of an infrastructure are not always the parts that should be unusual.\n\n## Two caches, because the requests are different\n\nThe HAProxy configuration defines two caches:\n\n\n    cache mediacache\n      total-max-size 128\n      max-object-size 10000000\n      max-age 3600\n      process-vary on\n      max-secondary-entries 12\n\n    cache jsoncache\n      total-max-size 16\n      max-object-size 1000000\n      max-age 60\n      process-vary on\n      max-secondary-entries 12\n\n\nI keep media and ActivityPub JSON separate because they are not the same kind of traffic.\n\nThe media cache is larger and has a longer maximum age. In FediMeteo, this mostly means the shared avatar and a few static-looking objects. Since there is intentionally almost no media, the important cached object is requested very often and remains warm.\n\nThe JSON cache is smaller and short-lived. It is there for public ActivityPub GET requests, not to store federation state forever. A 60 second cache is enough to collapse many repeated requests that arrive close together in time, without pretending that ActivityPub responses should be treated like immutable files.\n\nThis distinction is important. Caching is not one decision. It is a set of small decisions about what a response means, who can see it, how often it changes, and what happens if it is served again.\n\n## Recognizing media\n\nFor media, the ACL is based on file extensions:\n\n\n    acl is_media path_end -i .jpg .jpeg .png .gif .webp .svg .ico .mp4 .webm .mp3 .ogg .wav .flac .mov .avi .mkv .m4v\n\n\nThen I store the result in a transaction variable:\n\n\n    http-request set-var(txn.is_media) bool(true) if is_media\n\n\nThe cache lookup is straightforward:\n\n\n    http-request cache-use mediacache if { var(txn.is_media) -m bool true }\n\n\nAnd on the response side:\n\n\n    http-response set-header Cache-Control \"max-age=3600, public\" if { var(txn.is_media) -m bool true }\n    http-response del-header Set-Cookie if { var(txn.is_media) -m bool true }\n    http-response del-header Vary if { var(txn.is_media) -m bool true }\n    http-response cache-store mediacache if { var(txn.is_media) -m bool true }\n\n\nThe `Cache-Control` header makes the intent explicit. `Set-Cookie` is removed because a public media object should not carry session information. `Vary` is removed because I do not want the same avatar to fragment into many cache entries because of harmless header differences.\n\nThis is aggressive only if removed from its context. In this service, with this media policy, it is a reasonable choice. FediMeteo is not serving private media under these paths. It is mostly serving the same public avatar over and over.\n\nFor the same reason, I clean the request before it reaches the backend:\n\n\n    http-request del-header Authorization if { var(txn.is_media) -m bool true }\n    http-request del-header Cookie        if { var(txn.is_media) -m bool true }\n\n\nI would not do this globally. I do it after deciding that the request is media. Scope is what makes these rules safe.\n\nThe result is exactly what I want: the shared avatar becomes an almost perfect cache object. Small, public, repeatedly requested, and served by HAProxy instead of snac.\n\n## ActivityPub JSON microcaching\n\nThe ActivityPub side starts from the `Accept` header:\n\n\n    acl is_ap_json   req.hdr(Accept),lower -m sub application/activity+json\n    acl is_ap_ldjson req.hdr(Accept),lower -m sub application/ld+json\n    acl is_outbox    path_end /outbox\n    acl is_get       method GET\n    acl has_auth     req.hdr(Authorization) -m found\n    acl has_cookie   req.hdr(Cookie) -m found\n\n\nThis part matters because ActivityPub uses content negotiation. The same path may return HTML to a browser and JSON to a remote instance. If the proxy pretends that a URL is always one thing, it will eventually cache the wrong representation.\n\nSo I only mark public ActivityPub GET requests as cacheable:\n\n\n    http-request set-var(txn.is_activitypub) bool(true) if is_get !is_outbox is_ap_json !has_auth !has_cookie\n    http-request set-var(txn.is_activitypub) bool(true) if is_get !is_outbox is_ap_ldjson !has_auth !has_cookie\n\n\nThere are several decisions here, all important.\n\nIt must be a `GET`, because I am not caching deliveries or anything that changes state. It must not be `/outbox`, because outbox collections are not the traffic I want to cache here. It must not have `Authorization`, and it must not have cookies, because authenticated or user-specific requests do not belong in a shared public cache.\n\nThen the cache can be used and populated:\n\n\n    http-request cache-use jsoncache if { var(txn.is_activitypub) -m bool true }\n\n    http-response set-header Cache-Control \"max-age=60, public\" if { var(txn.is_activitypub) -m bool true }\n    http-response cache-store jsoncache if { var(txn.is_activitypub) -m bool true }\n\n\nSixty seconds is short, but useful. Federation often creates small clusters of identical requests. A remote server fetches an actor, another fetches the same actor, something asks for the same object, something retries. I do not need to cache these responses for hours. I only need HAProxy to answer the second and third identical request during the same small burst.\n\nThis is microcaching in the most practical sense. It reduces repeated work without changing the nature of the service.\n\n## Static media paths\n\nThere is also a rule for static paths:\n\n\n    acl is_short_path path_reg ^/[^/]+/s/\n    http-request cache-use mediacache if is_short_path\n\n\nThis comes from the same observation that led me to cache snac media with nginx. snac uses static media paths, and those paths often represent the kind of public, repeatable traffic that should not consume backend threads if the proxy can serve it. I call them \"short\", not because they are, but because the first time I saw them, I thought the 's' stood for \"short\", not \"static\". The name just stuck.\n\nIn FediMeteo this is less central than on a normal social instance, because I deliberately do not use media except for the avatar and basic static objects. Still, the rule fits the general policy: let HAProxy handle repeatable edge work, and let snac spend its threads where they are actually needed.\n\n## `Vary`, but not without limits\n\nBoth caches have:\n\n\n    process-vary on\n    max-secondary-entries 12\n\n\nI want HAProxy to process `Vary`, because content negotiation is real, especially when ActivityPub is involved. But I also want variation to be bounded. If every slightly different header creates another cache entry, the cache becomes a complicated way to miss.\n\nFor media, I remove `Vary` before storing the response. A shared avatar does not need to vary by `Accept`. For ActivityPub JSON, I am more careful because the representation matters.\n\nAgain, the important thing is not the number itself. It is the decision to make variation explicit and limited.\n\n## Seeing whether it works\n\nDuring rollout, I like to expose a very small diagnostic header:\n\n\n    http-response set-header X-Cache-Status HIT if !{ srv_id -m found }\n    http-response set-header X-Cache-Status MISS if { srv_id -m found }\n\n\nThis is intentionally simple. If HAProxy selected a backend server, I call it a miss. If no backend server was selected, the response came from cache, so I call it a hit. It is not a complete observability system, but it is enough to answer the first question I usually have after changing a cache rule.\n\nDid this request reach snac?\n\nA test can be as simple as:\n\n\n    curl -I https://it.fedimeteo.com/path/to/avatar.png\n    curl -I https://it.fedimeteo.com/path/to/avatar.png\n\n\nThe second request should be a hit.\n\nFor ActivityPub JSON, the test must use the right `Accept` header:\n\n\n    curl -I \\\n      -H 'Accept: application/activity+json' \\\n      https://it.fedimeteo.com/some/activitypub/object\n\n\nAnd I also want to verify that cookies and authorization prevent public caching:\n\n\n    curl -I \\\n      -H 'Cookie: test=value' \\\n      -H 'Accept: application/activity+json' \\\n      https://it.fedimeteo.com/some/activitypub/object\n\n    curl -I \\\n      -H 'Authorization: Bearer fake' \\\n      -H 'Accept: application/activity+json' \\\n      https://it.fedimeteo.com/some/activitypub/object\n\n\nA cache that works should be visible. A cache that is invisible can be correct, but it can also be silently wrong. I prefer to know.\n\n## Compression and operational paths\n\nHAProxy also handles gzip compression:\n\n\n    filter compression\n    compression algo gzip\n    compression type text/css text/html text/javascript application/javascript text/plain text/xml application/json application/activity+json\n\n\nThis keeps another common responsibility at the edge. The country instances can stay focused on snac and the forecast data, while HAProxy deals with client-facing compression for HTML, JSON, and ActivityPub responses.\n\nThere is also a local Prometheus exporter:\n\n\n    frontend prometheus\n      bind 127.0.0.1:8405\n      mode http\n      http-request use-service prometheus-exporter\n      no log\n\n\nAnd I keep internal operational paths, such as statistics and Grafana, handled before the hostname map. These are small details, but ordering matters. Special paths should be explicit and early. The hostname map is for FediMeteo routing, not for every internal tool I happen to expose behind the same proxy.\n\n## What this changes in practice\n\nThe nice thing about this configuration is that none of its parts is particularly surprising.\n\nThe map keeps hostname routing manageable. The backend definitions keep each country isolated and limited. The static homepage avoids dynamic work for something that changes once per hour. The shared avatar gives HAProxy one very hot media object to serve directly. The media cache keeps public files away from snac. The JSON microcache absorbs short ActivityPub bursts. Header cleanup prevents useless variation. Connection reuse avoids unnecessary backend connection churn.\n\nBut all of this is only a longer way of saying one thing:\n\n_fewer requests reach snac_.\n\nThat is the metric I care about here.\n\nNot because snac is slow. If anything, FediMeteo exists in its current form because snac is efficient enough to make this kind of project possible on a very small VPS. But precisely because the whole architecture is small and pleasant, I do not want to waste resources where there is no need.\n\nThis is also consistent with the rest of the project. Forecasts are serialized by scripts. Updates happen every six hours. The homepage is regenerated hourly. Countries live in separate jails. Snapshots and backups are handled outside the application. No single component tries to be the entire system.\n\nHAProxy is just another small piece, but it sits in the right place to remove a lot of repeated work.\n\n## Caveats\n\nThis configuration is not a universal HAProxy recipe for ActivityPub services.\n\nIt matches FediMeteo as it is now: almost no media, one shared avatar, static homepage, public forecasts, many small snac instances, and ActivityPub traffic that can benefit from a short public cache when there are no cookies or authorization headers.\n\nIf I decide one day to use media in forecasts, the media cache rules will need to be reviewed. If I use different avatars for each city or country, the cache will still work, but I will lose the very nice property of one shared, always-hot avatar. If ActivityPub responses become actor-dependent, public JSON caching must be reconsidered. If one country grows a very different traffic pattern from the others, it may deserve a different limit or policy.\n\nThis is why I do not like presenting configurations as magic. A good configuration is a written form of the assumptions behind a service. When the assumptions change, the configuration must change too.\n\n## Conclusion\n\nFediMeteo started as a small idea and became larger than I expected, but I still want it to feel small in the right ways. Small does not mean fragile. Small means understandable. It means that each part has a reason to exist, and that unnecessary work is removed before it becomes a problem.\n\nThe HAProxy layer follows this idea. It terminates TLS, routes hostnames through a map, reuses backend connections, serves the shared avatar from cache, microcaches public ActivityPub JSON, avoids authenticated and cookie-based traffic, and gives me a small diagnostic header to see what is happening.\n\nThere is no single brilliant directive here. There is only the usual work of matching infrastructure to reality.\n\nFediMeteo publishes weather forecasts as text and emoji. The homepage is static HTML updated every hour. The accounts share the same avatar because it is enough, and because it is better for the cache. Each country has its own snac instance in its own FreeBSD jail. HAProxy stands in front of them and tries, quietly, not to bother them unless it has to.\n\nI like this kind of infrastructure.\n\nNot because it is invisible, but because when it works well, it leaves very little to say.",
  "title": "FediMeteo, HAProxy, and the art of not wasting snac threads"
}