increasing my compute spend

disclaimer: these instances are new and i'm not an expert at running relays / low-level atproto infra, this is experimental!!

i'm a firehose consumer

my "bsky trending topics" (called coral) implementation consumes jetstream to get the posts one can try to derive trending topics from

https://coral.waow.tech External Link • coral.waow.tech

my standard.site index has tap ingest and backfill posts and pubs from atmospheric blogs pckt.blog, leaflet.pub, offprint.app etc

https://pub-search.waow.tech External Link • pub-search.waow.tech

historically, these have ultimately depended on bsky infrastructure: jetstream*.bsky.network + relay*.bsky.network respectively.

now they use jetstream.waow.tech + relay.waow.tech respectively

coral reads bsky posts via jetstream + pub-search taps this relay

importantly, tap needs listReposByCollection which runs as a sidecar service seemingly not offered by prominent indie relays (unless i am missing something which i totally may be)

since bsky appears to push the image for collectiondir to a private registry, i built my own image for that and put it on atcr.io so tap can happily update itself on who has new collections it cares about

i have only started backfilling this tho! still thinking (and getting feedback) on this part, seems like there's a offline crawler thing i could run, but without the complete backfill it seems like tap is still not practically going to work as advertised

EDIT: it seems there's some prior discussion on this in the atproto /microcosm discord i only realized after i started working on this, around whether you can circumvent collectiondir etc, will be looking at this as i think about how/if i'm going to go network scale

i have a single-node k3s cluster in Virginia, USA via Hetzner that serve the relay, jetstream, and collectiondir behind traefik. i'll have to see about resource allocations given all the things i am running

ideally, the idea of checking coral and/or pub-search now to eyeball whether my relay infra is working (obviously not a completely reliable proxy, but a useful / quick smoke check!) is attractive

but for less-vibey observabililty, i have a public read-only grafana:

https://relay-metrics.waow.tech/d/relay-waow/relay-waow-tech?orgId=1&from=now-1h&to=now&timezone=browser External Link • relay-metrics.waow.tech

i encourage people to use these for non-critical things! tell me if it violates your expectations anyhow, that'd be much appreciated :)

i'm still learning atproto's lower levels, which i do by trying things!

here's the code, which reflects my very lazy declarative goals:

relay w listReposByCollection support on minimal Hetzner single node k8s ~ inspired by Bryan Newbold's $34 dollar relay blog post

.. i kinda like k8s sometimes 🫢 i'm sorry! i may reassess and very open to suggestions on different deployment topologies folks use irl. seems i'm on track to spend around the <= $34 per month? we'll see

i ended up adding listReposByCollection because of tap in pub-search but just now realized this subject had intersected with (what appears to be related to) the atproto hit-piece discourse earlier on

that reminded me i ran (naively) into that with tap a while ago, so i was able to connect the dots and put up a pod with that sidecar

🙇🏻‍♂️ ty @bad-example.com and @sri.xyz for (afaik) leading the way!

OG independent relays

https://relay.fire.hose.cam/ External Link • relay.fire.hose.cam

https://firehose.network External Link • firehose.network