Relay good news
I'm finally joining the firehose club.
Introducing firehose.network - public infrastructure and tools for ATProto relays.
The initial release includes 3 public ATProto relays deployed globally: North America, Europe and Asia (new).
All are available for public use with a 72-hour backfill/replay window.
Check them out at:
I've been experimenting with indigo relay setup and maintenance for a few months now.
Setup is straightforward but daily operations are where it gets interesting.
A few things I'm running to keep things healthy:
PDS discovery
A cron job pulls the PDS host list from the main bsky.network relay every few hours, diffs it against firehose.network's relays, and adds anything missing.
PDS re-crawls
A script periodically checks for offline PDSes and sends requestCrawl to see if they've come back online.
Monitoring + Alerting
Email alerts for CPU, memory, disk space. Grafana dashboards for overall and per-relay metrics (alerting still WIP)
There's also a public status page.
(FYI, Asia relay is pretty new. Moved to a bigger dedicated server very recently)
What's next:
There are some early discussions on #relay-operators channel in ATProto Touchers Discord about making relay operations easier and have better insights, and a few tools are already built in this space.
My personal wishlist - either as a separate service / built in indigo itself
Still working on a few more tools to help both PDS admins and relay operators. More to come.
All these are hugely inspired by fig's(@bad-example.com) work on microcosm public infra and based on some suggestions from Bryan(@bnewbold.net) during the ATProto_NYC meetup.
The relays are public and free to use. They already power a few of my own apps. Please do let me know if you use them and I can keep you in loop for future updates.
Follow @firehose.network for announcements.
Discussion in the ATmosphere