Selfhosted Postmortem, Jan 13th, 2026
Cheating a bit by writing this before everything is fully back, but it should be soon, hopefully. First of all, I am sorry about the extended downtime. Not at all what I would have wanted, and I have tried everything in my power to speed it up to get y'all back to posting.
We had a user who backfilled 20k+ records in the last 24 hours and it caused us to get limited by Bluesky's relay(the thing that takes our posts and shows them on bsky.app). This is not the user's fault, but mine. I found out today that PDS rate limits are not on by default (they are now); you have to set the env variable PDS_RATE_LIMITS_ENABLED for API limits. With that set, a user could not generate enough records to cause the relay to rate limit us. The user was backfilling an account with their historical twitter history, and it was nothing malicious and I want to reiterate it's not their fault and I should have had the proper protections in place so that the user knew the rate limits and would have stopped anything before it affected everyone. That is now set, and this should not happen again. I am deeply sorry and hope this is the last of our growing pains as a PDS.
The extend of the outage is waiting on Bluesky's relay to catch up with the current firehose events emitted from the PDS.
A timeline of events
Discussion in the ATmosphere