Building More Resilient Local-First Software with atproto

What exactly makes software local-first? I like these three bullet points from Martin Kleppmann's talk The Past, Present and Future of Local-First: If it's local-only, it's not local-first. If it doesn't work with the wi-fi off, it's not local-first. If it doesn't work if the app developer goes out of business and shuts down their servers, it's not local-first. Most local-first software focuses almost exclusively on the first two. I'm certainly guilty of it. In my article about building a local-first travel app, I claimed that the open source sync engine I used wasn't a critical dependency because you could run your own version if mine went down. But then I chose a managed service for the sync engine to avoid a big Cloudflare bill. Then the company that ran the service got acquired, and then the service got shut down. My app no longer works; I'd have to put in time and money to bring it back online. You could clone the repo and run it yourself, but let's be real: no one's gonna do that. What if there was a better way? I've been cooking up an idea that I'm pretty excited about. To demo it, I've built a simple collaborative text editor. The catch? There's no sync server. No Zero, no Electric, no Y-Sweet — nada. It's a real-time collaborative app built purely with static files. Obviously, that's not the whole story — there's a server somewhere syncing stuff! The trick is that the server is atproto. Most people who've heard of atproto know it as protocol that Bluesky uses. But it actually powers a whole constellation of apps: Tangled for version control, Leaflet and Pckt for blogging, Wisp for web hosting and many more. Am I not just moving the goalposts here? There are a zillion services that will add real-time sync to your static app. What makes atproto different from Liveblocks or Instant or Convex? Ink & Switch addresses this in their PushPin case study: Where servers are used, we want them to be as simple, generic, and fungible as possible, so that one unavailable server can easily be replaced by another. Further, these servers should ideally not be centralised: any user or organisation should be able to provide servers to serve their needs. Proprietary sync services don't fit the bill. Replacing one with another can be very difficult! But my initial interpretation of "simple, generic and fungible" was too narrow. The ability to replace an unavailable server is a low bar, because it places the onus on users to actually do so. My thesis is that atproto fulfills this plea far better — by relying on a federation of servers maintained by a large and diverse community that already powers a network of applications. If it lives up to its wildest ambitions, depending on atproto will be like depending on DNS or HTTP: public infrastructure so ubiquitous that the risks of lock-in and obsolescence all but disappear. You can see a working demo of the app here. To try it out, sign in with your Internet handle and send the URL to someone. Architecture So: no sync server, only atproto. How can we build sync on top of a protocol that doesn't know about it? We use a CRDT! CRDTs (Conflict-free Replicated Data Types) are data structures that let us merge changes from multiple sources without a centralized authority to resolve conflicts. Crucially, they ensure that everyone with the same set of changes ends up with exactly the same result. You can think of a CRDT as a sort of network-agnostic data layer. It doesn't matter how the changes get from one client to another, or how long it takes. As long as everyone can eventually find each other and exchange their changes, they'll all see the same document. That's where atproto comes in. Each user is uniquely identified by a DID (Decentralized Identifier). That DID allows other clients to find the user's PDS (Personal Data Server), which is where all their data lives. A PDS is a bit like a Git repository — except instead of arbitrary files, apps commit structured JSON that any other atproto app can read. That should be enough background to understand how this all fits together! If you're interested in learning more about CRDTs, I wrote an interactive intro that shows how they work in depth. If you're interested in learning more about atproto, Dan Abramov has a great post called A Social Filesystem that breaks down how data flows through it. For this proof of concept, I used a CRDT library called Yjs. I chose it because it's popular and good, but another CRDT library like Automerge or Loro would work as well. Let's go over how this all works. We'll skip details like authentication and text editing so we can focus on what actually makes this thing tick: the interaction between the CRDT and atproto. Persistence Before we can start collaborating with other people, we need to get the Yjs data into our PDS. To do that, we'll build a Yjs provider, which lets us sync our document with an external source. Here's the start of our provider class: I've included some convenience methods that use the atcute library to make authenticated HTTP requests to a user's PDS. The finer details are beyond the scope of this article; you can read a more in-depth overview of how exactly we find each user's PDS in Dan Abramov's Where It's at://. Whenever a document changes, Yjs emits an update event: Each event contains a binary description of the change in that update. We'll encode it as base 64 and store it in an updates collection in our PDS, along with a unique document ID so we can disambiguate between updates for other documents: We're now saving document changes to our PDS! Next, we need to load our document when the page loads. To do that, we can iterate through all the records in our updates collection, omit the updates with other document IDs and apply all the remaining ones to our document: That's it! In a few lines of code, we're using our atproto PDS as a persistence layer for Yjs documents. There is one optimization we should make. Most users are on a Bluesky-hosted PDS, which means they're subject to Bluesky's rate limits. Bluesky allows users to create 1,666 records in an hour, and if we're not careful we might blow past that. To prevent that from happening, we'll buffer and send our Yjs updates in five second intervals: Collaboration Working on a document in isolation is no fun. We want to collaborate with other people! The main challenge here is discovery: how do we find updates from other people working on the same document? We'll use the document ID as the "rkey" (record key) of a new record in a docs collection. Records can be fetched directly using their rkey, which means other clients who know both our DID and document ID can get this record without iterating through the entire collection: After creating the record, the document owner sends the full atproto URI to their collaborators out of band somehow (via a Bluesky DM, for instance). What's that editors array? The document creator adds the DID of every other user they want to be able to edit the document. Each of those users adds any changes they make to the updates collection in their own repo. When a user loads a document, they first get the record from the docs collection at the known atproto URI. Then, in addition to fetching their own updates, they also fetch the updates from the other editors: What, were you expecting more code? Editor updates are exactly the same as "owner" updates. Because Yjs is a CRDT, we're guaranteed to have the most up-to-date version of the document once all the updates are applied. Realtime Editing Of course, it's not a good experience to incorporate other editors' changes only on page load. We want to see things happening in real time. atproto can push updates to clients in a few ways. The one we'll use is called the Jetstream: a WebSocket connection that streams events from select repositories and collections as they happen. After loading the editors list, we connect to the Jetstream and subscribe to those DIDs (plus the owner's) and listen for new records in the ‌doc and updates collections: Unfortunately, the Jetstream strips away the signing data that verifies that the record actually came from a given user. That means the Jetstream could forge updates! To prevent this, whenever we get anything from the Jetstream, we'll also fetch it from the repo and compare the CIDs (content IDs), which are hashes of the content: If the CIDs match, we're good to go. For the updates collection, that means applying the updates to our document: For the docs collection, an event means the list of editors may have changed. The easiest way to handle that is to simply disconnect from the Jetstream and reload the document with the new list of editors: It's possible that we could miss an event while disconnected from the Jetstream — not only during this process, but when e.g. closing a laptop. To solve this, we keep track of the most recent record we received and pass it as a cursor when connecting to the Jetstream: Awareness Let's add one last finishing touch. You're probably familiar with this from Google Docs: live cursor positions as other editors navigate the document. Yjs calls this type of feature awareness. It still uses a CRDT to store this ephemeral state — but outside of the document, with no edit history. Because the constraints are so different, we'll use a dedicated awareness collection to store this state. Rather than create a new record with every change, we can create a single record using the document ID and overwrite it when our awareness state changes. We'll also want to throttle this to prevent our users from hitting rate limits: As with updates, when loading a document, we merge the awareness records from the other editors: We also listen for updates to the awareness collection in the Jetstream to get live changes: And there we have it! A real-time collaborative text editor with no dedicated server, using atproto to sync our changes. Caveats Alas: nothing in this life is perfect. The lack of signed data in the Jetstream isn't ideal. Comparing the CID to the actual PDS record makes it much harder for a malicious Jetstream relay to spoof updates, but it would be nice to actually cryptographically verify the updates are legit. Also, all the data is public. We could try to encrypt it somehow, but it still makes me uneasy to have private data floating out there in my PDS — even if it's ostensibly protected. The proper solution there is probably to wait for atproto to support private and shared private data. I don't love that there's an initial document owner who determines which other users can edit it. I guess that's the Google Docs–type UX that people expect, but I see it as a limitation. CRDTs can be merged without a centralized authority, so in theory a document doesn't need an "owner" — every copy can be its own canonical version, as though you'd emailed someone a Photoshop document or Excel spreadsheet. Relatedly, there's no good way to have publicly editable documents. I wanted this blog post to have an embedded demo that everyone could edit, but the document owner needs to specifically add each person to the editors list. Finally, it's not quite real-time! We're sending updates at most every five seconds. And even if we weren't, the Jetstream latency seems to be just too high to make remote keystrokes appear "live" (which makes sense, since we're filtering down a firehose of all atproto network activity). I have some ideas for how to dramatically improve the performance, though — watch this space! Parting Thoughts Reflecting on the atproto conference in Vancouver this past weekend, Chad Fowler posted on Bluesky: Observation from \#ATmosphereConf: The AT folks and localfirst folks (not to mention the "good parts" of web3) are attacking the same problems from different angles. There is not NEARLY enough overlap in these communities. I totally agree. Both communities have a lot of shared goals: user agency, data ownership, interoperability, files over apps. My hope in publishing this is to inspire more people to play around where the two overlap. By The Way When I was working on this proof of concept, I stumbled on a package called y-atproto that works very similarly to my implementation. npm install it and give it a whirl!