Brendan O'Brien - n0, Iroh and the Future of Peer to Peer

{/ TAB: SHOW NOTES /} This week we're joined by Brendan O'Brien (b5), founder and CEO of n0, the company behind Iroh - a peer-to-peer networking library that prioritizes reliability and "just works." Iroh enables developers to establish direct, authenticated connections between any two devices using only their public keys, achieving near 100% connection success rates. We discuss the pragmatic approach to P2P networking, why they chose to focus solely on the transport layer, and how Iroh is already running in production on hundreds of thousands of devices. - https://twitter.com/b5 - https://github.com/b5 - https://github.com/n0-computer - https://iroh.computer/ - https://www.iroh.computer/docs - https://github.com/n0-computer/iroh - https://github.com/n0-computer/iroh-examples - https://github.com/n0-computer/awesome-iroh - https://perf.iroh.computer/ - https://discord.gg/n0 - https://n0.computer/ {/ LINKS /} {/ Paste show notes /} {/ TAB: SECTIONS /} [00:00:00] Introduction [00:02:04] The Genesis of N Zero and Iroh [00:05:54] Iroh's Unique Approach to Peer-to-Peer [00:14:43] Understanding Quik and Multipath [00:22:02] Content Addressable Storage in Iroh [00:26:14] Establishing Connections in Iroh [00:33:15] Challenges and Solutions in Relay Servers [00:38:18] The Future of Peer-to-Peer Networking [00:56:46] Building Sustainable Open Source Businesses [01:05:44] Conclusion and Final Thoughts {/ TAB: TRANSCRIPT /} 148 Brendan: the modern way of shipping an app is to ship half an app. You ship the front end, and we call them front ends because there's a back of house that you don't get, and the core thesis around peer, like forget peer-to-peer, all we're saying is ship the whole app, put the client and the server in the hole in the same binary. Yes. It's hard. ​ [00:00:19] Introduction Andrew: Hello, welcome to Dev Tools fm. This is a podcast about developer tools and the people who make 'em. I'm Andrew, and this is my co-host Justin. Justin: Hey everyone, uh, we're really excited to have Brendan O'Brien on. Uh, so Brendan, you're known as B five, uh, online, uh, and I've run into you in a lot of, in person Brooklyn meetups and also in Berlin at local first conference, which is really fun. Um, so you are the one of the, the, the founder, CEO of n zero, or do you call it number zero? Uh, Brendan: We take either, uh, and number zero is what we started with. But nowadays, N zero is one. We get a lot. Justin: Cool, cool, cool. Uh, so you are, uh, working on Iroh, this, uh, breast based peer-to-peer networking library. Um, so we're really excited to talk about that today. Um, this is gonna be a pretty technical episode, uh, but I think it's gonna be really, really interesting. So I'm excited to dig in. But before we start talking about that, would you like to tell our listeners a little bit more about yourself? Brendan: Totally. Yeah. Um, I go by B five professionally. Um, I'm more internet native than I am meat space native these days. Uh, I've been working in and around, uh, peer-to-peer systems and really chasing this notion of what we now refer to as user agency, uh, for the better part of 10 or 11 years at this point. Uh, and so I, it's been a real pleasure and honor of my career to get to build and work in open source on a day-to-day basis. Um, trying to figure out some of the nuances of that, trying to work in around that space and what it means to sort of create and build in the open and have that be a driver of, um, wonderful intangible, intangible value out in the wild is, is kind of been a total delight. Um, so that's kinda brought me to here, to this, this work on Ira. [00:02:04] The Genesis of N Zero and IRO Andrew: So what led you to start building? Uh, number zero and, uh, IIRO itself. Brendan: Uh, totally. My co-founder and I met, um, we were both working in and around the IPFS project, which is the interplanetary file system. And, uh, we decided to start number zero really as a, uh, at a, at a sort of an impasse for the IPFS project where we realized that what we felt was missing was just like a really great hardcore engineering take on, uh, a good peer-to-peer networking library. And that we felt like that would be a sort of step function change if we had that in like a really, like I. Google Chrome grade engineering work. And so that meant like writing in a systems language and really taking things down a couple of layers. And we could only do that because, uh, the IPFS project had really illuminated this opportunity in this incredible space, uh, for doing, uh, reviving a lot of peer-to-peer thinking. And that's sort of what drove us here. Uh, and so we started the project right around that. Basically just, Hey, let's do the most soup to nuts, uh, performance oriented runs in as many places as possible, iteration of a peer-to-peer networking library. Uh, and let's try and do that in 2025 or 2023 when we started it. Uh, with as many lessons as we've learned from other projects like Web RTC and lip B two P, um, as we could. Justin: So what was the sort of moment where you decided that this layer of the stack was something that was really important to you? Just like sort of digging in a little bit deeper, because like, peer-to-peer is pretty, like, I, I don't wanna say it's like niche, uh, because obviously, you know, there's a, there's a long history of peer-to-peer applications. Uh, but like what, what about this space? Like really compelled you? Brendan: Yeah, I think you're absolutely, I think it's completely fine to call it niche, like as it is. Like if we just think about the amount of traffic that just flows over HT P to regular servers is, that's the vast majority of the internet, right? And so, like, it's absolutely niche and it's absolutely this, um, small space. My involvement actually, uh, started doing a lot of government data archiving work, uh, in 2016 where we started to realize that sort of like took this core thesis that like there are swat large swaths of the internet that are just gated by DNS, right? Like you can turn off whole bunch of the internet and you can turn off a lot of access to the internet, um, just through DNS controls. And this is, um, location based addressing, or this is just the fact that we all use URLs to describe and, and reference stuff on the internet. And we started to like problematize that in our heads it's like, wait a second. Why does the internet work that way? Right? Like, and, and could it work better, better? Or at least could we have a viable alternative? Right? And I don't think this is necessarily a Iroh is very much not a project that's aimed at replacing HTP. Um, this is much more like thinking about a world in which we have more tools in the toolbox to kind of build on and iterate with. And so I think that's really what got me excited was this early project, we're doing a lot of archiving work. We're like, wait a second, why does this work this way? Can we come up with a better approach or a, a viable alternative that would allow us to sort of move data in ways that are a little more, uh, friendly to. The interest of the users at hand, right? In, in that use case, we have a whole bunch of users who are just trying to get data sets to do their regular day-to-day work. In this case it was on climate change analysis, right? Just downloading daily weather, which like if you look at, uh, NOA in the us they produce like on the order of a petabyte of data a week, like just really silly amounts of data and, uh, making sure that you have continued access to that. How does that look? How does that work? How do you manage that infrastructure? And then now we're officially down in the internet tubes trying to figure out how we move bytes around and how people access those bytes. And, and lo and behold, you're starting to like ask questions that are pretty foundational, uh, to the networking stack. I Andrew: So IO ha Brendan: go ahead, sorry. [00:05:54] IRO's Unique Approach to Peer-to-Peer Andrew: IROH has a pretty unique philosophy, uh, when it comes to peer-to-peer compared to other projects. Uh, you've said, I think in other posts online that, uh, other systems try, try to boil the ocean and do a whole lot of things. I was reading an article where you guys talked about Lib P two P and like it's doing a whole bunch of stuff and Iroh takes a slightly different approach. Can you detail what that approach is to us? Brendan: Totally. Yeah. I, I think, um, and, and this is not to throw shade, right? Like there's less, and one of the things when we, we talk about like a modern peer-to-peer library. Uh, we've seen this sort of trend over the last 10 years of work in this space, I'd say, where, uh, the metaphor I like is like, you're trying to build an electric car, right? The whole world runs on gas and you wanna popularize a new way to do the same thing, right? You still want to, like, it's still gonna be a car, it's still gonna get you from A to B. It's still gonna be a thing that connects to other devices and moves bys around. But you, you have this like cold start problem of like, you know, to have an electric car, you need an electric charging station, you need a gas station, you need all of this infrastructure that just like doesn't exist. And then you need like plants to figure out how to make that stuff. And you need all the pieces that go into a car that maybe you don't make. And so now you need like an electric car or carburetor, which actually. Don't, uh, the metaphor breaks down when I start to talk about the internals of a car. But like the, the, the point being, we, you saw this sort of boil the ocean approach for a long time. Hey, we're gonna try and do it peer-to-peer. We're gonna do the connections and then we're gonna do every piece of the stack between the raw connection all the way up into user space. We're gonna tackle problems like authorization. We're gonna tackle problems like synchronization. We're gonna tackle problems. Like, like, it just kind of goes on and goes on and goes on. And the, and the joke that emerged internally was. Uh, or a slogan, I guess is like n Engine X published by Postgres, right? Like we, we don't see this in, in the other space where it's like, you don't expect the nGenx team to ship a database. That would be ridiculous. Like they make reverse proxies and, and that's hard enough. And we really, uh, when we try to kind of reinvent the stack, when we try to get people onto a different fuel source, it, it really invites, it's very tempting to kind of wanna do everything to build the entire car. And we don't need to do that. We don't need to boil the whole ocean. We can just focus IROH specifically on this incredibly low configuration, incredibly reliable piece of technology that just does one thing really well, which is you get a public key and you can dial the device by public key. It now works like an IP address. And, and that's the way that we think about. Connections, and that's the thing that IRA provides. Everything else is a layer on top that's open to community to iterate upon. You're able to compose that however you want. Um, we have lots of people who have like, sort of as we've opened up that space, lots of people have come in to fill it, uh, in a way that I think has been very fun and exciting to see from our, our perspective. But that's what we mean by don't blow the ocean. It's just like, take this new approach to building peer-to-peer project. Can we just really have the one primitive that we felt that we were missing, which was just really, really, really good, really reliable connection. Um, that that's the thing that we are shipping and continually trying to improve upon. Everything else is left to, to the devices of the. Justin: Okay, so Iroh a project to help, uh, facilitate those really, really good peer-to-peer connections. Um, can you like kind of package up, like give us a sales pitch. What is, what is IRO like from the sort of like full, a full definition? And then like, what is the rest of the ecosystem around IROH that you're building? Kind of look like, we're gonna talk more about specific aspects later, but I just like want to give people, uh, a good overview of like everything that you're trying to work on here. Brendan: Absolutely. Uh, we describe Iroh as P two P quik. Uh, so if you know what the quik protocol is that is, uh, it has now been anointed, it's going to be HTB three. Um, a lot of the internet traffic runs on, on quik, and that sort of highlights a couple of things about IRO that we really like. Uh, a it's peer-to-peer and so it's, you know, fundamentally your technology is gonna be connecting devices that couldn't previously connect easily. Uh, and it is based on standards wherever possible. And so, like we use BOG standard stuff wherever we can, referencing IETF specs, wherever we can. And so the project itself is really intended to give you, at the highest level this really basic abstraction of every device that you want to be. Dial gets a public key. And that public key is then discoverable and dial from any other device in the network. We also lean into quik in one other way, which, uh, quik, quik spec leaves open this thing called the application level layer protocol name or Albin. And that's where we do all of our pro protocol plugging and protocols are basically like HTP middleware. You can compose 'em however you want, you can use them to do different things. And so some examples of that for us are like, we have ones for data transfer, and so just, hey, do I wanna take files, which we call blobs, and have that move around in a, in an easy way. Great. Uh, another one is for like, uh, broadcasting messages. So we can build these things called gossip swarms and we can broadcast messages to all nodes that are in a given topic. Um. And so like you can build, publish, subscribe patterns based on that and all kinds of different primitives. Um, and then all the way up into audio and video and, uh, some other experiments that are running. And then integrations with existing stuff. So it's also technically an Iroh protocol to take something like auto merge the project for CDTs, make and switch and have that run on top of virus, peer-to-peer networking stack. And that would be the auto emerge protocol for Iroh. Um, and so that's kinda the way we think about it and the way the developer experience generally is, uh, start project, create an Iroh endpoint and that's what gives you the note id add whatever protocols you want to add. And that's how you compose together the functionality that you would've done in HTP world to get whatever features you want to put in your app. Justin: So let's dig into some of the like, technical details for a little bit. Um, so you, you sort of mentioned Iroh, you're, you're making these like very um, like high success rate connections, uh, between two peers. Uh, what happens under the hood to sort of facilitate that? So I saw in the docs there's this like reference to Magic sock and, and you have like a hole punching approach, like what does all this mean and like how does it all fit together? Brendan: Totally, um, whole punching, uh, just for those who like don't know what that term is, is, uh, um, the whole problem we have here comes from IP VV four, right? The idea that, uh, what we have in all of our routers at home is this thing called network address translation, which I won't bore you too much of the details of that, but basically it means that your device sitting at your home, behind your home router doesn't have a public facing IP address that's really easy to dial. And so we have to punch a hole through the network's firewall, which, um, sounds bad, but is actually really good when you mean to do it. And, and, and that's the way that we can get direct connections on the internet. IROH takes a really specific, very modern approach to how we do this. Uh, and it's also pretty pragmatic in the sense that the emphasis here is it should work a hundred percent of the time, even if it's not actually possible for you to pull punch. If you are working in a, uh, corporate environment, often that's the one easiest place to see where they just won't allow this to happen. And so in those cases, what will happen with Iroh is you will still get a connection. You'll just be talking through an encrypted relay. And that encrypted relay will still be using quik connections over the hood. Uh, and the relay can't see the conversation that's happening. It can just facilitate it. And so it's no longer direct, but it does work. Right? And the really cool thing that's quite magical about this is using a combination of quik and a number of really difficult networking techniques. IRA will just dramatic, dynamically pick the right and best path through the internet for the connection as it's happening. And so you can imagine these classic things that often will trip up a nap today where you start, uh, with an app open on your phone, you start a request, and then you walk out of your house and you leave wifi range. And now all of a sudden you like the, you have to refresh things. IROH does not have that problem. We figure out which packets have not been sent, we back pressure manage those and like send it for you. And we will figure out, hey, you have a direct connection. If you flip on your VPN and all of a sudden your IP address moves to a different country, uh, Iroh can still manage that and keep those packets flowing through that and so's that ability to sort of negotiate changes in network topology as that connection is happening and keep it alive and not lose packets is the real magic. Um, so yes, we're trying to establish direct connection for you, but we're also trying to get you just generally a more robust connection between the two devices. Sometimes that's a server, right? But it's, it's useful in either context. Justin: Yeah, I think a lot of people listening to this will probably have experienced, uh, it's like you're on your home wifi and you like walk out of range and it switches over to, uh, like cellular network and you like lose connectivity or you like can't access something that you're previously accessing or, you know, whatever. It happens, happens all the time. So it's probably a much harder problem than it seems like to solve that. Brendan: It's, it's pretty gnarly to say the least. Um, if you want to see a whole bunch of jaded engineers who have chuckled this problem head on, just join, jump in our discord, and they'll, they'll happily regale you. But, uh, [00:14:43] Understanding Quik and Multipath Justin: So I, I wanna dig a little bit more into, uh, quik, so you're building on top of quik, like you had mentioned, H two B three quik is interesting. It's like a UDP stream. Uh, so like, not reliable, like the TCP reliable, uh, mechanisms. Uh, so packets can drop and that's okay, and you just like send more of 'em. Uh, so why, why quit versus like a traditional, uh, T-C-P-T-L-S stack? Brendan: TCP vice versa. Yeah. Yeah. Um, I think this is my favorite question, uh, to answer because if you look at the intent behind quik, it really mimics what we're trying to do in Iroh the court of sort of like base statement for the, for the work around quik. When it, when the project started was basically we have this problem inside of TCP, it's called head of line blocking. We get these situations where we can't actually have like nice independent streams. And so one big packet of data, it can interrupt that whole TCP stream and clog things up. And the reason that head of line blocking is a real problem is because the TCP protocol has ossified, we can't actually. Iterate on that design because middle boxes through the internet have through convention figured out ways to rely on the unencrypted parts of TCP packets. And so we now have this world where we like straight up can't upgrade TCP. It's not possible, right? And so what quik sort of said from the jump was like, alright, let's see if we can back up, redo this in a way that will give us the capability and capacity to iterate. Um, which in practical terms, it means moving the congestion controller out of kernel space and into an encrypted packet payload. And so, yes, this is the part of the podcast that gets deeply technical, but like, um, this basically means let's the core, the core thing that quik is trying to do is like, Hey, let's just encrypt it all. If it's encrypted, then the middle boxes can't touch it, they can't mess with it. And if they can't touch it and mess with it, then we can have more control over the way that data will flow through the internet. And that's the way that we're gonna build a better HT p in the long run. And like, being totally honest, quik today is not as fast. Like, I don't know if you've seen some of these like conversations where like TCP compared to quik in a data center, like Quik is not as quik and the. Reason for that is you have all of these kernel optimizations of TCP and so there's way less conversation between the kernel and user space when you're actually doing these connections. And so yeah, you're gonna get a faster connection, but now in a latency sensitive context, like pretend you're trying to get a phone to talk to a laptop in the wild quik is way better today. Right? It's a better protocol as we're talking, uh, because it's got this way better head of line blocking characteristics. You get this new trick called cheap streams where you can now say like, I wanna have a hundred different streams. And in a browser tab context, that's a huge deal because that's all of the bunker stuff that vet or webpac is fitting out for you these days. And so if you wanna do a hundred requests in parallel, over four connections, that's gonna be fine. And, and nothing's gonna sort of clog up any of those four pipes. And that for us is like this spiritual alignment between what we're trying to do in Iroh at the higher level. It's like we wanna take. The quik concept to its natural conclusion, which is like, great, and now that should work across any device anywhere and we should be able to do whole swaths of the internet this way, and we should pass that on to developers so they can build apps that do cooler stuff in the wild, right? That's the kind of core step function for why we we're sort of all in on quik. Justin: Yeah, that's wild. Uh, quik is definitely not ossified. Definitely moving fast. Um, there's like one I I, I noticed there was like, uh, one sort of thing in the spec that's like relatively new, but I feel like it's up y'all's alley, like multi-path quik where you can kind of have, uh, and I don't fully understand it, it's like a single connection, but it can happen like over multiple networks. Uh, like are y'all preparing for that? Are you planning on using it? Do you already use it? Like what is the state of the spec? I don't even know, like. Brendan: That's a great question. Uh, thank you for, thank you for this tee up. Um, so our team has been working on Multipath for at least six months now. Um, we have been grinding on it, uh, to the core of our project and it is the biggest thing that we are coupling with our 1.0 release, which is scheduled for the back half of this year, last couple of months of the year. In a nutshell, multipath is basically a quik connection that is aware of multiple paths to the internet and can make use of them both. And so if you can imagine if you had, um, an IPV four and an IPV six connection, you could use them both and you could use them both to just get the bytes as fast as possible from A to B. And so this is the idea that a single logical connection can actually leverage multiple paths at the sort of like sending packets layer, uh, through the internet. Now, where this gets really interesting for us is we mentioned those relays earlier, right? Where that is one path that data can take. Through, uh, the internet to talk between two Iroh connections, or sorry, two Iroh endpoints to use proper parlance, MULTIPATH would allow us the, what we're doing right now. Basically the, the current version of Iroh is all of that work is happening outside of the congestion controller. And so there's this thing that like is aware of multiple paths. It, it kind of figures out what's happening and, and quik is sort of doing its thing, but it's not totally aware of natively of this idea of multiple paths. And so what we wanna do is we wanna have multipath and we wanna label that for the relay and for a direct connection and ideally for relay direct for IPV four and I BV six, and have all of that flow through the congestion controller so that it's aware of all of these, it can leverage them all. And so there are worlds in the future where you could be sending some data over the relay and some data over direct connection depending on what's faster and have that just be like dynamically switching. Where this gets really interesting is, uh. There is nothing in the spec that says that we can't leverage different types of definitions of paths. Web RTC could be a path, and so we could actually have devices that can talk to browsers via a multipath congestion controlled system and have that all just be dynamic. Wifi aware could tactically be a multi-path path, which is this new spec that has emerged where you can finally, if every, if you've ever used airdrop on an Apple device, that is the core of the wifi aware thing. It basically leverages the idea that, uh, your, all of our modern devices have two wifi antennas and you can actually skip the router and just send data directly between the devices without going through the redder. That's how airdrop is fast and that's why it's faster than like other stuff that exists on that. Apple has a proprietary protocol for that, but recent European regulations have forced that to be uniform across Android and iOS, and so there's actually a world where Iroh can be aware of a, a path between an Android device and an iOS device. Using a combination of web RTC, ipv six or encrypted relay and wifi aware, and it will just figure out the right connection, do the right thing, and get you the fastest possible sort of, um, connectivity between those two devices. And all of this is under an abstraction. You don't have to think about, you just dial you think in streams. And all of that gap that I just mentioned can just be stuff you heard is cool on a podcast and you never have to worry about it. And that's, that's kind of the, the sort of core reason why we really think Multipath is like there, there's some like congruencies here that are starting to line up now that we've been at the project for a couple years. And it's just really fun to sort of see things like this line up where the spec is highlighting this thing that we've really wanted to do the whole time. Um, implementing it is hard. Opening and closing pass is really gnarly. Uh, doing it in rust is, is even more fun. But, uh, that's kind of where we're at these days. Andrew: Um, [00:22:02] Content Addressable Storage in IRO Andrew: so one of the cool things, uh, about Iroh is that it uses content addresses rather than IP addresses. Uh, this hearkens back to your IPFS days where an interplanetary file system, it's not gonna work with just normal IP addresses. So, uh, why, what are some of the benefits of using content addressable storage, and what have been some of the challenges? Brendan: Totally. Um, I should highlight that it's, uh, optional to use content address storage. Most of our streaming use cases like don't actually use it. Um, because streaming is really, it's really hard to hash a stream. Uh, I dunno if you've ever tried. It's really, really difficult. Um, but, uh, content addressing is basically that you, you take the content that you wanna move around, you hash it, and you refer to it by its hash. That's content addressing. And when you refer to something by its hash, what you can do is this magical trick. When you get it on the other side, you can calculate the hash of the thing you got. And if they don't match, then you know, somebody lied to you. They gave you data that you didn't ask for. That's a really great primitive for moving data around without being able to trust, without needing to trust the origin. Right. And this is a lovely sort of like primitive that allows us to say, Hey, we can get data from wherever we need because we know the hash that we're looking for. Now hashes are super ugly. They're not human readable things. And so we have lots of, this is kind of where we were really excited to ship. I was a library where the. We don't think it's a, a win for in the UX space for users to see hashes. Even those of us who are used to using GI like day in day, we, we think in branches, we don't think in commit hashes. Like really only in like times of desperation are we passing around, you know, the hashes. Um, and so we think that that's like kind of the right approach and we work a lot with projects, um, that are building on top of IR to say, Hey, how do you think about content? How do you label things? And let's get, use that as the system to do pointers to this really powerful primitive of content address, data transfer. Um, we have a really, really, really intense version of that built on the Blake three hashing algorithm that allows us to incrementally verify data as it's coming in down to the kilobyte. Um, and we do all kinds of stuff in there. We can also take data and stream that in from multiple providers. And so you get this classic like, Hey, I know five people have this data set. I want, please cut up the request across five, uh, people and just fetch it incrementally from the all rebalancing that as you figure out who has what or who has a faster connection. And so like a lot of that is sort of handled for you. And the idea in the I ecosystem is you just like, cool, let's just use alos for that. We'll set it up. And that's, that's the same as, it would be as like installing OAuth in an HT P app. Right? You just like bring that in, wire it up and get to work. Justin: Is that sort of on the same layer is you, you had mentioned earlier that there's this, like you're using public keys to identify, uh, like connection points other than IP addresses. Is like the public key dialing the same as like the, you know, connecting to like a piece of, like a content addressed, uh, hash or is that, or are those like different layers? Brendan: Distinct layers. Yeah. And so you, the connection always is always between two public keys. And so in this world, because we're peer-to-peer, we don't have, traditionally, we don't have TLS certificates, right? You don't have like, Hey, I'm a descendant of this origin domain source. And that's, uh, and so instead what you do is you just create a key and you say, cool, this is my key. And every. Uh, packet is encrypted with that key. And so this is, uh, if you're familiar with MTLS, which is mutual TLS, this is kind of in that flavoring of things. And we actually do use TLS under the hood, like the real version of it. Um, and we're using Edwards Curve keys to sign things. And so whenever you're having a conversation in, I, it is always not ally, end-to-end encrypted using those, uh, keys. And the nice thing is those keys can then become primitives for building author authorization schemes. And so you can say like, Hey, I'm gonna, I have this user profile and this user profile possesses the following keys. That key is my phone. That other key is my laptop. And now when you talk about them in that way, every single byte through that connection is authenticated. Right? And that's like a really exciting jump off point for authentication. But that is a lower level primitive than the content address stuff that's, uh, up at the protocol level. And so you, you would pick that, you, you would be just as, uh, justified using like, I wanna do video streaming instead and just, no, no content address, data transfer, just video streaming. That's totally a valid use case. Justin: Gotcha. Cool. That makes sense. [00:26:14] Establishing Connections in IRO Justin: Um, so maybe just like to form sort of a holistic picture of what you're building here. Can you sort of just walk us through what happens when there are like two devices that are connecting on Iroh? Like everything from like the initial handshake to, you know, maybe, uh, establishing the connection or like falling back to a relay. Like what is, what is the path, like what happens? Brendan: Yeah, totally. I'll, I'll, I'll do the fast version of this thing because the, the, the slow version will take an hour and a half. But, uh, uh, in summary, what happens is you are gonna get some instruction. Uh, we will start with the most interesting case, which is just. I want a connection to that key and we'll, we'll call that Bob's key. We'll use the classic Bob and Alice nomenclature. So I'm gonna be Alice, I'm gonna try and dial Bob. Bob. Somehow I got a key from Bob. Usually this is, and this is where we are not like a classic peer-to-peer project. We think it's completely valid to put Bob's key in a database and like get that as an API endpoint. That's totally fine. So you went and you did an API call and you said, okay, the app told me to go dial Bob. And Bob has this key, you now need to discover the IP address of that key. And so the first thing that will happen is some sort of discovery system. Iroh has a number of discovery systems built into it. They're pluggable, uh, some of them actually will tell you about, uh, keys that it's found. Uh, and so that's like MDNS. And so if you wanna just do local offline only, I only wanna do local network connectivity, that's our MDNS stuff. And it will actually tell you like, Hey, here's a stream of keys that I'm finding. But in this case, let's assume that the common case where Bob is actually a device that is not in my local house and is instead. Across the country. And so what I'll do is I'll first go through a DNS discovery service by default. Um, and so I will actually take Bob's key. I will do DNS Iroh link, and I will look up a literal, actual DNS record. And that's the magic from now mapping a public key to a dial bowl set of details in the background. Before this ever started, Bob went online. So Bob's endpoint connected and Bob put a DNS record on this service. Um, Bob signed a packet that said, Hey, I'm Bob. I've gone and talked to the Relay server. I know that my public dialing details are here and the my home Relay, the relay that I use, if you wanna talk to me, is this one. Um, and that'll be, usually those are geographically named, so that'll be like US West and they'll be like, cool. Bob's Home Relay is US West on the Public Hour Network. And, uh, Bob will put that in the DNS record. My lookup, uh, as Alice will discover, Hey, I wanna talk to Bob. Bob's IP address is x and Bob's Home Relay is y. I will then create a connection to the home relay and send the first packets. I will just start literally sending the data that I'm trying to send to Bob. And so that'll just go, and so like, what, what is happening here is we're trying to get the fastest time to first bite that we possibly can. And so that data is going to, like, we're just gonna start se sending data. That first packet is always gonna flow through the relay because the Relay knows how to talk to Bob, right? And so Bob's gonna get that packet in parallel, we are gonna start Interactive Connectivity establishment, right? Which is the whole dance. If you've ever heard of Stun Turn and Ice, if these, if those concepts are, uh, familiar to you, I'm sorry, you've had to learn about hole punching and, and, and your life is a little bit worse for it. Uh, if you haven't heard those concepts, don't worry about it. What's happening in parallel is a direct connection is being established, and the moment that we have a direct connection packets will just start flowing over it. And so this gets you a really nice, fast time to first bite through an encrypted relay. And gets you all of the niceness of like, cool, once that thing switches direct you, you'll just sort of actually see it happen. Um, and so that's kind of the core of how you start sending data to Bob under the hood. That that primitive that you're working with, the thing that you're trying to send is you're just opening a quik connection. This is the same as if you were trying to dial through quik. And so you usually would say, I want either a unidirectional or a bidirectional stream. Uh, bidirectional is gonna feel more like a web socket. Unidirectional is gonna feel more request response. Um, and that's gonna actually, it's not totally true. Unidirectional just means I just wanna shoot data at you and, and you can't respond. Um, but yeah, that's kind of the, the core premise. You're not thinking much about UDP, you're not dealing in the like, lower level gac. Um, we can, you can really tweak that and set it up to be lossy if you super want to, but like the experience that you're sort of getting at the high level feels way more HT P like, Hey, I wanna take this data, I wanna send it over there. Um, but like, just like the low levels of HT p, the parts where you're like actually, I guess TLS to be super, uh, specific. Justin: a lot of sense. Uh, so you have to have that initial relay connection. That's like, uh, my, my understanding, just like quikly grant glancing into the dogs, I thought that the relay connections were mostly just for fallback. Um, if you couldn't establish a direct connection, uh, but so that, that initial connection does happen through Relay, and then it establishes the, the direct connection. Gotcha. Brendan: yeah. You can now Ira is totally able to do them just no relay. So like, it, like when you discover peers locally, that conversation never crosses the relay 'cause you have a direct connection. Like why would you? Um, but yeah, in like the majority of use cases, I'd say, and now just to give you a sense, are very upset if ever more than a kilobyte of data is sent over the relay in the, in the process of establishing a connection. Like that's, that's something is wrong. And so like, usually it's first couple of packets and then we're offered to a direct connection. Like, this happens very fast, but this is the modern internet. It's really critical that, that, that initial data flows. As fast as the latency is. And so that's how we guarantee that, right? We want this to be competitive with existing normal non fun fi peer-to-peer libraries, which means subsecond connections, which means none of this like, hey, waiting that you would normally experience. And so that's why we always flow through the release at first. It is the, it's the reliable path and, and that we use both as the fallback, but the thing about our connections is they're constantly transitioning based on needs. Justin: and I think this light really highlights the practicality of what you're doing. You know, especially see a lot of conversations on Blue Sky and Macedon or whatever about like, what does it mean to be decentralized and like, you know, a lot of like, uh, just. Pedantic arguments about like, how these things shape up. And, and you've taken a very practical approach here. It's like, okay, we like want to facilitate this, like, these like direct connections in as many cases as we can, but like, we also don't wanna sacrifice performance to do so, or like, you know, make it, you know, incredibly difficult for the user to like, use. Uh, so I like the, I like the, uh, practical trade Brendan: Yeah, like I, I, I built on top of a peer-to-peer stack for like five years and we had this one giant if statement at the bottom of our stack, and it was basically like, Hey, does P two P feel like working today? If yes, let's use that. And if no, just use htp. And like that's a nightmare, right? Like, that's like an objectively not okay. Like you need to be able to use Iroh in 100% of context. Like it needs to be a reliable communication channel, right? And if it's not, then we're just creating problems for people. We're not actually sort of like helping move the ball forward. And so it really is that if statement that we've been like trying to attack for the better part of three years. Justin: Uh, [00:33:15] Challenges and Solutions in Relay Servers Justin: so we, we kind of covered it in this, um, but I, I'll sort of ask this question anyway and we can kind of transition. Um, so you're using DUR servers, I think Durp is, uh, so design encrypted, relay for packets or something that was by like tel scale designed that or made that. Brendan: Yeah, originally, yep. Uh, we started there. Uh, we no longer actually Oh, okay, as, as spect. Um, and so it's, it's been, that's just more of a technical detail, um, and is a byproduct of us going all in on quik. And so now we actually do everything inside of the quik protocol. And so like, yes, we have an encrypted relay, but like this is, uh, in many ways it just looks like a quik connection. Justin: yeah, yeah. Yeah. I was gonna, I guess I was gonna ask sort of how you balance, uh, centralization with, you know, reliability in terms of relays, but I guess we, we sort of covered that. Uh, it's like established initial connections and then fallback if, if you absolutely need to. Brendan: yeah. And we, we, there's one detail we didn't cover. We made a very specific choice to address the relays by URLs, uh, and to publish the relay code in addition to the Iroh connection code. So there, we both pieces of the software are sort of available as open source that you can spin up your own thing. As, as you'd like. I'm with you. I sort of like, I the sort of decentralization, hand wringing. I'm like, I'm, I'm kind of like academically here for like an online mud fight, but like I, in practical terms, we, we really just want it to function before, before you have the debate about the whole thing. And so, uh, at best I always federated because it relies on these relay servers, right? The, uh, but you can deploy your own network and almost all of our big production users have their own relay servers. Most of them are, they start off on the public relay servers that number zero runs and then they sort of like graduate to their own set and they sort of build their own. Uh, we have a service where we run that for people, but like we have, it's also just open source. And so we work with a bunch with projects that will do more exotic stuff like fuse a relay server with some of their existing infrastructure or sort of run that as. Just like the others who will hybridize where uh, the public number zero network is a fallback. And so like then now you have like true federation across organizations in that sense. Um, and so yeah, we can still have the decentralization sort of like blathering match, but it's, uh, but it's also like, you know, that, uh, at the end of the day we really focus on just like, can we get the thing to function? Andrew: speaking of functioning, you guys have boasted some pretty high connection, uh, rates. I've seen 200 k thrown around for active connections. Uh, what have some been some of the biggest challenges getting to that scale and, uh, what have, what surprised you about real world usage patterns? Brendan: Every single app that we have worked with. And every single like developer team that we've worked with has like a dramatically different story around their network. And it's been wild to sort of like experience that. Um, just to give you some like examples. Yeah. Like we, we've worked like everybody in the streaming video space. Often has just like a silly number of concurrent connections, like a bonkers, like the liveness property of apps that have voice and video chat built into them. Really, like, just like it, we very quikly had to scale the relay service to be able to do a million concurrent connections. Um, like that was a, that was a prerequisite for working in that space. In contrast, some of the more data intensive stuff we've been doing, a lot of work lately in and around the AI and uh, like AI training, uh, training of actual LLMs, and they really exchange like a lot of data in these bursty rounds. And, uh, so like each node will be producing like. On the order of tens to low hundreds of megabytes of data that needs to be broadcast to everybody, but in very specific moments, and like we ran into this pattern problem, we're like, oh shoot, we had way too much data actually flowing over the relays that like one kilobyte thing I talked about earlier. We had like, these machines are so high powered that they were able to jam tens of megabytes into the, into the pipes before anything could even happen because they're really high powered machines. And so like we just had a completely different deployment pattern and usage pattern. Uh, and then like also just like stuff that showed up outta the wild that we really didn't expect as kind of these off-label use cases. We're getting a lot of people using Iroh now even in the cloud because it has service portability, right? You just take that key and you'll move it around anywhere. Some of that could be running on GCP, some of that could be running on AWS. Some of that could be running on Hener and you just talk to services by key. As long as you like, you can get the throughput that you need. It's a really convenient way to get portability in the cloud. And we're all of a sudden, like, that's not like peer-to-peer. Like we don't have hole punching problems. These are publicly dial devices, but like oddly, a lot of the folks we're working with do have hole punching problems. And a lot of it's like configuration of the firewall and your VPS and like all of this like sort of like infrastructure G of DevOps that Iroh had really helps glaze over where now it doesn't really matter if your app is being deployed to an employee's laptop or to a cloud, um, vm. They both just sort of like get keys and talk to each other. And that has made for like a really interesting, um, sort of usage pattern. So we've seen some wild stuff. Like it's, it's been absolutely wild scaling this thing for real. [00:38:18] The Future of Peer-to-Peer Networking Brendan: Um, the last thing I will say on that topic though is the coolest bit about peer-to-peer networks is the scaling characteristics. You can get this thing called subline scaling outta peer-to-peer, where it is genuinely possible to run. Of millions of devices on a raspberry pie as the relay server, right? Which is just like a bonkers thing because it, you have to kind of invert the model of thinking. The client is the server. Now it can do server things. And so every time you onboard a new user, you're adding a server, you're adding capacity to your network. And that when those people are talking over direct connections, that has no usage bill to you, the service, uh, provider, right? And so as the dev, you are able to get bonkers scaling characteristics. And that is just like we, that is, uh, formerly known as subline scaling, which is the idea that as you add capacity, the total cost per unit goes down. And like most of the time that we talk to people, they sort of like, when they think about peer-to-peer, they're like quietly in the back of their head thinking of like bit torent. And they're like, well, that made my computer hot. And, and we are, we are well beyond that world. We're in a world where like the amount of traffic that a single node in some of these apps will do is smaller than the ads you would be downloading in another app, right? Where it's just like the, like you can do one request an hour, and if you have a million users doing a million online at a given point. And they all do a request every hour. That's a million requests that you could serve, right? And so the capacity, just the scaling characteristics are completely different from anything that you see in a normal sort of space. And that's where the real magic shows up. And like, sadly, with peer-to-peer networks, and I think this is part of why they kind of die, is like they, you don't get to see that benefit when you're like 10 nodes, you know, just starting off and you're, you're building an app from scratch. Where that magic really shows up is when you hit some of these scaling moments, and if you're prepared for it, you all of a sudden just watch your costs level and they no longer go up. And that's like a really wonderful moment. But like, the number of teams that get to see that moment are quite low. And so I think that you don't see a ton of advocacy for peer-to-peer in the wild because it's just, it's really hard to get there. Justin: Yeah, that's wild to think about it. Maybe like when I, I guess when I was first thinking of like the network topology, I was like just thinking like, you know, traditional is like, oh, Bob wants to contact Alice, like Bob Con contacts the relay. The relay helps establish, establish a direct connection to Alice. But like, there's something that I didn't think about is like, if there are like multiple people in the channel and like someone else is already connected to Alice is like, is it possible to hop between them? Is like, I mean, are you doing like these like sort of dynamic typologies or is it really just trying to, as much as you can just establish a direct connection between the two users? Two devices. It's, and this is where we have lots of space to iterate. Um, and so some of our protocols do actually have that hopping characteristic where things will move through peers and some of them don't. Right? Uh, we have leader election algorithms where like, Hey, you wanna run raft consensus? You wanna pick one node? Brendan: You want it to be the sort of like driver of the conversation. Um, and so, uh, this is the fun part about stopping boiling the ocean and saying, no, no, let's leave that open as a design space. All of those, even if you're doing hops between nodes, those are still pairwise connections, right? And so you, the primitive you need. At the end of the day is being able to dial one device to another. Even if you're building a broadcast system. We, there are, I won't bore you with the academia, but like we, we, all of our connectivity on the internet is based on this, this device talking to that device and, and, well actually, I can't really specific this port and that port is sort of discussing, uh, something with each other. But yeah, it's, uh, and I think you, you're sort of like at the doorstep of where, of why we got into this game in the first place, which is really we're, yes, we get like peer-to-peer is the way that we express our goal, but we're really, we think about ourselves as a user agency company, not as a peer-to-peer connectivity company. If you kind of like, just to get a little meta for a second that I don't know, are y'all familiar with the dead internet theory? Like, is this, um, just to like quikly elucidate that this idea that like the public internet is like kind of dead, it's just full of robots and like people that have like really kind of made it very difficult to know if you're actually talking to a person, if what the motivations of that people are and what we're sort of seeing. Whether you come from the very privacy conscious side of things or where you're just like kind of wandering through the internet these days, wondering whether this is AI or not. You're experiencing a bit of a loss of agency, you're, you're experiencing this. Like, oh, you know, I can't, I can't do all the things that I used to, to do, and I sort of have lost a degree of control. And, and a lot of that is really characterized these days in, in terms of like, hey, being more privacy conscious and like de Googling and all this stuff. Sure. But like, I think in practical terms, what a lot of people are doing is they're just retreating into the cozy web, which is like discord channels, web, WhatsApp groups, and like signal and whatnot. And that's, that's you claiming agency back. You're saying, Hey, I wanna be able to like take control over who's part of this conversation and how this is sort of like working between people. You're seeing this in blue sky at the top level of this, where it's like, hey, we want to be sort of like really concerned about who runs and controls this, these public systems of dissemination. But the reason we care about. P two P is our core thesis is that if you wanna take that conversation to its natural conclusion, those cozy conversations, those discord channels are really where the internet is going. That's where we're all going to be sort of like building and working and iterating. And so that means the world's gonna look more like signal, like the internet isn't dead, it's just changing. We're changing the way that we talk to each other. And so we need systems for building stuff that looks and behaves more like Signal and less like Facebook, right? And for that, we really need to kind of come up with this like missing primitive, which is like, how do I have an encrypted end end connection? So the person that I'm talking to, right, specifically, and we think that's kind of the step function change that we're trying to anticipate and be a part of. And we think this like electric engine is just fundamentally missing from the networking stack. We need to be able to do this stuff. We need to be able to construct these apps that know how to dial our friends without ever dialing some home server, right? And like whether, what developers choose to do that with that, it's totally there. That's like up to them to decide how to, how to use that technology. But for us, we're like, this needs to exist and it needs to be robust and it needs to be sort of on the level that is comparable with HT P. And that's, that's kind of how we, why we're doing what we're doing. We're really sort of like trying to facilitate this set of tools that will allow developers. To build that next generation of apps. It has these incredible scaling characteristics. It's possible for some kid in a college dorm room to ship an app to tens of millions that will cost them on the order of hundreds to thousands of dollars. That's a really cool byproduct, but it also means that that person could, could ship an app that's competitive with big incumbent things because it is both lighter and ships more capabilities to the end user's device. And so that's the internet I wanna live in. And so that's, that's the sort of change that we're trying to facilitate where I can have, not necessarily more privacy, but just more control. Um. Justin: Yeah's strong echoes there with the like local first movement. I mean, so we ran in each other at local first conference and, and that's been a big conversation in that space is like agency and control and connection. You know, it's like local first, not local only is the thing that I'd like to say. It's like, it's still important to have connection, but like sort of maintaining some agency in that space is important. So I, I like your, your framing on this. Um, and I think a good transition here is like looking at local first apps and edge computing. What do you see are like the next big opportunities for peer-to-peer networking? Um, so like, what is the opportunity space here in the next few years? Brendan: Yeah, and I think, I think to like build on that sort of like long-winded rant. I think if you like really zoom out and go back, uh, at like what I. What the internet sort of like did to us in the beginning is like we initially got the ability to associate according to interest instead of place, right? Like if you think about the way that why the internet was really popular all of a sudden, like you could talk to people that shared your interests, not just people who lived in the same town as you. And I think now with this like cozy web conversation, thanks Maggie, who is a friend of the local first movement, uh, who Maggie Appleton is the author of the Cozy Web article that I'm drawing a lot of this from. I think the cozy web is actually the reinvention of place on the internet. I think we're in this space where we're now saying, Hey, actually we want some of that place back. We want a context where we can say, I know who I am in that space. I know my identity and I know who I'm talking to. And that can feel like a familiar watering hole. And to me, that reinvention of place and having that be driven through direct connections is the real opportunity, is the moment where it's like, okay, can we ship apps that like. Kind of feel more like chat apps, but can bloom in functionality a lot and can all of a sudden say like, oh cool, let's just like flip in some video voice chat in there and like, let's just like add some like new sort of like, let's jump out of this telegram and into some game that is only a part of this thing inside of this cozy placemaking space. And that's where we like really wanna see, um, iteration improvement. I think that's where we see a bunch of UX and some of these networking primitives really interacting in interesting and meaningful ways. Uh, and so we're, we're really gung-ho on that space in particular. Um, and we think that that's the most, and like that is like straight up local first, right? Like we, I went to local first coffee 'cause I was like, this is, and came back from it. Super excited that community's talking a lot and struggling a lot with like sync and like, that's like a core sort of challenge right now. Um, and I think a, in talking with a lot of the local first folks, it's just like, well, if somebody could ship us good peer-to-peer, that would be nice. And like, and this is part of where we were like. Excited about this evolution. There's a maturity now that wasn't here five, six years ago, where like people really wanted to invent that whole stack themselves, really wanted to do all the peer-to-peer networking themselves. And like it kind of didn't work well and like, and it, and it ended up being too big. You spent too much peer innovation budget. And so now I think you're seeing this modern approach, particularly sort of like typified by the local first movement where we are now, now back in a space of, of composition where we're willing to work together. And like our team is just like kind of hard limited on this. Like, cool, we're gonna do the connection part and everything else is gonna be collaborations. And that's, and we have other people who are like, Hey, we're doing the sync part and the networking thing is not our problem. And like, to me, that's really where we start to see that accumulating to a benefit in the end user in the long run. Um, and so, yeah, I, I really think a lot of things rest on these types of communities. Local first is a great example of it, getting their act together, shipping things that are usable. At scale by like the massive TypeScript community and like really making sure that that is like, we're putting good tools in people's hands that are grable. Um, it's really hard to take P two P quik and turn it into a TypeScript, API that's easy to consume and that works in the browser. And like, but that's, that's where we need to go, Andrew: There's, there's this drama going on in the gaming community right now also where it's like gamers wanna be able to like, own their games past when game companies deem those non-profitable anymore. Uh, we, we live in an era where like, I'd say probably the last 20, 30 years, it's just gone. Like, there, there's nowhere for it to go. Like it's either held up in a company or it's a service that somebody doesn't wanna run anymore. So I see us moving towards this more peer-to-peer thing and like being able to compose these things and ship those things. Maybe it'll prove longevity for those things. 'cause it, it makes me immensely sad. Like when TikTok was gonna go away, it's like, that's like a slice of culture and like history that's just like gone because of how we build applications right now. Brendan: And I think, I think you've really hit the nail on the head, right? Like, I think the modern way of shipping an app is to ship half an app. Like you ship the, you ship the front end, right? And, and we call them front ends because there's a back of house that you don't get, right? And, and the core thesis around peer, like forget peer-to-peer, all we're saying is ship the whole app, right? Like put the client and the server in the hole in the same binary. Yes. It's hard. You don't get the web platform development unless you wander through a whole bunch of Wasm G and, and we're working on it. But like, at the core, the, the game server is, is like such a great example. We, we have a couple of projects that are sort of like really building on top of this. Uh, uh, I, I would shout out, uh, jumpy, which was the first folks in, uh, bones, a game engine in, um, in Rust. Uh, we recently, uh, a community member contributed a a go dot or. Uh, engine plugin and like this, the core thesis is just like, I don't want the game server to go away. I want, I want to, when I buy stuff, I wanna own it and I want it to function so that like, hey, and like you saw this in the gaming community, like, can I have land parties back? Right? Like, you remember how like we just can't have land parties anymore. It just ridiculous that like, we can't all sit in the same room and play the same thing. Right? And, and we have a whole bunch of people in our community crawling through grass to like, like someone just shipped a UDP socket layer on top of dumb pipe, which is this Unix socket thing that, or this, uh, socket thing that we've built specifically. So they could play counterstrike with their friends. Like, it was just like, that was what they wanted. And they were like, can we have all of this machinery just fake a UDP socket so that I can play counterstrike with my friends? And like, yeah, it's weird that a whole generation has come up, only consuming half an app, right? Like it's, it's, yeah, it's a bonkers notion. Justin: You wanna do the next one, Andrew? Andrew: sure. Uh, so, um, building P two P apps is hard. What advice would you give to developers that have been burned by this type of technology in the past? Brendan: Oh, the, the jaded folks who have said, don't do it that way. You'll never make a dime. I, uh, I, you are right to feel that way. You are right to be upset the last time that you tried this stuff. Uh, this time it works, I promise. Um, the, the, but I think that it's this boil the ocean thing that we really have to get past, right? Like there are use cases for IRA that I wouldn't recommend today because we don't have robust, uh, protocols built to do those. Like I wouldn't try and kick off like a really great streaming video thing, unless you're like really pre prepared to do some innovation budget today. A lot of the streaming video projects that we work with have like really sophisticated dev teams doing all the video stuff and they just need the networking layer. Um, and so like, I think this is, we will kind of like work more directly with the community to uh, sort of signal when we think certain use cases are opening up right now. If you wanna transfer files, if you wanna build chat apps, if you wanna build any of these things, like we are there, like we are in a great place. You wanna do voice calls, like just straight voice calling. That's kind of the latest thing that's making major progress that you could probably kind of clu together. But I think there are three layers that everyone needs to be aware of. There's us at, you know, actually there's four. There's the operating system development people and the like sis call development folks. We don't talk about them much, but they deserve a shout out on top of them are like. Primitives people, people who build databases and networking tools. That's us. Um, but that's also, that also includes the entire database community. And then for us in particular, one layer up, these are the middleware developers. These are protocol developers. These are people who are going to take our bonkers rust stuff and usually they're the ones that are putting grable workable APIs and getting you up into a higher level language. And so I don't expect that people working with Iroh in the long run will be touching rust. My hope is that they're actually touching TypeScript and the way that they're touching TypeScript is a really great protocol developer. Has said, well, hey, I've identified a whole bunch of people and they all wanna do local first development. They want statey, they probably wanna do like a little bit of video audio calling, but they need like a really good off story. And so you're getting these stacks that feel a lot like, you know, the, the tan stacks a lot like sort of the next JS of the world that have like, thought through the use case and or, uh, super base is another example where it's like we have an API drawn up that is really tailor made to what you're trying to do. And that's where I think most app developers should be consuming Iroh is not actually like, I don't, you know, I eventually, if we're, if we do it right, it'll just like say, like, Iroh inside or I, I'm sure there's some trademark around that, but like, you know, like similar ideas where it's like ideas. You'll know that there's Iroh down there somewhere and, uh, some really great protocol developer has done the work of taking IROs primitives and turning it into something really useful for you up higher up the stack. If you're the type of person who's comfortable writing distributed systems, who is interested in rust. Yeah, you should be, you should be one of the sort of like small thousands of people. That are working directly with us, and we are constantly shipping tools for you to do invariant testing and metrics collection and checks and balances on like, Hey, how well does this thing work when there are 5,000 nodes and 20% of nodes churn? And, and like a lot of these really difficult distributed systems problems that we cover in great detail with the protocol developer community. But I think if you're looking to take advantage of this stuff, I think you should just start looking for frameworks that include IROs a dependency, and those are the ones that you should try to consume. I'm not recommending that everybody stop and learn rust so that they can take advantage of some of this like fanciness, right? I instead think that we should be focusing on this composition over, uh, boil the ocean approach. Um, and so there's lots of like really great projects that are, uh, that we are constantly working with to sort of like deliver some of this value to, but like my hope is that. At maturity, you're not seeing, you're not having to like consume rust directly. Like it's, it's a great language. It's critical that we write it that way. Like Ira runs on an SP 32 and that's only because there's no garbage collector. And the whole thing is memory safe. And like we have, like, we put a lot of time and energy into making this a really robust library, but consuming that like, uh, it would be a lot nicer if you had just like HTP fetch as the API that you worked with. And so like, that's the kind of thing that we wanna see. Um, and so you really, you just gotta pick the altitude that you're working at. If you're a distributed systems engineer, we wanna talk to you yesterday. If you're an app developer, we go and go and talk to your framework and say like, Hey, do you do peer to peer? And what's, if not, how would you go about doing that? You know? 'cause for a whole bunch of 'em, it's not even possible. They'll rely on specific primitives and, and concepts that are just like inherently. Centralized, which isn't bad, right? We can pluralize those things, but, um, you don't, uh, there are places where this will go. You'll get like 10 x the benefit because you're building on A-C-R-D-T and all of a sudden ctts just naturally work everywhere. And, and they have this like, lovely property and great, you, you get a win because you picked the right tech stack and they hit the right maturity. Um, but yeah. So that was kind of long-winded, I dunno. Justin: That was great. No, that, that was fantastic. [00:56:46] Building Sustainable Open Source Businesses Justin: Uh, I, I have one question that I wanna ask that we, we normally talk about. So there's a lot of components of Iroh that are open source. Um, and we talk to a lot of founders on the podcast about building sustainable open source businesses. Um, and I, I kind of want to talk to you about the same thing and, and there's like another thing that I think you overlap with. You, you, you said like you're at the infrastructure layer layer and you consider like the protocol people to be like a layer above. But like a lot of the protocol companies that I've seen really struggle to survive. And I think, you know, your, your, your past working in the P two P space and looking at PFS and stuff, there's like a lot of dead companies in that space. So how are you thinking about the. Business of Iroh and like you, you're building all this great technology, but how do you make it sustainable and how do you like, you know, ensure the long-term viability of this like thing that we obviously really deeply need. Brendan: I'm so glad you asked. Um, some people like don't have the guts to like, ask the money questions and, um, my, my job is, is largely the money questions. Um, my favorite anecdote, I I, she's an investor at RedPoint Capital, she wrote a blog post, I can't remember her name, but, uh, I wanna attribute the quote to her that she said, doing an open source startup is like trying to hit two home runs in the same company, right? You first have to like, get. The open source project to be successful, and you have to get it adopted, and then you need to figure out a monetization scheme. And like, I've been having this like lovely, uh, debate with PVH for making and switch where he, like, he had a, a great rant from the local first, uh, conference where he was like, the metaphor of building an open source project these days is like you bake a bunch of pies and you wander down to the local, uh, farmer's market and you put all your pies on your stand and you put up a sign that says $0, and you give them all away. And then you go home and you say, well, why didn't they make any money on my pies? Right? Like, everybody's very happy, they're giving it all away, but like, we haven't, you know, I had a business card stuck in my car that sort of like said that like, oh, I would, I would do services for you if you asked me nicely, but I never bothered to show anybody that. And I think that's what this, like both of these things, this like two home runs and uh, metaphor around like just naming a price for the value you create in the world is the necessity of like really getting the split vein modality right. In open source, we fundamentally are exchanging value in a way that does not pass through dollars, right? We, we don't, we wanna just give our software away. We, that's the way that we feel, uh, creates the most benefit in the world, but we still need to turn our code into food. And, and, and to be able to do that, we have to pass through dollars. And so, um, we've, with a networking stack, that challenge is quite. Uh, acute because if you can imagine for a second, uh, a like commercial license for HTP, you're kind of dead in the water, right? Like, you, you, no one's gonna, like, no one's gonna put that in their stack. And so we've taken a different approach, um, that is much more around the, hey, this is peer-to-peer is really hard to hold, right? And so we have a, a complimentary service called Nodes, N zero Ds, nodes I computer. And for 5,500 bucks a month, you get a direct connection with the core team and we will make sure that you're holding Iroh, right? We'll spin up a network for you so all the relays run for you. And we'll do Bluegreen deployments for you. We'll, we will collect anonymized metrics for you and we will analyze that traffic and understand where it is and isn't working. And we'll work directly with your teams being like, Hey, this new protocol just hit VO 90 'cause we just did this little release. You should bump to that. You should change this, you should hold this differently. And that, uh, we have, we have meaningful revenue and a meaningful growing customer base. From that, the products are distinct. The, uh, num Iroh is an open source project. All aspects of it are given away, including the relay servers that we're running on the commercial side. But on the commercial side, you are getting direct access to the team that is building it. You are talking to us in a way that we worked really hard to get that number into a uniform box where we're not like sizing you up based on the kind of company that you are and trying to charge you something different based on who you are. Um, and that for us has been like really hard to get right. It took us a long time of like actually just doing something that looked a lot more like high powered consultancy, where we were working directly with companies just to figure out lighthouse customers, Hey, you're trying to deploy this. How does this work? And we've been scaling down the cost of this. In the next six or seven months, we're gonna launch something that looks a lot like a versal for peer-to-peer, where we will take your code, we will build your code, we will test your code in a, uh, simulated distributed system. We'll enforce all of those in variants for you. And then when your code passes and you merged domain, we'll deploy that around the world and those will be running and those will form the backbone of your applications network. And we're gonna do that sort of for like on the order 50 bucks a month. And the theory there is, hey, we should really be kind of like leaning on this idea of being able to deploy these services. Now that product is really pointed directly at the protocol developer community. It's not necessarily directed at the, like, I wanna deploy an app community. These are people who are really familiar with the notion of invariant testing. They wanna sort of assert that like. Regardless of the amount of churn, the following should never happen. Uh, or like the, uh, pathological cases, the network should never do X, Y, and Z. It gets really complicated, right? But we have built systems for testing that internally. And so we're just taking all the tools we've built to make Iroh and we're pushing them up the stack and we're making it like, okay, cool. We do all of that today through collecting telemetry and defining the inv variance over telemetry. And so that has been like a trick that we've learned that we can sort of now fit into this uniform interface that will get you a push to deploy style feel, uh, and will hopefully allow us to iterate a lot faster. One of the biggest things that we see in peer-to-peer development is just like, it's slow, is all get out. Like, like, like I, I I I, I love maintaining the Iroh computer website 'cause I just merged to Maine and it's live and like, and that's the iteration cycle, right? And uh, and that's just incredible. Where we want to be in a year and where we need to be in a year is I. You push domain and that is live deploying 2 million strong node networks where it's cool, that's an update. And now we fixed some pathological use case and like we need that iteration speed. And so we're working on that as, as quikly as we can to sort of run that to ground. All of that is gonna be like, that's all in our view. Convenience, right? Like we're not, we're not taking away any capability from you. Um, we're not saying, Hey, you can't use Iroh for X, Y, and Z. We're not taking away any protocols, but we are saying like, Hey, yeah, if you want this to be like really the, the expertise of the team that built this expresses a product. Hey, you're gonna pay us a monthly fee for that. We're gonna run infrastructure for you. And, and that sort of like coupling of infrastructure, automatic deployment to testing and like all of this sort of like stuff that frankly I talk to a lot of developers and they say that they will do it and they just don't like doing it. 'cause it's a lot of work to like, you know, go in and deal with all that gap. They'd much rather write code. Um, we're we're building tools that do that for them. So far it's been successful. Honestly, it's taken us a long time to sort of get to that space and like cracking the. Um, sustainability challenge in an open source project has been, is the single, like hardest thing that I've worked on as a founder in the open source world. We always knew we were gonna do open source. We're always like, just hard committed to like, we're gonna use a very, very, very permissive license for all of Iroh. And we just sort of backpedaled into like, okay, great, now we have to figure out revenue. Um, and then like, which I think that may not sound unfamiliar, that may not be an open source specific problem, but, um, yeah, we're, we're very happy with where we're at in terms of the product offering. This complimentary node service that rolling out has been a lot of fun. Um, and I don't feel like we're taking an, an extractive relationship with the community. To me, that's always the litmus pest. Like, is it, does it feel like we're actually like taking something away? Like we're taking some like piece that like, would actually be really useful if everybody had it, but we're just like keeping it on our side of the fence to make sure that people pay us. Um, yeah, we're not doing that and I don't feel like we're doing that. And so I'm, I'm quite happy with where we've landed. Justin: The high road and also the hard road. Brendan: Dude. Yes. But yeah, I, that being said, I, I have kind of changed my tune a little bit when I talk to folks these days developing open source. I think LLMs have really changed things like as open source maintainers. We are getting different challenges now, but we get a, a lot more issues sent to us from people who are talking to an agent that has hallucinated something that, uh, and, and now becomes our problem. Or even worse, they've like filed a pull request that claims to do X, y, or z and we, we can't actually verify that it does that without a careful read. And, and that's creating way more like signal than we have time to sort of, uh, ingest and work on as a team. Um, and so I think that you will, I I I, I think that even open source is like due for some contours and changes as, as a, as a culture of practice in the new world. Um, I dunno what that looks like yet. I don't have any answers there, but we're seeing it, we're kind of keeping an eye on it. Love to talk about it in a year once we've learned something, but uh, I dunno if that's helpful. [01:05:44] Conclusion and Final Thoughts Andrew: Well, that wraps it up for our questions. It seems like you've built a really cool thing and the services you plan to offer and the path to 1.0 also seem very exciting. So thanks for coming on the show and talking about it. Brendan: Much appreciated guys. Thank you so much for having me. I really appreciate it. Your podcast is lovely. Thank you for you too. Justin: Thank you so much. Yeah, and Brendan, y'all, y'all do fantastic work. Uh, I'm excited to see how IROH develops and hopefully I can play with it soon. Uh, but yeah, thanks for coming on and yeah, good luck with 1.0. Brendan: Thanks so much guys. Have a great day.