{
  "$type": "site.standard.document",
  "canonicalUrl": "https://justingarrison.com/blog/state-of-immutable-linux",
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreif25eb5cq6a76fdhybpc2th6ivb7lddv6nuirevpmol26nojydtp4"
    },
    "mimeType": "image/jpeg",
    "size": 139886
  },
  "description": "Lies, damn lies, and read only filesystems",
  "path": "/blog/state-of-immutable-linux",
  "publishedAt": "2026-03-20T00:00:00.000Z",
  "site": "at://did:plc:p7uix7mresfq4nfzxp3klgfa/site.standard.publication/3mmdn7mg2qm2d",
  "textContent": "I gave a talk at SoCal Linux Expo explaining what an \"immutable\" OS means and how some of the popular distributions implement immutability.\nThe talk is a lot more fun than this blog post and has props to visualize what's happening.\n\n{{< youtube jvdPuTcdGXs >}}\n\nLinux was never designed to start up and change nothing.\nThere's no such thing as a 100% immutable.\nThe problem immutability is trying to solve is preditcable and scoped changes.\n\nUsually, the distribution will market immutability to mean, more secure, more reliable, or the ability to roll back.\nThese are all things traditional, mutable, distros struggled with and were often causes for outages and toil.\n\nHere's a summary from my talk about how each distribution implements immutability.\n\nFlatcar Linux\n\nCoreOS, later named Container Linux, was the first distro I was aware of in this space.\nChromebooks originally implemented this update mechanism, called Omaha and later Android adopted it too.\nCoreOS, the company, got bought by Red Hat who got bought by IBM.\n\nOmaha is cool and efficient with block level diffs between versions.\nIf you're upgrading from version 3218 -> 3230 you will download only the blocks that are different as long as that diff has been calculated ahead of time.\nThis doesn't work for every version change, but it is agnostic to what type of filesystem or files change.\n\nContainer Linux under Red Hat was abandoned, abranded?, to rename their Project Atomic to Fedora CoreOS.\nThere is almost no similarities between the projects except the brand name.\n\nAt the time of aquisition, Container Linux was forked into Flatcar Linux which was later bought by Microsoft as part of the Kinvolk aquisition and recently donated to the CNCF.\nFlatcar is the real successor to CoreOS Linux imutability implementation.\n\nWhile most distributions describe their update mechanism as A/B partitions, Flatcar is the only one that has actual partitions on disk.\nThe OS has USR-A and USR-B partitions that write the immutable parts of the distro and boots back and forth between them.\nEvery other distro has A/B trees of files or images on the same partition.\n\nFlatcar makes the files in those partitions read only and creates overlay mounts for all of the things a user can change.\nWhile you can't change a file in the actual USR partition, you can still change what the OS uses because your changes will get implemnted on top of the base configuration.\n\nFlatcar isn't immutable.\nThe \"immutable\" root is the lower directory and you make change in the running system in an upper directory that persists across reboots.\nSome of the files can't be changed, but it's more like a Linux distro without a traditional package manager.\n\nCustomizations happen via ignition, cloud-init, systemed system extensions (distributed as raw disk images), and ssh.\nA common pattern for most of these distributions that allow certain changes at different layers of the management cycle.\n\nFedora CoreOS\n\nFCOS is the server and Silverblue is the desktop implementation of ostree.\nDistros like Bazzite and Bluefin are opinionated builds of Silverblue.\nOstree started with Project Atomic, mentioned above, and is essentially \"git for your filesystem.\"\n\nWhat if every file was tracked and linked on the filesystem?\nAnd OS versions were distributed as a \"commit.\"\nEveryone would get the same tree of files and we could switch between them.\n\nThe nice thing is your wouldn't be limited to 2 commits.\nYou can have more than just an A/B rollback strategy.\nThe bad thing is this isn't as flexible as Nix or MicroOS.\nNix does tracking at an individual package layer.\nThere's no single \"commit\" for Nix like there is for ostree and MicroOS just snapshots the entile filesystem.\n\nBecause ostree doesn't have the flexibility for individual packages, they created rpm-ostree which lets you overlay an RPM on top of your base OS.\nThis defeats the whole purpose of immutability because rpm-ostree install -A $PACKAGKE would install $PACKAGE on a live system.\n\nBut package overlays sometimes break the system.\nThis is why ~Red Hat~ IBM is trying to encourage people move away from rpm-ostree with bootc, which doesn't support RPM overlays.\nA downside of ostree is it has a bespoke toolchain to build and distribute updates.\nbootc is still ostree under the hood, but it's distributed as a ✨container✨ image instead of files.\n\nOstree updates in the past were files based.\nYou only had to download the files you needed to perform the update.\nThis is simaly to Flatcar's block level updates, but could be more efficient because it would download the contents of the block (a.k.a. a file), not the whole block.\n\nMoving to containers loses this benefit because now all files are obscured into layers and layers are based on build steps, not contents.\nSo now I can download container layer diffs, but those are much larger and opaque to what I actually need.\n\nThe system is customized via ~ignition~ butane and cloud-init, but with bootc you also have to customize the base OS via a Containerfile similar to Kairos below.\nYou also use cloud-init, ssh, and sysext depending on what you're trying to customize.\n\nAt this point, bootc based distributions are the most complicated to manage if you want to customize the distro and dispite personally using them for years, I don't like where they're going.\n\nKairos\n\nKairos is a \"meta\" distro.\nKairos takes an existing distro while adding ✨immutability✨.\nKairos packages the file system as a .img file and distributes it as a container.\n\nThis means you build the distro in a Dockerfile, but on disk it has overlay mounts like Flatcar does.\nInstead of two full partitions, Kairos boots from one loopback-mounted image at a time.\nUpdates overwrite the .img and updates grub.\n\nKairos doesn't support ignition for early (pre-systemd) configuration.\nInstead, you're expected to rebuild the base container image for base configuration (like bootc) and all customization happens via cloud-init and ssh.\nBecause you're repackaging tradatitional Linux distributions, you still have all the bloat of apt and dnf, but they don't work because of the read-only folder mounting.\nThis is probably why they're building Hadron to have some more control over the base distro.\n\nKairos uses the fact that cloud-init runs at each system boot in its favor.\nSome files and folders can be changed in the live system, but they don't persist after reboot.\nThis forces you to make all changes via cloud-init and re-configure the system each time.\n\nMicroOS\n\nMicroOS is from openSUSE and it's the most traditional Linux distro on this list.\nIt still runs and acts like SUSE, but it uses btrfs snapshots to solve the update rollback problem.\n\nInstead of two fixed partitions, you have a chain of snapshots that you can roll back through.\nYou still end up with an active and a pending state, but the mechanism is more flexible and transparent to the user, and the filesystem isn't polluted with weird mounts or overlay folders.\n\nThe mutability story is similar to the others—there are overlays, and parts of the filesystem you can change without rebooting.\nBut it's not a fundamentally locked-down system, it's a system with structured, rollback-safe update mechanisms.\n\nBottlerocket\n\nBottlerocket is an AWS Linux distribution for running containers.\nIt has an \"API\" and dosent' have SSH.\n\nOn paper it sounds a lot like Talos.\nIn practice it's nothing like it.\nI wrote a blog post about this so you should read that if you're interested.\n\nBottlerocket will become the default container OS in AWS just like Container-optimized OS is the default for GCP.\nDon't bother looking at it if you're not in AWS.\n\nNixOS\n\nNix is not immutable, it's reproducable.\n\nNix is the most mutable system on this list.\nI'd say it's even more mutable (if that's a thing) than traditional Linux distributions.\nIt tries to make every mutation safe, via nix packaging, which means all changes are isolated and versioned.\n\nWhat Nix gives you is the guarantee that if you declare something in a flake, you can rebuild it anywhere, any time, identically.\nThe guarantee is about reproducibility of the build, not immutability of the running system.\n\nThe symlinks at the root of a NixOS system point into /nix/store, and you can have as many versions installed as you want.\nThere's only one live view of the base system at a time.\n\nYou customize the system via flake and reboot into the new filesystem tree for packages that require reboots to load.\nEverything else can be changed live on the system without rebooting.\n\nTalos Linux\n\nDisclaimer, I work at Sidero Labs\n\nFor Talos, immutability is not the point.\nImmutability is an artifact of how Talos achieves its actual goal of API-driven management.\n\nTalos never finishes the normal Linux boot sequence.\nThe normal sequence for Linux is: UEFI → bootloader → kernel + initramfs → pivot to the \"real\" filesystem → PID 1.\nTalos runs entire OS inside initramfs via a Unified Kernel Image (UKI).\nThere's no root filesystem on disk to pivot to.\n\nTalos is packaged and distributed as a container image and also has a concept of system extensions (also packaged as containers).\nSystem extensions are overlayed on the root filesystem to bring additional packages, drivers, and services.\n\nThere are mutable files on disk such as container images and a state partition, but the actual OS is running immutability in memory.\nChanges are not persisted across reboots, similar to Kairos.\nAt boot Talos reads the declarative state from the stored STATE partition or cloud-init and makes changes.\nThis is similar and different from cloud-init which is sometimes declarative and sometimes an imparitive bash script.\n\nCustomizations to the base system are done via \"packages\" which are combined into a containerized \"installer\" image.\nThe installer image is what writes the UKI bundle to disk and also formats a disk during installation.\n\nIn practice Talos has A/B booting, reads from a desired state configuration, and applies cloud-init-like configuration to th",
  "title": "The State of Immutable Linux"
}