{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreicagqiw2kcllfb4fnu6wvbilu74oqpeqryxinmyizdhnxoni2ucme",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mpu2seudygo2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreifmsdam2lfngje6gmfvgpuxhpn7yjzhc4a5xuuxxvp3aqv2hub2bm"
    },
    "mimeType": "image/webp",
    "size": 406494
  },
  "path": "/henryosei/i-thought-i-understood-containers-then-i-tried-building-one-5a80",
  "publishedAt": "2026-07-04T21:48:57.000Z",
  "site": "https://dev.to",
  "tags": [
    "devops",
    "docker",
    "learning",
    "linux"
  ],
  "textContent": "I had just aced my mentor’s Docker exam, so of course I thought I understood containers.\n\nI had said all the right words: namespaces, cgroups, images, layers, PID 1, Kubernetes Pods. Then I typed my first serious command and Linux reminded me that knowing the nouns is not the same thing as building the thing.\n\n\n\n    $ sudo unshare -p 1 test\n    unshare: failed to execute 1: No such file or directory\n\n\nThat was the opening scene. I had not even built anything yet. I had typed the flags wrong and accidentally asked `unshare` to execute a program called `1`. This was going to be less “implement Docker” and more “let the kernel correct my confidence, one error at a time.”\n\n##  v1: namespaces, or the first time PID 1 lied to me\n\nThe first version was supposed to be easy: run a process in a new PID namespace and prove it sees itself as PID 1.\n\nSo I ran the command the way I thought it worked:\n\n\n\n    $ sudo unshare --pid bash\n    # echo $$\n    25184\n\n\nThat was not PID 1. That was just embarrassing.\n\nThe rule I had missed is simple: PID namespaces apply to children. The process that calls `unshare --pid` does not magically become PID 1. You need to fork. The first child born into the new namespace becomes PID 1.\n\nSo the working version was:\n\n\n\n    $ sudo unshare --pid --fork bash\n    # echo $$\n    1\n\n\nThat one line changed the tone. I was inside a different process universe. The shell thought it was process 1. Signals felt different. Orphans came home to it.\n\nThen I ran `ps`, and got humbled again.\n\n\n\n    # ps -o pid,ppid,comm\n        PID    PPID COMMAND\n      25310   25304 bash\n      25344   25310 ps\n\n\nThat made no sense at first. I was PID 1, but `ps` was showing host-looking PIDs. The next reveal: `ps` does not ask the kernel some pure “what processes exist?” question. It reads files. If `/proc` still points at the host procfs, your tools will tell you the host story.\n\nSo I remounted `/proc` from inside the namespace:\n\n\n\n    # mount -t proc proc /proc\n    # ps -o pid,ppid,comm\n        PID    PPID COMMAND\n          1       0 bash\n          7       1 ps\n\n\nThat was when it clicked. The namespace did not become real to my eyes until `/proc` agreed with it. Before that, I had isolation, but my tools were reading the old filesystem view.\n\nThe UTS namespace lesson was cleaner. I accidentally ran a science experiment.\n\nIn one terminal, without a UTS namespace:\n\n\n\n    $ hostname\n    ba149abae9bd\n\n\nThen inside a new UTS namespace:\n\n\n\n    $ sudo unshare --uts bash\n    # hostname toybox\n    # hostname\n    toybox\n\n\nBack outside that UTS namespace:\n\n\n\n    $ hostname\n    ba149abae9bd\n\n\nThat was my control and treatment. Same machine, same kernel, different hostname view — and the “host” was already a container hostname, which made the containers-inside-containers setup visible in the output. Nothing mystical. Just one isolated kernel data structure doing exactly what the docs said, except now I had seen it with my own hands.\n\n##  v2: pivot_root, the boss fight\n\nAfter namespaces, I got overconfident again. The next version was supposed to give the process its own filesystem: a tiny rootfs, a shell, maybe BusyBox. Very container-ish.\n\nMy repo had bash scripts for this, not some compiled runtime from a tutorial. So the shape of the attempt was `v2.sh`, a rootfs, and a command to run inside it.\n\nThe parade started with the obvious error:\n\n\n\n    $ sudo ./v2.sh ./rootfs /bin/sh\n    exec /bin/sh: no such file or directory\n\n\nFine. There was no shell where I said there would be a shell. I fixed that with Alpine’s own BusyBox and hit the more annoying version: the file existed, but the kernel still said it could not run it.\n\n\n\n    $ ./rootfs/bin/busybox sh\n    bash: ./rootfs/bin/busybox: cannot execute: required file not found\n\n\nThis is the kind of error that feels personal because you can list the file. You can see the symlink. The computer still refuses.\n\nThe plot twist came from `file`:\n\n\n\n    $ cd rootfs\n    $ file bin/busybox\n    bin/busybox: ELF 64-bit LSB pie executable, ARM aarch64, dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, stripped\n\n\nThe binary was there. The interpreter was not available from the old world. Linux was not saying “your BusyBox file does not exist.” It was saying “from here, I cannot load the interpreter this ELF needs.” Same surface error, different problem.\n\nThe fix was not what I first thought. Alpine’s BusyBox did not need to become static. Once Alpine became `/`, its musl loader would be at `/lib/ld-musl-aarch64.so.1`, and Alpine’s `/bin/sh` would be happy. The thing that needed help was the handoff itself: my Ubuntu slim image did not even have `pivot_root`.\n\n\n\n    $ pivot_root . put_old\n    bash: pivot_root: command not found\n    $ file /bin/busybox\n    /bin/busybox: ELF 64-bit LSB executable, ARM aarch64, statically linked, stripped\n    $ /bin/busybox pivot_root . put_old\n\n\nThat was the better plot twist: the old world could not perform the handoff without borrowing a static tool. `busybox-static` was not my replacement shell inside Alpine. It was the bridge that could run before and during the transition.\n\nThen I hit the Bash hash-cache moment. Alpine was now `/`, but Bash still remembered a command path from before the filesystem switch. It went hunting for `/usr/bin/mount` in a world that had just been evicted.\n\n\n\n    / # mount -t proc proc /proc\n    bash: /usr/bin/mount: No such file or directory\n    / # hash -r\n    / # mount -t proc proc /proc\n\n\nI had fixed the filesystem and was still debugging an old decision Bash had remembered for me. That is the kind of bug that makes you take a short walk.\n\nThen came the Mac problem. My setup was not “normal Linux laptop, local ext4 disk.” It was Apple Silicon Mac → privileged Ubuntu container → repo mounted from macOS. That means virtiofs was in the story whether I wanted it there or not.\n\nThe symptom showed up after the pivot, inside Alpine, which made it stranger. Applet symlinks like `mount` and `ls` could fail with `Permission denied` on the Mac-shared mount, while calling BusyBox directly still worked. The files were there; executing through those symlinks was the weird part.\n\n\n\n    / # ls\n    sh: ls: Permission denied\n    / # /bin/busybox ls\n    bin      dev      etc      lib      proc     put_old\n    / # mount -t proc proc /proc\n    sh: mount: Permission denied\n    / # /bin/busybox mount -t proc proc /proc\n\n\nThat is where I stopped trying to make the shared mount behave like a normal Linux filesystem. I moved the rootfs to a container-native path and reran the same idea from there. Boring fix, correct fix.\n\n`pivot_root` itself still had opinions.\n\n\n\n    pivot_root: invalid argument\n\n\nThis one was not glamorous either. The new root had to be a mount point. The old root needed somewhere to go. I had to bind-mount the new root onto itself, create `oldroot`, call `pivot_root(newroot, oldroot)`, `chdir(\"/\")`, and unmount the old root.\n\nWhen it finally worked, the reward was tiny and perfect:\n\n\n\n    / # cat /etc/os-release\n    NAME=\"Alpine Linux\"\n    ID=alpine\n    VERSION_ID=3.24.1\n    PRETTY_NAME=\"Alpine Linux v3.24\"\n\n\nThat output felt better than it should have. It was just `/etc/os-release`, but it meant the process was now living in the filesystem I had assembled. Not Docker magic. Just mounts, an ELF loader problem, a static `pivot_root` applet, a root pivot, and errors more useful than any clean diagram.\n\n##  v3: cgroups, or Linux as a filesystem API\n\nThe third version was about resource limits. This was where cgroups stopped being “that Kubernetes thing” and became a filesystem API I could write to.\n\nOn cgroup v2, everything looked like files:\n\n\n\n    $ ls /sys/fs/cgroup\n    cgroup.controllers  cgroup.procs  cgroup.subtree_control  memory.current  memory.max  pids.max\n\n\nBeautiful, until you write the wrong value to the wrong file at the wrong level.\n\nMy first attempt was to create a child cgroup, move the process, and enable controllers casually.\n\n\n\n    $ sudo mkdir /sys/fs/cgroup/tiny\n    $ echo +memory | sudo tee /sys/fs/cgroup/cgroup.subtree_control\n    tee: /sys/fs/cgroup/cgroup.subtree_control: Device or resource busy\n\n\nThat error was the “no internal processes” rule. In cgroup v2, you cannot enable domain controllers on a cgroup that still has normal processes in it. The parent has to become a manager; work happens in children.\n\nSo I had to do the evacuation dance: create a child for the shell, move myself into it, then enable controllers on the parent, then create the actual container cgroup.\n\n\n\n    $ sudo mkdir /sys/fs/cgroup/init\n    $ echo $$ | sudo tee /sys/fs/cgroup/init/cgroup.procs\n    18420\n    $ echo +memory +pids | sudo tee /sys/fs/cgroup/cgroup.subtree_control\n    +memory +pids\n    $ sudo mkdir /sys/fs/cgroup/tiny\n    $ echo 32M | sudo tee /sys/fs/cgroup/tiny/memory.max\n    32M\n    $ echo 64 | sudo tee /sys/fs/cgroup/tiny/pids.max\n    64\n\n\nAfter that, the runtime could put the child into `/sys/fs/cgroup/tiny/cgroup.procs` before executing the workload.\n\nThe first OOM test was almost too satisfying. I ran a small memory hog inside the limited cgroup and watched the kernel enforce the number.\n\n\n    $ dmesg | tail -n 3\n    [ 5221.143892] memory: usage 32768kB, limit 32768kB, failcnt 41\n    [ 5221.143901] Memory cgroup out of memory: Killed process 18432 (python3) total-vm:89344kB, anon-rss:31812kB, file-rss:0kB, shmem-rss:0kB\n    [ 5221.143944] oom_reaper: reaped process 18432 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB\n\n\nThat line made cgroups real. Not a Docker setting. Not YAML. The kernel enforcing a number I wrote into a file.\n\n##  Then Kubernetes looked smaller\n\nAfter this, I deployed to Kubernetes and the concepts no longer felt distant. A Pod was not magic anymore. It was these same pieces with an API in front: namespaces to shape what the process can see, a filesystem root to control what it can touch, and cgroups to limit what it can consume. I would not call my tiny runtime useful, but it made the abstractions stop floating. Docker, containerd, and Kubernetes became less like brands and more like organized versions of the three scripts I had just fought into existence.",
  "title": "I Thought I Understood Containers. Then I Tried Building One."
}