{
  "$type": "site.standard.document",
  "canonicalUrl": "https://justingarrison.com/blog/2024-06-06-ten-years-of-kubernetes",
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreiektvheygjbbybvxvp65glddppk4el3atsuzwufywcr65rivr7xry"
    },
    "mimeType": "image/png",
    "size": 116358
  },
  "description": "How I got started with Kubernetes and where it has taken me.",
  "path": "/blog/2024-06-06-ten-years-of-kubernetes",
  "publishedAt": "2024-06-06T00:00:00.000Z",
  "site": "at://did:plc:p7uix7mresfq4nfzxp3klgfa/site.standard.publication/3mmdn7mg2qm2d",
  "textContent": "My background\n\nIn February 2014 I saw a presentation by Jérôme Petazzoni about a new tool called Docker.\nI was pretty cautious about adopting a new tool because I was a system administrator and was still in the process of deploying config management in our environment.\nThat same year I gave a presentation about Ansible which was the most recent technology I was abusing.\n\nJune 2nd 2014, 4 days before Kubernetes was announced, I started a new role at Disney Animation.\nI was blown away by the infrastructure.\nAll the hard things I was trying to do at my last job were already done.\nPuppet was deployed everywhere, OS provisioning was automatic, even PXE booting and DNS were well established and reliable.\n\nDisney is where I first started playing with containers, but my experience was really bad.\nWe used Red Hat and if you know the history between Docker and Red Hat you'll know it wasn't good.\n\nRHEL 6 had really slow devicemapper storage, an old Linux Kernel--even for 2014 standards--and I didn't see a need for it in our environment.\nBut in 2015 I attended DockerCon because I was starting to get more interested in use cases.\nAnd saw a lot of cool stuff, including presentations from other Disney employees about using Mesos to scale applications.\n\nThat year I also attended Hashiconf where they announced Nomad.\nThis space was getting interesting fast and I started to see the potential for how containers could be used for things I was working on.\n\nThe main thing I wanted to containerize was our render workloads.\nIt was 99% of what we ran in the studio and it was traditional HPC batch scheduling with a custom scheduler.\n\nI was going down the Mesos path because it had documented custom scheduling interfaces that seemed like a good fit.\nIt also didn't require workloads to be containerized which would be a good first step for us.\n\nThe Kelsey moment\n\nI met Kelsey Hightower at Hashiconf in 2015 and in early 2016 he was in LA and tweeted he had some time before his flight.\nI replied and invited him up to the Disney lot in Burbank and he came.\nWhile showing him around I mentioned I was in the process of deploying Mesos because it had custom scheduling and that's something we needed.\n\nHe mentioned Kubernetes allowed for custom scheduling too, but it wasn't well defined yet.\nHe had just joined Google and was going to be working on Kubernetes more and I invited him to give a Disney wide Kubernetes training in early 2016.\n\nAfter that training I started digging into the API and controllers more and built bashScheduler so I could understand how the scheduling flow worked.\n\nMy first cluster\n\nAt this point I had been running containers for some internal workloads still without much success.\nThe RHEL based platform combined with many other things would cause Docker to crash on an almost weekly basis and I knew the problem lied within the RHEL 6 kernel.\nThe easiest fix was to move off of RHEL.\n\nSummer of 2016 I took an internal ticket to let our web team self deploy workloads.\nThey were the easiest to containerize and didn't rely on too much of our internal NFS storage or process.\n\nI went through a lot of internal reviews to get CoreOS, Alpine Linux, and various other components working with our provisioning systems.\nIn about 2 months I had an on-prem, bare metal Kubernetes cluster.\nIt was integrated into our internal provisioning tools and even had some documentation.\n\nI spent the rest of 2016 working with internal dev teams to migrate applications to containers and talking to partners to help with support, monitoring, etc.\nI started the sig-on-prem group within the Kubernetes community and was loved by many startups at the time because I was a customer (even if I wasn't a paying customer yet).\n\nA Kubernetes break\n\nIn 2017 my manager pulled me off of Kubernetes completely.\nI changed teams and was now helping with internal initiatives I found extremely boring.\n\nTo stay involved with the Kubernetes ecosystem I wrote Cloud Native infrastructure with Kris Nóva.\nThe book came out in the fall of 2017 and it was intentionally not a Kubernetes book.\n\nThe main idea in the book was infrastructure as software which is the core idea behind Kubernetes controllers and things like GitOps.\n\nIn 2018 I was still not allowed to touch Kubernetes internally and I was getting very anxious.\nI wanted to do more cloud things and my manager also did not allow me to be involved with our cloud migration efforts.\n\nI hit a point when I knew I had to leave if I wanted to work on the technologies I was interested in.\nIn fall of 2018 I moved to Disney Streaming Services because I heard we were making a streaming platform.\n\nWhen I interviewed, Disney+ was announced but it wasn't until after I started that we knew it was supposed to launch in 2019.\nThe infrastructure I was responsible for was all AWS based and built entirely on Amazon ECS.\n\nThis was great because it was still containers and I figured \"how different could it be?\"\nIt turns out it was very different from Kubernetes (especially in 2018).\n\nWe couldn't have launched Disney+ without ECS.\nKubernetes would have been the wrong choice at the time because we had a small infrastructure team (at one point only 2 of us), and we needed to offload as much as possible to Amazon.\n\nWe stablized and scaled the system for the first half of 2019 and launched Disney+ on November 6.\n\nAt that point I was pretty far from Kubernetes.\nI didn't pay any attention to it for almost 2 years and outside of playing with it at home didn't know if I would ever go back.\n\nAmazon\n\nA lot of things happened in my time at Disney Streaming and I left in March of 2020 (2 weeks after our successful UK launch).\n\nI joined Amazon as a developer advocate for conatiners and got to work trying to improve all of our container services.\nI started with ECS because that's what I was most familiar with and in 2021 we launched App Runner.\n\nI still enjoyed Kubernetes, but was very disconnected from how it was evolving.\n\nIn Summer of 2021 Amazon split the containers developer advocate teams and I was given a choice of continuing to work with ECS or dedicating to EKS.\nI decided to go with EKS because it's what I preferred and where I saw the most growth.\n\nI focused mostly on scalability issues customers were having and took over the Containers from the Couch channel.\n\nInternally I was working with Karpenter and EKS Anywhere.\nLarge sale deployments and on-prem were my two passions to help Kubernetes and customers grow.\n\nI learned a lot 2022-2024 and finally had some time to get involved with Kubernetes again.\nAmazon has a lot of internal turmoil and focus changed frequently.\n\nI spent the majority of 2023 helping customers understand how to reduce their cloud spend.\nKarpenter consolidation was a big win for many of them because it could reduce waste, but it wasn't enough.\n\nI did the math over and over again and realized that the cloud was just too expensive for medium to large scale customers.\nSavings plans and reserved instances were still not as cheap or efficient as buying and managing servers.\n\nIn mid 2023 Amazon mostly disbanded the EKS Anywhere team.\nThere was a small group of core developers keeping the lights on for customers, but it was clear they weren't going to invest in it.\n\nFrom my experience, on-prem is more financially stable than the cloud and it's where I wanted to work again.\n\nSidero\n\nI knew about Talos Linux from when Andrew announced it in 2019.\nIt's the thing I wanted to exist for a Kubernetes Linux distro when I worked at Disney Animation.\n\nI was at Amazon when Bottlerocket was announced and GAd and it felt very clunky compared to Talos.\nI had lots of internal feedback for the Bottlerocket team and it was never accepted.\n\nWhen Sidero launched Omni in 2023 I knew it was a better approach for Kubernetes management.\nEverything Kubernetes was too focused on decoupled microservices and Omni was a monolith.\n\nIt simplified the operations drastically and I made multiple internal proposals to shift EKS Anywhere into more of a Omni-like platform.\nI even proposed that Amazon should buy Sidero labs to repurpose the products and higher the people to jumpstart our products.\nAmazon said no and in 2024 I joined Sidero.\n\nI have been in and around Kubernetes for 9 years.\nI deployed Kubernetes 1.2-1.5 at Disney and have watched the community and project grow immensely over time.\n\nCongratulations Kubernetes for making it to 10 years!\n\nI attribute the longevity to the early community leaders like Sarah Novotny, Jaice DuMars, Joe Beda, and Tim Hockin.\nPeople who I didn't know and didn't know me, but they still would help me figure out the project, community, and my personal career.\n\nPeople like them are the reason I'm still involved with Kubernetes and their examples are reasons I try to give back my own time for people joining the community or tech in general.\n\nCongrats\n\nI wanted to share my story for anyone who has been involved with Kubernetes or looking to get involved.\nThere's still a lot of work to be done and a lot of ideas.\n\nI hope the next 10 years focuses on making Kubernetes easier for everyone to run, not just the cloud providers who have financial investment.\nThe vast majority of work done in the project is done by people who work at companies that make money directly or indirectly from the project and those incentives hold the project back.",
  "title": "Ten Years of Kubernetes"
}