{
  "$type": "site.standard.document",
  "canonicalUrl": "https://justingarrison.com/blog/2024-06-06-kubernetes-2.0",
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreicjuovr7rqwdfpwgs3hfnk2zylaej7qld3ah5fsd5yjatpfnr7igy"
    },
    "mimeType": "image/png",
    "size": 70242
  },
  "description": "Breaking changes I would like to see in Kubernetes 2.0",
  "path": "/blog/2024-06-06-kubernetes-2.0",
  "publishedAt": "2024-06-06T12:00:00.000Z",
  "site": "at://did:plc:p7uix7mresfq4nfzxp3klgfa/site.standard.publication/3mmdn7mg2qm2d",
  "textContent": "I just wrote about my path with Kubernetes and how I've seen the project grow.\nIt's probably a good time to talk about what could be next.\n\nKubernetes 2.0\n\nI would love to see some breaking changes in Kubernetes.\nIf we were building a Kubernetes 2.0, the top 4 things I would want to see changed are:\n\n1. Optional etcd\n1. Versionless\n1. More monoliths\n1. K8sfile\n\netcd\n\nKubernetes is an etcd API.\nIt was a great idea to make Kubernetes use a key value store instead of a traditional relational database or something like GraphQL.\n\nBut etcd has consistently been a bottleneck for the project and a hindrance for operationalizing Kubernetes.\nIt is inentional complexity to sell cloud services and it should be removed.\n\nThe only thing that really prevents people from self hosting Kubernetes is etcd.\nThey don't have the operational knowledge to run it on their own and it's a niche database that they won't hire experts to gain that knowledge.\nSo they outsource that knowledge to cloud providers.\n\nBut companies like Google don't use etcd.\nThey can shim the etcd API to whatever bespoke, internal key value storage system they _do_ have experties in.\n\nIn my opinion Kubernetes should make a local database like SQlite the default database.\nMake it simple to operate, understandable to backup and restore, and remove power from the cloud providers who can operate bespoke databases.\n\nKine has proven that it can work without breaking changes but it should be the default.\nAlmost every customer I worked with at AWS gained no benefits of etcd being the data store behind Kubernetes.\n\nA single node SQlite with EBS snapshots or HA RDS instance would have served them better.\nIt will be easier to operate and from people I've talked to, will provide better scaling numbers.\n\nI'm happy to be shown why I'm wrong from people who know more of the details.\n\nVersionless\n\nThe best feature about ECS is the API is completely versionless.\nAs a user you never have to upgrade it.\nIf you want to stay on an old version you can.\nIf you need new features, you upgrade your agent and the ECS API updates with you.\n\nI've pitched this idea at every company I've worked for.\nBase infrastructure should not have versions.\n\nDNS, DHCP, NFS have all been extremely stable interfaces that we are able to build on top of without worrying about the API changing.\nEven NFS, which has versions, could use the older API with configuration.\n\nKubernetes has been around long enough that we should have a versionless, stable-only API.\nThe only changes that happen are new features.\nNo v1alpha1, no beta, just stable APIs that can be supported going forward.\n\nKubernetes long term support isn't the solution.\nThat only lets users delay the pain.\nWe need a way in Kubernetes to configure only v2 API options and let the Kubelet version determine the version in use.\n\nIf etcd is why users moved to cloud hosted Kubernetes, upgrades is what keeps them there.\nDoing Kubernetes upgrades 3x per year is hard for home lab users.\nIt's impossible for large companies with millions of dollars riding on top of a moving target.\n\nVersionless Kubernetes would make Kubernetes easier to operate.\nCombined with SQlite by default it would be an extremely stable foundation for even more workloads in the future.\n\nMore monoliths\n\nThe project has grown by splitting ownership of components and building loosely coupled controllers that coordinate through the API.\nThis has added an extreme amount of flexibility at the cost of complexity.\n\nMicroservices are harder to operate than a monolith and the vast majority of Kubernetes users don't need the independent scaling abilities of microservices.\nThe project optimized for developer ownership at the cost of user ergonomics.\n\nKubernetes started as a monolith and was broken up over time because there were too many features and extensions that couldn't be owned by the core project.\nWe have learned a lot about where extension should and should not exist.\nTaking what we've learned and closing off some unmaintainable extensions could be a great first step.\n\nI also would like to see more monoliths for managing Kubernetes.\nProjects like Cluster API went with microservices by default and the usability of that project shows.\nIt's too complex unless you're going to be selling it.\nIt's not for end users.\n\nWe should also have more monolithic resources.\nThe deployment + service combination shouldn't exist in Kubernetes 2.0.\nMost of the successful internal platform solutions I've seen have a single resource that couples together multiple Kubernetes resources.\nJust like \"containers\" are a combination of 10+ Linux kernel features, Kubernetes needs more, monolithic, top level resources.\n\nMonolithic resources would be more adaptable to unexpected workloads.\nKubernetes is good at stateless, web-based workloads today but current trends for more HPC/AI workloads is showing we need to adapt faster to what users want.\n\nK8sfile\n\nI want Kubernetes to have an easier way to deploy a complete cluster as an artifact, and a way to reuse those artifacts with changes.\n\nJust like the Dockerfile allows a FROM command there should be a way for Kubernetes clusters to share configuration and holistically know a spec for what's deployed.\nI want to be able to do something like:\n\nThen build and release an artifact that lets me run deploy that in multiple environments.\nIdeally with optional defaults for things like the operating system and it should work outside of the cloud.\n\nMultiple provisioning systems (including cloud providers) could support the API that injests this artifact and it could make trying new things and moving clusters easier.\nIt also helps the community share a simplified spec without needing to learn the implementation details.\n\nI believe the main reason Docker has been successful has been the container registry.\nBeing able to start from examples and share your own implementation makes it so much easier to learn.",
  "title": "Kubernetes 2.0"
}