Anchor your .gitignore entries
Preface
I am an avid selfhoster, and I follow the gitops pattern with ArgoCD for controlling the state of my Kubernetes (k3s) cluster. Something which I probably do different than most folks is that I vendor charts by helm pull-ing and --untar-ring them in-tree -
# in the gitops repo root $ tree charts -L 3 charts ├── cert-manager │ └── 1.19.1 │ ├── Chart.yaml │ ├── templates │ └── values.yaml ├── forgejo │ └── 15.0.2 │ ├── Chart.lock │ ├── Chart.yaml │ ├── charts │ ├── docs │ ├── LICENSE │ ├── README.md │ ├── templates │ └── values.yaml ├── genapp │ └── 0.1.0 │ ├── Chart.yaml │ ├── templates │ └── values.yaml # ... output elided
I picked up this patten at one of the previous places I worked at. I like it because it gives me git worktree like abilites -
- I can edit the charts (and their different versions) in-place
- I can just create my own charts in-tree without having to upload them on a helm/OCI repository
- I don't have to worry about unreachable networks or hitting OCI pull rate-limits
Upgrading Forgejo
Last night, I was checking the settings of my 6 month old Forgejo deployment and I noticed I am 3 major versions behind (shudders!)
I checked the changelog, and there were no upgrade notes between v15 and v17, and so I went ahead and pulled the new chart in my gitops repo -
$ helm pull --untar --untardir charts/forgejo/17.1.1 \ oci://code.forgejo.org/forgejo-helm/forgejo $ mv charts/forgejo/17.1.1/{forgejo/*,} # move the chart one dir up $ rm -rf charts/forgejo/17.1.1/forgejo # delete the empty dir
This is how it looked after
$ tree charts/forgejo -L 2 charts/forgejo ├── 15.0.2 │ ├── Chart.lock │ ├── Chart.yaml │ ├── charts │ ├── docs │ ├── LICENSE │ ├── README.md │ ├── templates │ └── values.yaml └── 17.1.1 ├── Chart.lock ├── Chart.yaml ├── charts ├── docs ├── LICENSE ├── README.md ├── scripts ├── templates ├── uv.toml ├── values.schema.json └── values.yaml
After changing Forgejo's Argo app values to use the new helm chart, committing changes to git and pushing them, I eagerly waited on finding a new and healthy Forgejo deployment on my cluster. Only to find the new pod crashing, specifically the init-directories container -
$ kubectl get deployments forgejo NAME READY UP-TO-DATE AVAILABLE AGE forgejo 0/1 1 0 230d $ kubectl get pod forgejo-64454d8779-mgvzv NAME READY STATUS RESTARTS AGE forgejo-64454d8779-mgvzv 0/1 Init:CrashLoopBackOff 6 (20s ago) 8m $ kubectl logs forgejo-64454d8779-mgvzv -c init-directories exec /usr/sbin/init_directory_structure.sh: exec format error
Debugging
Wrong architecture ?
At first, I thought maybe, just somehow maybe, a wrong architecture's image got pulled, and so I confirmed that -
$ sudo nerdctl -a /run/k3s/containerd/containerd.sock -n k8s.io \ image inspect \ code.forgejo.org/forgejo/forgejo:15.0.3-rootless \ --format '{{.Architecture}} {{.Os}} {{.Variant}}' amd64 linux
Nope
What is init_directory_structure.sh ?
$ kubectl debug forgejo-64454d8779-mgvzv -it \ --copy-to=d-forgejo \ --container=init-directories -- sh # got dropped into the debug container ... /var/lib/gitea $ file /usr/sbin/init_directory_structure.sh sh: file: not found /var/lib/gitea $ ls -lah /usr/sbin/init_directory_structure.sh lrwxrwxrwx 1 root git 34 Jun 24 15:37 /usr/sbin/init_directory_structure.sh -> ..data/init_directory_structure.sh /var/lib/gitea $ cat /usr/sbin/init_directory_structure.sh /var/lib/gitea $
The symlink was intact but the file was empty! I quickly checked the secret which was volume-mounted for this file in container, and as expected it didn't get populated.
$ kubectl get secret forgejo-init -o yaml apiVersion: v1 kind: Secret # .. elided content data: configure_gitea.sh: "" configure_gpg_environment.sh: "" configure_ssh_signing.sh: "" init_directory_structure.sh: ""
Must be some drift in the chart that I didn't notice
$ diff -w \ <(helm template forgejo charts/forgejo/15.0.2 --values environments/alnitak/forgejo/values.yaml) \ <(helm template forgejo charts/forgejo/17.1.1 --values environments/alnitak/forgejo/values.yaml)
noisy output
diff < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 13,14c13,14 < app.kubernetes.io/version: "13.0.2" < version: "13.0.2" --- > app.kubernetes.io/version: "15.0.3" > version: "15.0.3" 54c54 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 58,59c58,59 < app.kubernetes.io/version: "13.0.2" < version: "13.0.2" --- > app.kubernetes.io/version: "15.0.3" > version: "15.0.3" 69c69 < printf "${1}\n" --- > printf '%b\n' "$1" 82c82,83 < local setting="$(awk -F '=' '{print $1}' <<< "${line}" | xargs echo -n)" --- > local setting > setting="$(awk -F '=' '{print $1}' <<< "${line}" | xargs echo -n)" 124c125,126 < local setting="$(awk -F '=' '{print $1}' <<< "${line}" | xargs echo -n)" --- > local setting > setting="$(awk -F '=' '{print $1}' <<< "${line}" | xargs echo -n)" 151c153,154 < local section="$(basename "${config_file}")" --- > local section > section="$(basename "${config_file}")" 171c174 < while read -d '' configFile; do --- > while read -r -d '' configFile; do 185,188c188,195 < export FORGEJO__SECURITY__INTERNAL_TOKEN=$(gitea generate secret INTERNAL_TOKEN) < export FORGEJO__SECURITY__SECRET_KEY=$(gitea generate secret SECRET_KEY) < export FORGEJO__OAUTH2__JWT_SECRET=$(gitea generate secret JWT_SECRET) < export FORGEJO__SERVER__LFS_JWT_SECRET=$(gitea generate secret LFS_JWT_SECRET) --- > FORGEJO__SECURITY__INTERNAL_TOKEN=$(gitea generate secret INTERNAL_TOKEN) > export FORGEJO__SECURITY__INTERNAL_TOKEN > FORGEJO__SECURITY__SECRET_KEY=$(gitea generate secret SECRET_KEY) > export FORGEJO__SECURITY__SECRET_KEY > FORGEJO__OAUTH2__JWT_SECRET=$(gitea generate secret JWT_SECRET) > export FORGEJO__OAUTH2__JWT_SECRET > FORGEJO__SERVER__LFS_JWT_SECRET=$(gitea generate secret LFS_JWT_SECRET) > export FORGEJO__SERVER__LFS_JWT_SECRET 209c216 < if [ -f ${GITEA_APP_INI} ]; then --- > if [ -f "${GITEA_APP_INI}" ]; then 222c229 < environment-to-ini -o $GITEA_APP_INI --- > environment-to-ini -o "${GITEA_APP_INI}" 223a231 > 232c240 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 236,237c244,245 < app.kubernetes.io/version: "13.0.2" < version: "13.0.2" --- > app.kubernetes.io/version: "15.0.3" > version: "15.0.3" 245a254,262 > > configure_ssh_signing.sh: |- > #!/usr/bin/env bash > set -eu > > install -m 600 /raw/ssh-signing-key /data/git/.ssh-signing/key > ssh-keygen -y -f /data/git/.ssh-signing/key > /data/git/.ssh-signing/key.pub > chmod 644 /data/git/.ssh-signing/key.pub > 261a279 > 337a356 > 346c365 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 350,351c369,370 < app.kubernetes.io/version: "13.0.2" < version: "13.0.2" --- > app.kubernetes.io/version: "15.0.3" > version: "15.0.3" 373c392 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 377,378c396,397 < app.kubernetes.io/version: "13.0.2" < version: "13.0.2" --- > app.kubernetes.io/version: "15.0.3" > version: "15.0.3" 402c421 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 406,407c425,426 < app.kubernetes.io/version: "13.0.2" < version: "13.0.2" --- > app.kubernetes.io/version: "15.0.3" > version: "15.0.3" 412,415c431 < type: RollingUpdate < rollingUpdate: < maxUnavailable: 0 < maxSurge: 100% --- > type: Recreate 423c439 < checksum/config: 2b6d6032d93735b7ef2b8d9f42a609b67854bd87d7bd26f165a603f3c1b0d108 --- > checksum/config: 7e25be8665026fb35742251f68fafbc357ebc5907669fc5a21759d98ae5daf2d 425c441 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 429,430c445,446 < app.kubernetes.io/version: "13.0.2" < version: "13.0.2" --- > app.kubernetes.io/version: "15.0.3" > version: "15.0.3" 438c454 < image: "code.forgejo.org/forgejo/forgejo:13.0.2-rootless" --- > image: "code.forgejo.org/forgejo/forgejo:15.0.3-rootless" 466c482 < image: "code.forgejo.org/forgejo/forgejo:13.0.2-rootless" --- > image: "code.forgejo.org/forgejo/forgejo:15.0.3-rootless" 518c534 < image: "code.forgejo.org/forgejo/forgejo:13.0.2-rootless" --- > image: "code.forgejo.org/forgejo/forgejo:15.0.3-rootless" 562c578 < image: "code.forgejo.org/forgejo/forgejo:13.0.2-rootless" --- > image: "code.forgejo.org/forgejo/forgejo:15.0.3-rootless" 640c656 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 644,645c660,661 < app.kubernetes.io/version: "13.0.2" < version: "13.0.2" --- > app.kubernetes.io/version: "15.0.3" > version: "15.0.3" 672c688 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 676,677c692,693 < app.kubernetes.io/version: "13.0.2" < version: "13.0.2" --- > app.kubernetes.io/version: "15.0.3" > version: "15.0.3" 694c710 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 698,699c714,715 < app.kubernetes.io/version: "13.0.2" < version: "13.0.2" --- > app.kubernetes.io/version: "15.0.3" > version: "15.0.3"
There was nothing that seemed problematic in all that noise
Let me locally verify if the secret is even getting rendered
$ helm template forgejo charts/forgejo/17.1.1 --values environments/alnitak/forgejo/values.yaml \ | grep Secret -A 100 \ | grep init_directory_structure -A 10 init_directory_structure.sh: |- #!/usr/bin/env bash set -euo pipefail set -x mkdir -p /data/git/.ssh chmod -R 700 /data/git/.ssh [ ! -d /data/gitea/conf ] && mkdir -p /data/gitea/conf # prepare temp directory structure
It was getting rendered correctly! I was perplexed. Why would there be a difference between my local helm template rendering and ArgoCD's helm rendering pipeline ?
diff-ing the charts
Everything seemed to be in the right place in the last stage of the helm pipeline (rendering), so my focus shifted to the precursors - the charts. I would have diff'd the chart early on if the difference between the versions was not so huge. But atleast with the context gathered about what's failing, I could narrow the diff-set down and look specifically for that part of the chart.
$ rg init_directory_structure charts/forgejo 15.0.2/templates/gitea/init.yaml 15: init_directory_structure.sh: |- 15.0.2/templates/gitea/deployment.yaml 65: command: ["/usr/sbin/init_directory_structure.sh"] 17.1.1/templates/gitea/init.yaml 14: init_directory_structure.sh: |- 15:{{ tpl (.Files.Get "scripts/init_directory_structure.sh.tpl") . | indent 4 }} 17.1.1/templates/gitea/deployment.yaml 65: command: ["/usr/sbin/init_directory_structure.sh"]
So, there was a change in how the init_directory_structure.sh secret key is rendered in the final yaml manifest. In the new chart, it referenced from the scripts/init_directory_structure.sh.tpl file within the chart.
Is the file present in-tree ?
$ fd "init_directory_structure.sh.tpl" charts/forgejo/17.1.1 charts/forgejo/17.1.1/scripts/init_directory_structure.sh.tpl
Of course it is, I had just rendered the chart locally with the secret populated.
.. And is it tracked in git (for ArgoCD) ?
$ git ls-files charts/forgejo/17.1.1/scripts/init_directory_structure.sh.tpl # ... crickets ...
I quickly checked my .gitignore -
$ cat .gitignore crash.log *.DS_Store .idea .metals settings.json manifests scripts
Ah >:
Fix
A year-older me had added some scripts within a directory called scripts in the gitops repo's root. To avoid tracking them on git, I appended .gitignore with scripts.
Since I did not anchor this entry to the repo root, git was faithfully also ignoring the scripts directory inside the Forgejo chart.
Quickly, I anchored the relevant entries, ..
diff --git a/.gitignore b/.gitignore index 74b080f..29b8872 100644 --- a/.gitignore +++ b/.gitignore @@ -3,5 +3,5 @@ crash.log .idea .metals settings.json -manifests -scripts +/manifests +/scripts
.. git added the helm chart again (staging the files ignored before), committed the changes and pushed it upstream.
After a few minutes, I checked the deployment again
$ kubectl get deployments forgejo NAME READY UP-TO-DATE AVAILABLE AGE forgejo 1/1 1 1 230d
Discussion in the ATmosphere