{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreida4kwcieqacvoeqf4tuwrnk2i2nar6kyrfqfh3zlz4h2ofglhswq",
    "uri": "at://did:plc:4oiydigwe2y63w2hp5wxkv7t/app.bsky.feed.post/3mp52ok3lamg2"
  },
  "path": "/blog/anchor-your-gitignore-entries/",
  "publishedAt": "2026-06-24T00:00:00.000Z",
  "site": "https://wantguns.dev",
  "tags": [
    "git worktree",
    "upgrade notes between v15 and v17",
    "Argo app"
  ],
  "textContent": "## Preface\n\nI am an avid selfhoster, and I follow the gitops pattern with ArgoCD for controlling the state of my Kubernetes (k3s) cluster. Something which I probably do different than most folks is that I vendor charts by `helm pull`-ing and `--untar`-ring them in-tree -\n\n\n    # in the gitops repo root $ tree charts -L 3 charts ├── cert-manager │  └── 1.19.1 │  ├── Chart.yaml │  ├── templates │  └── values.yaml ├── forgejo │  └── 15.0.2 │  ├── Chart.lock │  ├── Chart.yaml │  ├── charts │  ├── docs │  ├── LICENSE │  ├── README.md │  ├── templates │  └── values.yaml ├── genapp │  └── 0.1.0 │  ├── Chart.yaml │  ├── templates │  └── values.yaml # ... output elided\n\nI picked up this patten at one of the previous places I worked at. I like it because it gives me git worktree like abilites -\n\n  * I can edit the charts (and their different versions) in-place\n  * I can just create my own charts in-tree without having to upload them on a helm/OCI repository\n  * I don't have to worry about unreachable networks or hitting OCI pull rate-limits\n\n\n\n## Upgrading Forgejo\n\nLast night, I was checking the settings of my 6 month old Forgejo deployment and I noticed I am 3 major versions behind (shudders!)\n\nI checked the changelog, and there were no upgrade notes between v15 and v17, and so I went ahead and pulled the new chart in my gitops repo -\n\n\n    $ helm pull --untar --untardir charts/forgejo/17.1.1 \\  oci://code.forgejo.org/forgejo-helm/forgejo $ mv charts/forgejo/17.1.1/{forgejo/*,} # move the chart one dir up $ rm -rf charts/forgejo/17.1.1/forgejo # delete the empty dir\n\nThis is how it looked after\n\n\n    $ tree charts/forgejo -L 2 charts/forgejo ├── 15.0.2 │  ├── Chart.lock │  ├── Chart.yaml │  ├── charts │  ├── docs │  ├── LICENSE │  ├── README.md │  ├── templates │  └── values.yaml └── 17.1.1  ├── Chart.lock  ├── Chart.yaml  ├── charts  ├── docs  ├── LICENSE  ├── README.md  ├── scripts  ├── templates  ├── uv.toml  ├── values.schema.json  └── values.yaml\n\nAfter changing Forgejo's Argo app values to use the new helm chart, committing changes to git and pushing them, I eagerly waited on finding a new and healthy Forgejo deployment on my cluster. Only to find the new pod crashing, specifically the `init-directories` container -\n\n\n    $ kubectl get deployments forgejo NAME READY UP-TO-DATE AVAILABLE AGE forgejo 0/1 1 0 230d  $ kubectl get pod forgejo-64454d8779-mgvzv NAME READY STATUS RESTARTS AGE forgejo-64454d8779-mgvzv 0/1 Init:CrashLoopBackOff 6 (20s ago) 8m  $ kubectl logs forgejo-64454d8779-mgvzv -c init-directories exec /usr/sbin/init_directory_structure.sh: exec format error\n\n## Debugging\n\n#### Wrong architecture ?\n\nAt first, I thought maybe, just somehow maybe, a wrong architecture's image got pulled, and so I confirmed that -\n\n\n    $ sudo nerdctl -a /run/k3s/containerd/containerd.sock -n k8s.io \\  image inspect \\  code.forgejo.org/forgejo/forgejo:15.0.3-rootless \\  --format '{{.Architecture}} {{.Os}} {{.Variant}}' amd64 linux\n\nNope\n\n#### What is `init_directory_structure.sh` ?\n\n\n    $ kubectl debug forgejo-64454d8779-mgvzv -it \\  --copy-to=d-forgejo \\  --container=init-directories -- sh # got dropped into the debug container ... /var/lib/gitea $ file /usr/sbin/init_directory_structure.sh sh: file: not found /var/lib/gitea $ ls -lah /usr/sbin/init_directory_structure.sh lrwxrwxrwx 1 root git 34 Jun 24 15:37 /usr/sbin/init_directory_structure.sh -> ..data/init_directory_structure.sh /var/lib/gitea $ cat /usr/sbin/init_directory_structure.sh /var/lib/gitea $\n\nThe symlink was intact but the file was empty! I quickly checked the secret which was volume-mounted for this file in container, and as expected it didn't get populated.\n\n\n    $ kubectl get secret forgejo-init -o yaml apiVersion: v1 kind: Secret # .. elided content data:  configure_gitea.sh: \"\"  configure_gpg_environment.sh: \"\"  configure_ssh_signing.sh: \"\"  init_directory_structure.sh: \"\"\n\n#### Must be some drift in the chart that I didn't notice\n\n\n    $ diff -w \\  <(helm template forgejo charts/forgejo/15.0.2 --values environments/alnitak/forgejo/values.yaml) \\  <(helm template forgejo charts/forgejo/17.1.1 --values environments/alnitak/forgejo/values.yaml)\n\nnoisy output\n\n\n     diff < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 13,14c13,14 < app.kubernetes.io/version: \"13.0.2\" < version: \"13.0.2\" --- > app.kubernetes.io/version: \"15.0.3\" > version: \"15.0.3\" 54c54 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 58,59c58,59 < app.kubernetes.io/version: \"13.0.2\" < version: \"13.0.2\" --- > app.kubernetes.io/version: \"15.0.3\" > version: \"15.0.3\" 69c69 < printf \"${1}\\n\" --- > printf '%b\\n' \"$1\" 82c82,83 < local setting=\"$(awk -F '=' '{print $1}' <<< \"${line}\" | xargs echo -n)\" --- > local setting > setting=\"$(awk -F '=' '{print $1}' <<< \"${line}\" | xargs echo -n)\" 124c125,126 < local setting=\"$(awk -F '=' '{print $1}' <<< \"${line}\" | xargs echo -n)\" --- > local setting > setting=\"$(awk -F '=' '{print $1}' <<< \"${line}\" | xargs echo -n)\" 151c153,154 < local section=\"$(basename \"${config_file}\")\" --- > local section > section=\"$(basename \"${config_file}\")\" 171c174 < while read -d '' configFile; do --- > while read -r -d '' configFile; do 185,188c188,195 < export FORGEJO__SECURITY__INTERNAL_TOKEN=$(gitea generate secret INTERNAL_TOKEN) < export FORGEJO__SECURITY__SECRET_KEY=$(gitea generate secret SECRET_KEY) < export FORGEJO__OAUTH2__JWT_SECRET=$(gitea generate secret JWT_SECRET) < export FORGEJO__SERVER__LFS_JWT_SECRET=$(gitea generate secret LFS_JWT_SECRET) --- > FORGEJO__SECURITY__INTERNAL_TOKEN=$(gitea generate secret INTERNAL_TOKEN) > export FORGEJO__SECURITY__INTERNAL_TOKEN > FORGEJO__SECURITY__SECRET_KEY=$(gitea generate secret SECRET_KEY) > export FORGEJO__SECURITY__SECRET_KEY > FORGEJO__OAUTH2__JWT_SECRET=$(gitea generate secret JWT_SECRET) > export FORGEJO__OAUTH2__JWT_SECRET > FORGEJO__SERVER__LFS_JWT_SECRET=$(gitea generate secret LFS_JWT_SECRET) > export FORGEJO__SERVER__LFS_JWT_SECRET 209c216 < if [ -f ${GITEA_APP_INI} ]; then --- > if [ -f \"${GITEA_APP_INI}\" ]; then 222c229 < environment-to-ini -o $GITEA_APP_INI --- > environment-to-ini -o \"${GITEA_APP_INI}\" 223a231 >  232c240 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 236,237c244,245 < app.kubernetes.io/version: \"13.0.2\" < version: \"13.0.2\" --- > app.kubernetes.io/version: \"15.0.3\" > version: \"15.0.3\" 245a254,262 >  > configure_ssh_signing.sh: |- > #!/usr/bin/env bash > set -eu >  > install -m 600 /raw/ssh-signing-key /data/git/.ssh-signing/key > ssh-keygen -y -f /data/git/.ssh-signing/key > /data/git/.ssh-signing/key.pub > chmod 644 /data/git/.ssh-signing/key.pub >  261a279 >  337a356 >  346c365 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 350,351c369,370 < app.kubernetes.io/version: \"13.0.2\" < version: \"13.0.2\" --- > app.kubernetes.io/version: \"15.0.3\" > version: \"15.0.3\" 373c392 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 377,378c396,397 < app.kubernetes.io/version: \"13.0.2\" < version: \"13.0.2\" --- > app.kubernetes.io/version: \"15.0.3\" > version: \"15.0.3\" 402c421 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 406,407c425,426 < app.kubernetes.io/version: \"13.0.2\" < version: \"13.0.2\" --- > app.kubernetes.io/version: \"15.0.3\" > version: \"15.0.3\" 412,415c431 < type: RollingUpdate < rollingUpdate: < maxUnavailable: 0 < maxSurge: 100% --- > type: Recreate 423c439 < checksum/config: 2b6d6032d93735b7ef2b8d9f42a609b67854bd87d7bd26f165a603f3c1b0d108 --- > checksum/config: 7e25be8665026fb35742251f68fafbc357ebc5907669fc5a21759d98ae5daf2d 425c441 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 429,430c445,446 < app.kubernetes.io/version: \"13.0.2\" < version: \"13.0.2\" --- > app.kubernetes.io/version: \"15.0.3\" > version: \"15.0.3\" 438c454 < image: \"code.forgejo.org/forgejo/forgejo:13.0.2-rootless\" --- > image: \"code.forgejo.org/forgejo/forgejo:15.0.3-rootless\" 466c482 < image: \"code.forgejo.org/forgejo/forgejo:13.0.2-rootless\" --- > image: \"code.forgejo.org/forgejo/forgejo:15.0.3-rootless\" 518c534 < image: \"code.forgejo.org/forgejo/forgejo:13.0.2-rootless\" --- > image: \"code.forgejo.org/forgejo/forgejo:15.0.3-rootless\" 562c578 < image: \"code.forgejo.org/forgejo/forgejo:13.0.2-rootless\" --- > image: \"code.forgejo.org/forgejo/forgejo:15.0.3-rootless\" 640c656 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 644,645c660,661 < app.kubernetes.io/version: \"13.0.2\" < version: \"13.0.2\" --- > app.kubernetes.io/version: \"15.0.3\" > version: \"15.0.3\" 672c688 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 676,677c692,693 < app.kubernetes.io/version: \"13.0.2\" < version: \"13.0.2\" --- > app.kubernetes.io/version: \"15.0.3\" > version: \"15.0.3\" 694c710 < helm.sh/chart: forgejo-15.0.2 --- > helm.sh/chart: forgejo-17.1.1 698,699c714,715 < app.kubernetes.io/version: \"13.0.2\" < version: \"13.0.2\" --- > app.kubernetes.io/version: \"15.0.3\" > version: \"15.0.3\"\n\nThere was nothing that seemed problematic in all that noise\n\n#### Let me _locally_ verify if the secret is even getting rendered\n\n\n    $ helm template forgejo charts/forgejo/17.1.1 --values environments/alnitak/forgejo/values.yaml \\  | grep Secret -A 100 \\  | grep init_directory_structure -A 10  init_directory_structure.sh: |-  #!/usr/bin/env bash   set -euo pipefail   set -x  mkdir -p /data/git/.ssh  chmod -R 700 /data/git/.ssh  [ ! -d /data/gitea/conf ] && mkdir -p /data/gitea/conf   # prepare temp directory structure\n\nIt _was_ getting rendered correctly! I was perplexed. Why would there be a difference between my local helm template rendering and ArgoCD's helm rendering pipeline ?\n\n#### `diff`-ing the charts\n\nEverything seemed to be in the right place in the last stage of the helm pipeline (rendering), so my focus shifted to the precursors - the charts. I would have `diff`'d the chart early on if the difference between the versions was not so huge. But atleast with the context gathered about what's failing, I could narrow the diff-set down and look specifically for that part of the chart.\n\n\n    $ rg init_directory_structure charts/forgejo 15.0.2/templates/gitea/init.yaml 15: init_directory_structure.sh: |-  15.0.2/templates/gitea/deployment.yaml 65: command: [\"/usr/sbin/init_directory_structure.sh\"]  17.1.1/templates/gitea/init.yaml 14: init_directory_structure.sh: |- 15:{{ tpl (.Files.Get \"scripts/init_directory_structure.sh.tpl\") . | indent 4 }}  17.1.1/templates/gitea/deployment.yaml 65: command: [\"/usr/sbin/init_directory_structure.sh\"]\n\nSo, there was a change in how the `init_directory_structure.sh` secret key is rendered in the final yaml manifest. In the new chart, it referenced from the `scripts/init_directory_structure.sh.tpl` file within the chart.\n\n#### Is the file present in-tree ?\n\n\n    $ fd \"init_directory_structure.sh.tpl\" charts/forgejo/17.1.1 charts/forgejo/17.1.1/scripts/init_directory_structure.sh.tpl\n\nOf course it is, I had just rendered the chart _locally_ with the secret populated.\n\n#### .. And is it tracked in git (for ArgoCD) ?\n\n\n    $ git ls-files charts/forgejo/17.1.1/scripts/init_directory_structure.sh.tpl  # ... crickets ...\n\nI quickly checked my `.gitignore` -\n\n\n    $ cat .gitignore crash.log *.DS_Store .idea .metals settings.json manifests scripts\n\nAh >:\n\n## Fix\n\nA year-older me had added some scripts within a directory called `scripts` in the gitops repo's root. To avoid tracking them on git, I appended `.gitignore` with `scripts`.\n\nSince I did not anchor this entry to the repo root, `git` was faithfully also ignoring the `scripts` directory inside the Forgejo chart.\n\nQuickly, I anchored the relevant entries, ..\n\n\n    diff --git a/.gitignore b/.gitignore index 74b080f..29b8872 100644 --- a/.gitignore +++ b/.gitignore @@ -3,5 +3,5 @@ crash.log  .idea  .metals  settings.json -manifests -scripts +/manifests +/scripts\n\n.. `git add`ed the helm chart again (staging the files ignored before), committed the changes and pushed it upstream.\n\nAfter a few minutes, I checked the deployment again\n\n\n    $ kubectl get deployments forgejo NAME READY UP-TO-DATE AVAILABLE AGE forgejo 1/1 1 1 230d",
  "title": "Anchor your .gitignore entries",
  "updatedAt": "2026-06-24T00:00:00.000Z"
}