A cluster nobody dared upgrade, rebuilt so upgrades became routine
Kubernetes — self-managed HA control plane
Where they started
The cluster had run for three years without a version upgrade because the last attempt broke ingress. etcd backups ran nightly and had never been restored. Secrets sat in a Git repository that everyone had access to.
The problem, precisely
Nothing about the cluster was reversible. Every change was a one-way door, so the team made changes less and less often.
What we did
Built a staging cluster from an approved design first and proved the upgrade path there, then rebuilt production the same way — GitOps from the first commit so the cluster could be recreated from code.
Delivered
- HA control plane across three failure domains, external etcd with timed restore
- GitOps from day one — every workload, policy and add-on in a repository
- Cilium network policies replacing a convention-based multi-tenancy README
- Secrets moved out of Git into Vault-backed External Secrets
- Upgrade runbook validated on staging, then executed on production in a window
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 49 min measured control-plane upgrade, zero dropped requests
- 4 min measured etcd restore to a working control plane
- 0 plaintext secrets remaining in Git
- 38 min full cluster rebuild from repository, staging-verified
Rebuildability is the only upgrade strategy that survives staff turnover. If you can recreate the cluster from code in under an hour, version upgrades stop being scary.