Twenty-three clusters, one repository, and drift that gets corrected rather than reported
Multi-cluster fleet management
Where they started
Each team had built its own cluster, with its own add-ons and its own version. Cluster count had tripled in a year and nobody could say which ones differed in security-relevant ways.
The problem, precisely
Consistency and autonomy pull against each other. A central mandate would have been ignored; leaving it alone meant an unbounded blast radius.
What we did
Defined a small set of non-negotiable platform capabilities, made everything else optional through composition, and put both under Git with automated drift detection.
Delivered
- Baseline capability set defined and enforced through Git, not through review meetings
- Cluster provisioning via Cluster API so a new cluster is a pull request
- Drift detection that reports and optionally corrects deviations automatically
- Fleet inventory with version, policy status and ownership per cluster
- Cross-cluster backup and restore tested, including recovery to a different region
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 23 clusters under one declarative control plane
- 1 new cluster provisionable as a pull request
- — drift findings per week, trending down after the baseline
- 1 cross-region restore proven, then written into the DR plan
Standardise the small set of things that must not differ, and let everything else be optional. A mandate covering all of it produces clusters that are technically identical and practically abandoned.