From a shared Jenkins box to a runner fleet, with builds that no longer queue at 9am
CI/CD platform + artifact registry
Where they started
Everything ran on one Jenkins server with local executors. Builds queued from nine in the morning, and the pipeline was both the bottleneck and an unmonitored production dependency.
The problem, precisely
The CI platform had become critical infrastructure with none of the availability engineering that title implies.
What we did
Split the control plane from the executors, moved runners onto a Kubernetes cluster with autoscaling, and added the artifact registry HA that the pipeline had been missing.
Delivered
- Runner fleet on Kubernetes with class-based autoscaling and per-team quotas
- Artifact and container registry made highly available, with garbage collection policy
- Pipeline secrets moved to Vault with short-lived credentials instead of stored tokens
- Build cache moved to shared object storage, cutting median build time substantially
- Pipeline health metrics: queue depth, duration and failure rate, on a dashboard teams watch
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 3,400/day pipelines with no queue at peak hour
- — median build duration, measured before and after
- 0 long-lived static credentials in pipeline config
- 1 registry, HA, with a tested restore from object storage
Your CI platform is production infrastructure. Give it the same availability design, backup and monitoring you would give a customer-facing service.