Forty machines, one Git repository, no one logging in by hand
Bare metal provisioning plane
Where they started
Machines had been installed by whoever was available, from a wiki page that had stopped being true two years earlier. Three nodes had a RAID layout nobody could explain, and firmware versions ranged across four releases.
The problem, precisely
The team could not safely reinstall a node — the procedure existed only as tribal knowledge — so a failed disk meant a host down for days.
What we did
Replaced the wiki page with a provisioning plane and a machine inventory as the single source of truth. Rack-by-rack roll-through with two spare nodes as a buffer, starting with the least critical rack.
Delivered
- MAAS as the authoritative machine inventory, synchronised nightly from Redfish
- iPXE + preseed bring-up with automated RAID, bonding and firmware baseline
- Ansible roles for OS baseline, network config, package policy and unattended upgrades
- Firmware drift report, generated weekly and alerted on any deviation
- Bring-up runbook rehearsed twice with the customer team before handover
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 12 min per-node bring-up, measured on all 40
- 100% of machine state now described in Git
- 0 interactive logins required for a new node
- 9 nodes found on firmware with known CVEs
The provisioning plane matters less than the inventory. Every repeatable fleet we have built stood on one authoritative list of machines; every fragile one had three.