A firewall pair that had never failed over, and 41 devices with no configuration backup
Firewall HA cluster + network configuration management
Where they started
High availability had been configured years earlier and never intentionally failed over. No device configuration was backed up anywhere except in the memory of one engineer.
The problem, precisely
An untested failover pair and an un-backed-up configuration estate are the same risk described twice: the redundancy exists on paper only.
What we did
Tested the failover path in a maintenance window first, reproduced an MLAG member failure in a lab before it happened in production, then put every configuration under version control with drift detection.
Delivered
- Stateful failover tested and timers tuned, then made a repeatable quarterly drill
- MLAG member failure reproduced in containerlab before being verified on hardware
- Configuration backup for all 41 devices with per-change diffing into Git
- NetBox as the authoritative inventory, reconciled against live device state
- Firmware and end-of-life review with a prioritised remediation list
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 2.8s measured stateful failover
- 41 devices under version control for the first time
- 9 devices years behind validated firmware
- 1 failure mode reproduced in a lab before production saw it
Configuration backup is the cheapest availability feature in networking. Every device you cannot restore is a device whose recovery time you are guessing at.