Two nodes, real fencing, and a failover that finally happens without a human
Pacemaker + Corosync, DRBD replication
Where they started
An HA cluster had been configured years earlier, but failover was manual because the fencing devices had never been validated and the team was afraid of split-brain.
The problem, precisely
An untested HA cluster is worse than no cluster: it produces confidence without capability.
What we did
Fixed the fencing path first — because everything else depends on it — then rehearsed failover under load, including the case where the primary is unresponsive rather than cleanly down.
Delivered
- Fencing validated against both power and watchdog paths, with SBD timeouts tuned
- DRBD replication verified under load with a written split-brain recovery procedure
- Resource ordering and constraints rewritten so failover is deterministic
- Quarterly failover drill, scripted and timed, added to the operations calendar
- Runbook written for the case where the survivor is also degrading
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 14s measured automatic failover under production load
- 0 manual interventions required
- 2 fencing paths validated per node
- 1 split-brain, induced on purpose, recovered in 9 minutes
Validate fencing before anything else. If the failed node can still write to shared storage, the rest of the configuration is decoration.