Why your HA cluster has never actually failed over
Fencing is not a feature of high availability — it is the precondition for it. A cluster that cannot isolate a failed peer will, on a split, run both sides and write to the same data. Here is how to find out whether yours can.
There is a specific kind of conversation we have roughly once a month. Someone tells us their cluster has been running for four years without an incident. We ask when it last failed over. The answer, almost always, is that it has not.
This is presented as evidence of stability. It is actually evidence of nothing.
A cluster that has never failed over is untested, not healthy
Two things can be true of an HA cluster that has been up for years:
- Every component is genuinely redundant and the failover path works.
- The failover path has a defect that has simply not been exercised.
From the outside, these look identical. The uptime graph cannot distinguish them, and neither can the configuration file. The only thing that separates them is a deliberate test, and most clusters have never had one because the test sounds risky — you are, after all, deliberately breaking production.
That reluctance is understandable and it is how people end up discovering the defect during a real outage, at the worst possible time, with the least possible preparation.
Fencing is the part that matters
If you take one thing from this article, take this: a cluster without working fencing does not have high availability, regardless of how many replicas it runs.
Fencing — STONITH in Pacemaker terminology — is the mechanism by which a healthy node forcibly isolates a node it believes has failed. Power it off via IPMI, cut its storage access, disable its network. The method matters less than the guarantee.
The guarantee is this: at most one side of a split writes to the data.
Without that guarantee, a network partition produces the worst of all outcomes. Both halves decide the other is dead, both promote themselves, and both write. When the partition heals, you do not have a failover — you have divergent data and a manual reconciliation problem that may not have a correct answer.
This is why we say fencing is a precondition rather than a feature. It is not one capability among many. It is the thing that makes the rest of the design safe.
Three questions that reveal the real state
You can find out where you stand in about twenty minutes.
Can the fence device actually reach the victim, on more than one path? A fence that depends on the same network as the cluster is a fence that fails exactly when you need it. Check that the IPMI, PDU or storage-fabric path is independent of the cluster’s own communication.
Has it been triggered, on purpose, on a real node? Not simulated, not reviewed. Actually fenced, with the victim confirmed to have lost power or connectivity, and the surviving side confirmed to have taken over.
Does the failover complete inside the configured timeout? A fence that takes twice as long as the cluster’s patience allows produces a node that gets fenced and then returns, believing it is still primary.
What a proper test looks like
The test we run looks like this:
- Pick a non-critical node in a maintenance window, during business hours if possible and always with the application team on the call.
- Confirm the fencing configuration and both paths independently.
- Trigger the event.
- Time it — from the first sign of trouble to full service recovery, measured by something outside the cluster.
- Repeat in the opposite direction. Asymmetric failures are common and a single-direction test will not find them.
- Write down the number.
That last step is the one people skip, and it is the one that pays. A measured failover time is a number you can put in a contract, brief the board with, and compare against next quarter. An assumed one is a hope with a decimal point.
The uncomfortable part
If your cluster has been up for four years and never failed over, the most likely explanation is not that it is robust. It is that nobody has been willing to find out.
We would rather be the people who break it on a Tuesday afternoon with the team watching than the people who discover it at 3am on a Saturday during your busiest week. That is the entire argument for a drill schedule, and it is why every managed tier we sell includes one.