High Availability & Failover Cluster Engineering
Most HA clusters have never actually failed over. We build them so the failover works, and then we prove it by pulling the plug.
What this usually looks like
You have redundancy on paper. The question nobody has answered is whether the automatic path actually completes.
- The last failover was performed by hand, at night, by one person, under pressure.
- Fencing is configured but has never been triggered — and an unfenced HA cluster can corrupt data.
- Nobody knows the real RTO because nobody measured the last one.
- The runbook says "fail over to node 2" and stops there.
Three ways to buy this
Audit if you want to know how bad it is. Build if you have the team and want the capability. Managed if you don't want to own the pager. Most engagements start with the audit and decide afterwards.
Audit
A fixed-price health check — the cheapest way in
- Quorum and fencing review — can a split brain actually happen?
- Measured failover test in a maintenance window, with real timings
- Data replication integrity check
- Failure-domain map including the failures you have not planned for
Build
One-off delivery: design, code, hardening, handover
- Pacemaker + Corosync cluster with working STONITH (SBD, IPMI, or PDU backed)
- Keepalived/VRRP for floating VIPs with sub-second detection
- DRBD 9 + LINSTOR or shared-storage replication
- Load-balancer health checks that reflect reality, not TCP port liveness
- Documented and rehearsed failover and failback procedures
Managed
Build plus 7×24 on-call, SLA and quarterly drills
- Quarterly fault-injection drill with a written report and measured RTO/RPO
- Fencing device health verification
- Quorum topology review as you grow
- Alerting that distinguishes "degraded but serving" from "down"
What we actually deploy
Specific tooling, not categories. If a version matters to your decision, ask — we will tell you which one we would pick and why, including when the answer is the older one.
Cluster resource management
- Pacemaker 2.1
- Corosync 3
- pcs / crm shell
- pcsd web UI
- resource-agents
Fencing (non-negotiable)
- SBD
- IPMI / Redfish fencing
- PDU / switched outlet
- watchdog
- fence_mpath
- cloud/API fencing for hybrid
VIP & L4 failover
- Keepalived + VRRP
- UCARP
- CARP
- anycast with BGP
Replication
- DRBD 9 + LINSTOR
- DRBD Proxy
- ZFS send/recv
- rsync + lsyncd (last resort)
Database-level HA
- Patroni
- repmgr
- MySQL Group Replication
- Redis Sentinel
- proxy failover (ProxySQL / HAProxy)
Chaos & validation
- Chaos Mesh
- LitmusChaos
- manual power-cut drills
- network partition tests
- fio + pgbench under failover
What we will be measured on
These are the numbers we put in the delivery report. If we cannot measure it, we do not promise it.
- Announced failover completes within the agreed RTO, measured during a drill, not estimated.
- Split-brain is impossible by construction: verified fencing, verified quorum device behaviour.
- VIP drift on single-node loss in under 3 seconds.
- Quarterly drill report with real numbers, delivered whether it went well or badly.
- Failback is a documented one-command procedure, also rehearsed.
What we would build, and where the weak point is
The VIP is an intentional single point of failure — it is the interface, not the service. What matters is that the standby node is fenced before it ever tries to take over.
marks a single point of failure we would raise with you rather than hide. Every architecture has one; the question is whether you know where it is.
What we don't do
Every vendor says yes to everything. Here is where our answer is no, or where someone else needs to be in the room. If this list looks wrong for your situation, tell us — we would rather argue about scope before the contract than after.
- Out of scope
HA without fencing hardware or an equivalent power-control mechanism
We will not build an HA cluster that cannot fence its peer. It is worse than no HA cluster, because it gives false confidence.
- Requires partner
Sub-second RTO for stateful workloads across sites
Physics and replication latency set the floor. We document the real achievable number rather than the one on the slide.
- Requires partner
Application-level session draining or idempotency design
We will agree the contract with your application team; we do not own their code.
- Out of scope
Guaranteeing zero data loss for asynchronous replication
Asynchronous means non-zero RPO. We will size and label it precisely — sync across a WAN is often the wrong answer, and we will say so.
What it looked like last time
A two-site PostgreSQL HA environment that had never completed an automatic failover: fencing rebuilt on PDU-backed STONITH, drill executed, measured cross-site failover in 42 seconds with zero data loss and no application-side changes.
Anonymised under NDA. No client names, no logos, no invented metrics. Full case library on the work page.
Talk to the ha & infrastructure engineer about your high availability & failover
Send us the architecture, or just describe what broke most recently. You'll get an engineer's answer, not a discovery questionnaire.
Talk to a ha & infrastructure engineer: botonwa@gmail.com · +86 130 5181 3099