Eleven-minute outages became eight-second failovers, with restores nobody had ever timed
PostgreSQL — Patroni HA across three failure domains
Where they started
Failover required a senior engineer to be reachable, average unplanned outages ran to eleven minutes, and no backup restore had ever been timed in production.
The problem, precisely
The database held the ledger. Both the availability story and the recovery story had to become measured facts rather than confident assumptions.
What we did
Rebuilt the consensus layer across three genuinely separate failure domains, then ran the first real restore into a scratch environment and published the time.
Delivered
- Patroni-managed cluster with etcd quorum spanning three failure domains
- Synchronous commit policy per workload class, documented trade-offs
- pgBackRest with off-site storage, WAL archiving and retention policy
- First timed point-in-time restore, run with the customer watching
- Failover and restore runbooks, both rehearsed by the customer team
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 8s measured automatic failover
- 14 min measured full restore from backup
- 100% of failovers now automatic
- 11 min → 8s unplanned outage duration, before → after
Consensus needs three real failure domains. Three nodes in one rack gives you the operational cost of quorum with none of the protection.