The load balancer was the single point of failure everyone had forgotten
L4/L7 load balancing — highly available pair
Where they started
A single HAProxy instance fronted every public service. It had been rebuilt twice in place during upgrades, and the configuration existed only on that host.
The problem, precisely
The most available part of the architecture had a single host and an untested health-check model — several backends were marked healthy while serving errors.
What we did
Rebuilt as a proper pair with a floating address and, more importantly, replaced the shallow health checks with ones that test the thing the user actually receives.
Delivered
- Active/passive pair with a floating VIP, failover tested under production traffic
- Health checks rewritten to test application readiness rather than a TCP port
- Configuration managed in Git with validation and a staged reload procedure
- Backend pool rebuilt from static lists to service discovery, removing manual edits
- Capacity test: request rate at which the pair saturates, documented
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- < 1s measured VIP failover under load
- 140 backends now discovered, not hand-listed
- 0 services unreachable during the migration
- — saturation point measured rather than assumed
A health check that passes while the application returns 500s is worse than no health check. Test what the user receives, not what the socket accepts.