One DNS server was the whole company single point of failure. Now four, over two sites.
Anycast DNS cluster + DHCP HA
Where they started
A single physical DNS server on a rack shelf served internal resolution for the entire group. When it had a problem, every office lost name resolution and every application stopped.
The problem, precisely
Nobody could safely change the zone file, DNS and DHCP had no redundancy, and the design had not been revisited since it was one office.
What we did
Rebuilt resolution as an anycast pair per site with a hidden primary, then moved DHCP to Kea in an HA pair with the same discipline around change control.
Delivered
- Anycast pair per site with a hidden primary, so zone changes are staged rather than live
- Zone management moved into Git with validation and a two-step publish process
- Kea DHCP in an HA pair with lease database replication and a tested failover
- DNSSEC signing enabled with a key rollover procedure written down
- Failure drill: one site taken out of service, resolution continued from the other
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 1 → 4 resolver instances, across two sites
- 0 resolution outages in the following two quarters
- 1 documented zone change process, replacing a phone call
- 12,000 clients with the same DHCP reservations, now replicated
Nobody plans capacity for DNS because it is invisible. It is also the one service where failure is simultaneous for every user in the company.