ZooKeeper retired without a single consumer message lost
Kafka — KRaft mode migration
Where they started
A ZooKeeper-based Kafka cluster was approaching end of support, and the coordination tier had become the part of the stack nobody wanted to operate.
The problem, precisely
Migration touches metadata for every topic and every consumer offset. A mistake is not a broker restart, it is a data-loss event.
What we did
Ran the full migration on a mirror cluster carrying replayed production traffic before touching the real one, then migrated with consumers rehearsing the switch in the same window.
Delivered
- KRaft migration rehearsed end to end on a mirror cluster with replayed traffic
- Controller quorum sized and placed across three failure domains
- Consumer group offset verification before and after, with a diff report
- Partition reassignment plan so no broker was above a defined leadership share
- Rollback position documented for every step, with the abort criteria agreed in advance
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 0 messages lost across the migration
- 120 consumer groups verified by offset diff
- 1 dress rehearsal before production was touched
- — metadata operation latency, measured before and after
Rehearse the whole migration against replayed traffic. Kafka migrations fail on consumer offsets far more often than on broker configuration.