An event backbone across three sites, with leaf nodes that survive a site losing its link
NATS JetStream super-cluster
Where they started
Sites published telemetry to a central broker over WAN links that were not reliable, and a link failure meant losing data from that site entirely.
The problem, precisely
The design had to tolerate a site running disconnected for hours without losing telemetry or requiring manual reconciliation.
What we did
Deployed a leaf node per site with local JetStream persistence and an account-based subject hierarchy, so a disconnected site buffers locally and drains on reconnect.
Delivered
- Super-cluster across three sites with leaf nodes and defined stream replication factors
- JetStream persistence sized against measured worst-case link outage duration
- Account and subject design so tenants can be isolated without separate brokers
- NKey/JWT credential rotation procedure, tested including a compromised-key drill
- Link-failure test: a site disconnected for six hours, reconnected, drained cleanly
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 0 messages lost during a six-hour link outage test
- 20k/s telemetry messages sustained at peak
- 3 sites operating autonomously
- 1 credential compromise drill completed
Size the local buffer against the longest realistic disconnection, not the average one. WAN outages are measured in hours, not in packet loss.