Four monitoring tools, three of them lying. One cluster that everyone trusts.
Observability cluster (metrics, logs, alerting)
Where they started
Metrics, logs and alerting had accumulated over years into four overlapping tools. Every team trusted a different dashboard, and alert fatigue had led to routing rules being muted.
The problem, precisely
Consolidating monitoring is politically harder than building it, and any downtime in visibility during the migration would be treated as an outage.
What we did
Ran the new stack in parallel with the old for six weeks, compared alert output on the same infrastructure, and only retired a tool when its replacement had demonstrated parity on real incidents.
Delivered
- Long-term metric storage with deduplication across HA Prometheus pairs
- Log pipeline with structured retention tiers instead of one retention for everything
- Alert rules rewritten around user-visible symptoms, with ownership attached to each alert
- Six-week parallel run with an alert-by-alert parity comparison before any retirement
- On-call handbook: every alert page has a runbook link and a named owner
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 4 → 1 monitoring stacks after consolidation
- 9M active series with long-term retention
- — page volume per week, measured before and after
- 0 visibility gaps during the parallel run
Consolidate with a parallel period and a parity comparison. Retiring a monitoring tool before its replacement has handled a real incident is how teams lose trust in observability permanently.