Observability & Logging Cluster Engineering
Metrics, logs and traces that survive a node loss, alert on things that matter, and cost less than the infrastructure they watch.
What this usually looks like
You have dashboards. What you do not have is an answer at 3am, or a monitoring bill you can defend.
- Alert fatigue: everyone has muted the channel.
- The monitoring stack runs on the same cluster it monitors, so it dies with it.
- Metrics retention is short because storage is expensive, so you cannot compare to last quarter.
- Nobody knows what any given service costs per month.
Three ways to buy this
Audit if you want to know how bad it is. Build if you have the team and want the capability. Managed if you don't want to own the pager. Most engagements start with the audit and decide afterwards.
Audit
A fixed-price health check — the cheapest way in
- Alert review: how many fire, how many are actionable, how many are noise
- Monitoring stack availability and independence analysis
- Retention and cardinality cost review
- SLO coverage: which user-facing promises are actually measured
Build
One-off delivery: design, code, hardening, handover
- Prometheus in HA with Thanos or Mimir for long-term, queryable retention
- Loki or Elastic for logs with structured parsing and retention tiers
- OpenTelemetry Collector pipeline for traces, with sampling that makes sense
- Alertmanager routing to PagerDuty/Opsgenie/Grafana OnCall with escalation policy
- SLO/SLI definitions with error-budget dashboards
- Deep diagnostics tooling: Cilium Hubble, Pixie, bpftrace, perf
Managed
Build plus 7×24 on-call, SLA and quarterly drills
- Alert-noise reduction with a measured before/after count
- Retention and cardinality tuning as services change
- SLO review and error-budget reporting per quarter
- Runbook automation for the incidents you have already seen twice
What we actually deploy
Specific tooling, not categories. If a version matters to your decision, ask — we will tell you which one we would pick and why, including when the answer is the older one.
Metrics
- Prometheus 2.5x
- Thanos
- Grafana Mimir
- VictoriaMetrics
- Zabbix
- Telegraf
- node_exporter
- DCGM exporter
Logs
- Loki + Promtail
- Grafana Alloy
- Elasticsearch / OpenSearch
- Vector
- Fluent Bit
- Graylog
Traces
- Grafana Tempo
- Jaeger
- OpenTelemetry Collector
- Tail sampling
Visualisation & SLO
- Grafana
- Perses
- Sloth
- Pyrra
- error-budget dashboards
On-call
- Alertmanager
- PagerDuty
- Opsgenie
- Grafana OnCall
- escalation policies
- runbook links in alerts
Cost
- OpenCost
- kubecost
- cloud billing exporters
- per-namespace cost allocation
Deep diagnostics
- Cilium Hubble
- Pixie
- bpftrace / BCC
- perf + flame graphs
- fio
- iperf3
- sysbench
What we will be measured on
These are the numbers we put in the delivery report. If we cannot measure it, we do not promise it.
- Alert volume reduced to a measured, quantified number — with the count before and after.
- MTTD (mean time to detect) reported, not promised.
- Every alert links directly to the runbook that resolves it.
- Monitoring remains available when the monitored cluster is not.
- Long-term retention queryable across quarters at a stated cost per month.
What we would build, and where the weak point is
Alertmanager with a single replica and no readiness alerting is the classic silent failure: monitoring keeps collecting, and nobody gets paged. It gets replicated, or at minimum monitored by something outside itself.
marks a single point of failure we would raise with you rather than hide. Every architecture has one; the question is whether you know where it is.
What we don't do
Every vendor says yes to everything. Here is where our answer is no, or where someone else needs to be in the room. If this list looks wrong for your situation, tell us — we would rather argue about scope before the contract than after.
- Out of scope
Building dashboards for every team and product KPI
We deliver the platform, the SLO framework and a small set of golden dashboards. Dashboards that reflect your product are your product team's work.
- Requires partner
Acting as your 24×7 NOC for events outside infrastructure
Our on-call covers infrastructure we operate. Application-level incidents need your application on-call.
- Out of scope
Unlimited log retention at fixed price
Retention is storage, and storage has a price. We will design tiers and show you the trade-off curve.
- Requires partner
Replacing a dedicated APM SaaS for deep application profiling
OpenTelemetry gets you far. Some teams still want a commercial APM for language-specific profiling; we integrate rather than fight it.
What it looked like last time
A platform team receiving roughly 900 alerts a week: thresholds reworked around symptoms instead of causes, alert volume cut to about 60 per week, with mean time to detect on real incidents improving from minutes to under 90 seconds.
Anonymised under NDA. No client names, no logos, no invented metrics. Full case library on the work page.
Talk to the observability engineer about your observability clusters
Send us the architecture, or just describe what broke most recently. You'll get an engineer's answer, not a discovery questionnaire.
Talk to a observability engineer: botonwa@gmail.com · +86 130 5181 3099