Message Queue, Streaming & Cache Cluster Services
Kafka, Pulsar, NATS and Redis clusters that survive a broker loss without losing a record, and that tell you about consumer lag before your customers do.
What this usually looks like
Your event backbone is doing more critical work than your database, and it gets a fraction of the attention.
- Nobody knows the true retention-to-disk ratio, so capacity planning is guesswork.
- Broker maintenance is scary because a rebalance could stall consumers for minutes.
- Consumer lag is visible only when someone complains downstream.
- Cache invalidation is a group chat.
Three ways to buy this
Audit if you want to know how bad it is. Build if you have the team and want the capability. Managed if you don't want to own the pager. Most engagements start with the audit and decide afterwards.
Audit
A fixed-price health check — the cheapest way in
- Partition, replication-factor and ISR review
- Consumer-lag and rebalance-behaviour analysis under peak
- Retention and capacity projection
- Cache hit-ratio and eviction-policy review
Build
One-off delivery: design, code, hardening, handover
- Kafka in KRaft mode (no ZooKeeper) with tiered storage where it pays off
- Strimzi-managed Kafka on Kubernetes with rack awareness
- Pulsar, NATS/JetStream, RabbitMQ quorum queues — chosen per workload, not per fashion
- Redis Cluster / Sentinel with persistence configured deliberately
- MirrorMaker2 / cluster-linking for cross-site replication
- Schema Registry and CDC pipelines (Debezium, Flink CDC)
Managed
Build plus 7×24 on-call, SLA and quarterly drills
- Rolling broker restarts with rebalance minimisation
- Lag alerting with thresholds tuned per consumer group
- Retention and tiered-storage policy management
- Throughput and disk-growth forecasting
What we actually deploy
Specific tooling, not categories. If a version matters to your decision, ask — we will tell you which one we would pick and why, including when the answer is the older one.
Kafka ecosystem
- Kafka 3.7+ KRaft
- tiered storage
- Strimzi operator
- MirrorMaker 2
- Confluent Schema Registry
- Kafka Connect
- Kraft controller quorum
Alternatives
- Apache Pulsar
- NATS / JetStream
- Redpanda
- RabbitMQ quorum queues
- RocketMQ
- ActiveMQ Artemis (legacy)
Cache & KV
- Redis Cluster
- Redis Sentinel
- Valkey
- Dragonfly
- KeyDB
- Memcached
CDC & streaming ETL
- Debezium
- Flink CDC
- Kafka Streams
- ksqlDB
- Materialize (evaluation)
Coordination
- ZooKeeper (legacy estates)
- etcd
- Consul
- KRaft migration paths
What we will be measured on
These are the numbers we put in the delivery report. If we cannot measure it, we do not promise it.
- A single broker loss causes no production outage and no record loss, demonstrated by test.
- Consumer lag has an alert threshold per group, so you learn about it before your users do.
- Rolling broker restart with rebalances constrained and the impact measured.
- Cross-site replication latency is a monitored metric with a stated SLO.
- Retention and capacity modelled against real growth, not a default 7 days.
What we would build, and where the weak point is
In KRaft mode the controller quorum carries the metadata every broker depends on. Three controllers across three failure domains is the floor — on one rack it is a liability.
marks a single point of failure we would raise with you rather than hide. Every architecture has one; the question is whether you know where it is.
What we don't do
Every vendor says yes to everything. Here is where our answer is no, or where someone else needs to be in the room. If this list looks wrong for your situation, tell us — we would rather argue about scope before the contract than after.
- Out of scope
Authoring stream-processing business logic
We build and operate the platform. Flink or Kafka Streams jobs that encode your rules are your application.
- Requires partner
Guaranteeing exactly-once semantics end to end
Exactly-once requires idempotent sinks your team controls. We will document where the guarantee actually holds and where it degrades to at-least-once.
- Requires partner
Message broker schema compatibility decisions
We run Schema Registry and enforce compatibility policy you define.
- Out of scope
Unlimited retention without a storage budget
Retention always has a cost. We will show you the number and let you choose it.
What it looked like last time
A Kafka cluster on ZooKeeper migrated to KRaft with zero message loss across the cutover, and a three-broker rolling maintenance window reduced from a projected 20-minute consumer stall to under 4 seconds of measured lag increase.
Anonymised under NDA. No client names, no logos, no invented metrics. Full case library on the work page.
Talk to the data platform engineer about your caching & messaging
Send us the architecture, or just describe what broke most recently. You'll get an engineer's answer, not a discovery questionnaire.
Talk to a data platform engineer: botonwa@gmail.com · +86 130 5181 3099