A replica set that outgrew one machine, split into a sharded cluster with a controlled balancer
MongoDB — sharded cluster
Where they started
A three-member replica set was near the write ceiling of its primary. Vertical scaling had reached the top of the available instance size and IOPS were saturating.
The problem, precisely
Sharding introduces a new failure mode: the balancer. Uncontrolled chunk migration will saturate the same IOPS you were trying to relieve.
What we did
Chose a shard key against the actual query patterns before creating any shards, then enabled balancing with a window and a migration rate limit.
Delivered
- Shard key selection analysis against production query patterns, with the alternative rejected and why
- Sharded cluster with config server replica set and multiple mongos routers behind a VIP
- Balancer window set outside peak hours, with a migration rate ceiling
- Consistent backup with snapshot verification and a tested restore procedure
- Runbook for adding a shard without unbalancing the hot collection
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 45k writes per second sustained after migration
- 0 user-visible slowness during chunk migration
- 1 shard key chosen against real query patterns, not defaults
- < 30 min measured restore of a sharded snapshot to scratch
Pick the shard key from the query log, not from the schema diagram. A sharded cluster with the wrong key is slower than the single node it replaced.