A Ceph cluster losing OSDs weekly, rebuilt around placement instead of hardware
Ceph — distributed storage cluster
Where they started
The cluster dropped to a degraded state most weeks. Hardware had been replaced repeatedly, and every incident ended with a backfill storm that slowed production rendering.
The problem, precisely
The problem was not failing disks. It was a CRUSH layout and recovery configuration that turned one disk failure into a cluster-wide performance event.
What we did
Rebuilt the failure-domain model first, tuned recovery and backfill against measured client latency, then rolled the layout out rack by rack without taking the cluster offline.
Delivered
- Failure-domain map rebuilt: rack-level CRUSH rules replacing a flat layout
- Recovery, backfill and scrub settings tuned to a client-latency budget
- NVMe WAL/DB separation for the HDD tier, which removed a hidden write bottleneck
- Rolling layout change with per-rack verification, no cluster outage required
- Capacity and rebalance forecast tooled so growth is planned rather than survived
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- weekly → 0 degraded-state events in the following quarter
- — client latency under recovery, measured against budget
- 1.1PB raw capacity with no data migration required
- 0 hardware replaced to fix the underlying problem
Ceph gives you exactly the failure domains you describe. Most "unstable Ceph" is a CRUSH map that says every disk is equally related to every other.