Ceph CRUSH map mistakes we keep finding

Ceph is one of the few systems where the thing that determines your durability is not the thing most people configure. Administrators set the replication count to three, see HEALTH_OK, and conclude they are protected against two simultaneous failures.

They are not. They are protected against two simultaneous OSD failures — and only if the CRUSH map decides to place the copies in three different places. If it decides otherwise, the replication count is decoration.

The failure domain is the product, the replication count is a request

CRUSH takes a placement request and a topology, then chooses OSDs that satisfy the request. The request says “three copies”. The topology determines whether three means three disks, three hosts, three racks or three sites.

The mistake is that people set the first and never examine the second.

The classic version: a cluster with size=3 on a pool whose CRUSH rule is chooseleaf firstn 0 type osd. Every copy goes to a different OSD — technically correct — but with no constraint that those OSDs live on different hosts. On a cluster where hosts hold many OSDs, the probability of two copies landing on the same host is not small, and the probability of three on two hosts is higher than anyone expects.

That is a two-host-failure away from data loss with a replication factor of three.

Four configurations we find repeatedly

Same-host collision. Failure domain set to osd on a cluster with a dozen OSDs per host. Fix: domain at host for the base durability claim, rack if racks are a real failure boundary.

A rack domain that does not exist physically. The CRUSH hierarchy declares two racks so pools can spread across them. Physically, both racks share a PDU and one top-of-rack switch. The map describes redundancy that the building does not provide.

Site replication by CRUSH on a single-site cluster. A stretch-cluster configuration where the second “site” is a set of hosts in the same room, on the same power. This gives you the operational cost of a stretch cluster without the availability property.

Journal and data on the same device. A durability failure rather than a placement one, but it appears in the same reviews. When the device fails, both the data and the intent log go with it, and the recovery path is a restore rather than a rebuild.

How to actually check

Start from the topology, not from the pool.

# What does CRUSH think the hierarchy is?
ceph osd tree

# What rule is the pool actually using, and what does it say?
ceph osd pool get <pool> crush_rule
ceph osd crush rule dump <rule-name>

Then ask the question the output does not answer for you: does the declared hierarchy match the building?

Map every host to its rack, its power feed and its switch. Compare that against ceph osd tree. Where they disagree, the map is wrong, because the map is what Ceph believes.

A useful sanity check is to simulate the loss and read the result rather than reasoning about it:

# Which OSDs would be chosen for a placement group?
ceph osd map <pool> <pgid>

# What happens to availability if a whole host disappears?
ceph osd crush rule dump <rule>

If you can remove a host from the map and a placement group still has all its copies, you have found a problem before it found you.

The rate limit is the next mistake

Once placement is fixed, the next issue is usually rebalance behaviour. A replaced disk starts a backfill that can saturate the storage network, and a cluster that is technically healthy becomes slow enough for users to notice. Every disk replacement becomes a service incident.

Set recovery and backfill throttles from measured bandwidth headroom rather than from the defaults, and test the setting with a deliberate full-disk rebuild during business hours in staging. The number you want is front-end latency staying inside target while a disk rebuilds, not the rebuild finishing quickly.

What good looks like

A Ceph cluster in good shape usually has three properties:

• The CRUSH hierarchy matches the physical failure domains, including power and switching, not just rack labels.
• Every pool’s rule has been chosen deliberately, and someone can explain what failures each pool survives.
• Recovery is throttled from measurement, and a disk rebuild does not produce a user-visible slowdown.

None of this is exotic. It is all findable in an afternoon. The reason it is not found is that HEALTH_OK is a reassuring colour, and it does not mean what people think it means.