Four nodes to eleven, with the storage rebalancing planned rather than survived
Hyperconverged cluster (Proxmox VE + Ceph / Harvester)
Where they started
A four-node cluster was running at 82% capacity with no headroom for a failover, and the last expansion attempt had caused a visible performance dip during the day.
The problem, precisely
Adding nodes rebalances data across the whole cluster, and the previous attempt did it under peak load with no throttling.
What we did
Modelled the CRUSH redistribution before touching hardware, then added nodes one at a time with backfill throttled to a measured ceiling and monitoring on client latency rather than cluster health alone.
Delivered
- Growth model: capacity, failure-domain and rebalance-time forecast to 20 nodes
- Three expansion steps, each with backfill throttled and latency watched
- Failure-domain review — the previous layout had two nodes sharing a PDU
- Client-visible latency budget written down and monitored during every change
- Spare-node plan so a failure during expansion does not become an outage
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 82% → 54% cluster utilisation after expansion
- 0 user-visible incidents during rebalancing
- 2 nodes found sharing a single power feed
- 1 failover now survivable at full production load
Rebalancing is a scheduling problem, not a hardware problem. Throttle backfill against client latency and expansion stops being a maintenance-window event.