A search cluster that fell over at 60% load, rebuilt around shard planning
Elasticsearch / OpenSearch cluster
Where they started
The cluster was running at 60% of CPU and still timing out. Nobody had looked at shard counts since the data had been one twentieth of its current size.
The problem, precisely
Shard count had grown to over 3,000, most of them small, and the cluster spent more time on coordination overhead than on searching.
What we did
Reindexed to a shard strategy based on actual index sizes and query patterns, split hot and cold data into different tiers, and gave the cluster a lifecycle policy instead of manual deletion.
Delivered
- Shard strategy rebuilt around index size and query volume, with the reasoning documented
- Hot-warm-cold tiers so expensive storage only holds what is actually queried
- Lifecycle policy replacing manual index deletion and the outages it caused
- Snapshot repository with a tested restore of a single index and a full cluster
- Query-level slow-log analysis with three specific heavy aggregations refactored
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 3,000+ → 240 shards after reindexing
- — p95 query latency measured before and after
- 0 index deletions by hand since the policy was set
- 1 node failure survived during business hours, unremarked
Shard count is the most neglected number in search operations. Most "the cluster is slow" tickets are a reindexing problem pretending to be a capacity problem.