Cache outages stopped taking the site down, on a cluster sized for real failover
Redis Cluster / Valkey
Where they started
A single large Redis instance with one replica served all caching and session state. A failover lost user sessions, and memory pressure caused evictions under load.
The problem, precisely
The cache and the session store had different durability requirements but shared one instance, so eviction policy was a compromise that suited neither.
What we did
Split the two workloads, then clustered the caching tier with replicas sized so that losing a shard does not overload the survivors.
Delivered
- Cache tier as a proper cluster with per-shard replicas and documented slot mapping
- Session tier separated onto Sentinel-managed instances with persistence configured
- Eviction policy and maxmemory tuned per tier, with alerting before eviction starts
- Failover drilled: one shard primary killed, key routing verified automatically
- Hot-key analysis with a fix for one key responsible for 8% of traffic
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 1 shard loss tolerated with no user-visible impact
- 8% of traffic traced to a single hot key, now sharded
- 0 sessions lost during failover drills
- 900k operations per second sustained at peak
Cache and session state are two different products sharing a process. Separate them, because your eviction policy cannot be both permissive and protective.