The 38-point cluster health checklist

An HA pair that has never failed over under traffic is a configuration, not a capability. The failure mode people discover in production is session state that does not synchronise for the protocol they actually run.

Schedule a failover during business hours with real traffic. Time it from the first dropped packet to full recovery. Repeat in the reverse direction — asymmetric paths are the ones that break.

Two switches in an MLAG pair plugged into the same PDU, and both uplinked to the same upstream switch, is one switch with extra cabling. This is the most common false redundancy we find.

Trace power for both members back to the breaker. Trace every uplink to its upstream device. Draw it — the drawing usually makes the problem obvious.

BGP hold timers default to 90 seconds. Without BFD, a dead link can black-hole traffic for the better part of two minutes while the control plane decides.

Check whether BFD is configured per neighbour and what the detection multiplier is. Then pull a link and measure the actual reconvergence.

Backups that have never been restored are a hypothesis. Device firmware differences routinely make a restored config fail to boot the exact way it was saved.

Take a spare unit (or a lab instance), restore the newest backup, and bring the device into service. Record how long it took and what had to be adjusted by hand.

If switch management lives on the network that the switches provide, a failed configuration change takes away your ability to fix it. This is how a 20-minute outage becomes a site visit.

Confirm management interfaces are on a dedicated VLAN or physically separate switch, reachable when the data plane is down.

Jumbo frames configured on the storage VLAN but not on the interswitch links produce a cluster that works until an MTU-sensitive transfer touches a specific path. Symptoms look random.

Ping with the don't-fragment bit set at 8972 bytes (for a 9000 MTU) between every pair in the storage path. Watch interface counters for giants.

A replica that has never been promoted is a backup, not high availability. The failure that matters is not the promotion — it is whether the application reconnects without a restart.

Promote the replica in a controlled window with the application connected. Time the interruption. Then fail back and confirm replication resumes without a full rebuild.

Backup success is not restore success. Compression bugs, missing WAL segments, and permissions drift on the target all pass the backup job and fail the restore.

Pick a random backup from at least a month ago. Restore it to a clean host and bring the service up against it. Time the whole thing including validation.

Every modern ransomware incident finds the backups and encrypts them, because the backup target is reachable with the same credentials as everything else.

Identify whether any copy is append-only, object-locked, or physically offline. Ask what credential would be needed to delete it, and from where.

A replicated pool that places all three copies across OSDs on the same host is a two-copy pool with extra ceremony. Durability claims come from the map, not the replication number.

Inspect the CRUSH tree against the physical topology. Confirm failure domains are declared at host, rack and site level where they exist, and that pools use the right ruleset.

A disk replacement can trigger a rebalance that saturates the storage network and makes the healthy cluster slow enough that users notice — turning a disk failure into a service incident.

Check the recovery and backfill throttles against measured network headroom. Then test with a large rebalance during business hours, ideally in staging.

Two paths configured but one silently down means you have no redundancy and no alarm. The failure is only discovered when the surviving path fails too.

Inspect path state per LUN and per initiator. Look for failed or ghost paths. Verify path_selector matches the array's ALUA behaviour.

Storage that returns wrong data without an error is worse than storage that fails, because the corruption propagates into your backups.

Confirm scrubbing or checksums are enabled and running: pool scrub for Ceph, media scrub for arrays, pg_checksums or equivalent for databases.

Waiting for a disk to fail means you are always reacting, and in a rebuild-heavy week that means degraded redundancy for days.

Check SMART attributes for reallocated sectors, pending sectors, and growth trends. Confirm alerts fire on trend, not just on the failure flag.

Losing etcd means losing every object in the cluster. It is the single highest-consequence, lowest-cost thing to protect, and it is routinely the least tested.

Confirm snapshotting is scheduled, snapshots leave the node, and retention covers more than the interval. Then actually restore into a scratch cluster.

Clusters drift until the upgrade you need requires three intermediate versions, none of which were tested against your workloads. Skipping minor versions is not supported by anyone.

Compare the running version against the current supported releases. Check whether the next upgrade is a supported single step or a chain.

No limits means one workload can evict its neighbours. Limits set far too low means OOMKill loops that look like application bugs. Both are common and both are cheap to fix.

Compare declared requests and limits against a week of measured usage per workload. Look for the two extremes specifically.

A single misconfigured ingress resource can take down every route that shares the controller. Nobody notices until someone's typo becomes everyone's outage.

Identify what shares a controller and where a bad resource would fail. Check admission policy: is anything validating resources before they reach the cluster?

Credentials committed into manifests, or stored in a namespace everyone can read, are the most common finding in any cluster review. Rotation is the part that reveals whether management is real.

Find every place a credential is stored. Then perform one rotation end to end and record what broke.

If the registry is down, you cannot scale, roll back, or recover. It is usually deployed as an afterthought, without redundancy or capacity planning for garbage collection.

Check registry storage, replication and garbage-collection schedule. Test pulling an image with the registry's primary storage degraded.

Public certificates usually have automation. Internal ones usually do not, and internal expiry is where clusters go dark at 3am with a confusing error.

Enumerate every certificate the platform depends on, internal mTLS included. Compare expiry dates against who would notice and who would renew.