The array was blamed for slow writes. The fabric was the problem.
SAN fabric — Fibre Channel, multipathing
Where they started
Write latency had doubled over a quarter. The array vendor found nothing wrong, and the fabric had never been reviewed since an integrator built it four years earlier.
The problem, precisely
Every layer produced healthy-looking counters, and the blame was being routed between vendors with the customer in the middle.
What we did
Mapped both fabrics, sized the ISLs against measured host traffic, and measured per-path latency instead of per-array — which is where the divergence showed up.
Delivered
- Both fabrics mapped with ISL oversubscription calculated against real traffic, not port counts
- A failing optic found on one port, negotiating at half speed for an unknown period
- Host multipath corrected on 21 hosts that were silently single-pathed
- Zoning reviewed to a single-initiator single-target policy with a change plan
- Quarterly path-failover drill written into the operations calendar
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- ~2× p99 write latency improvement
- 21 hosts found single-pathed by accident
- 0 storage purchased to fix it
- 1 fabric review, four years overdue
Measure per path, not per array. Storage performance arguments between vendors are usually resolved by a latency number neither of them was collecting.