Expensive accelerators sitting idle, made visible and fixed without buying more
GPU training cluster — utilisation & scheduling
Where they started
GPUs were the largest line in the infrastructure budget, and the team described utilisation as "probably fine" without being able to produce a number.
The problem, precisely
Cards were allocated to jobs that were not using them, and no one could distinguish a queueing problem from a training-efficiency problem.
What we did
Instrumented per-card utilisation and memory first, then separated the two failure modes the data revealed: idle-but-allocated cards and jobs running far below expected scaling efficiency.
Delivered
- Per-card utilisation and memory instrumentation with per-user attribution
- Scheduler policy changed so allocated-but-idle cards are reclaimed after a grace period
- Multi-node scaling tests run per job class, revealing a fabric misconfiguration
- Gang scheduling so multi-node jobs stop half-starting and holding partial allocations
- Monthly utilisation report per team, published to the researchers themselves
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 31 → 78% GPU utilisation, before → after
- 0 additional GPUs purchased
- 1 fabric misconfiguration found by scaling tests
- 100% of GPU hours now attributed to a team
Instrument before you buy. In every GPU cluster we have measured, at least a fifth of the expensive capacity was allocated to work that was not running.