Queue time cut from days to hours by fixing the scheduler, not by buying nodes
Slurm — HPC batch cluster
Where they started
Users reported that jobs waited days for a slot while node utilisation sat near 60%. The default assumption was that the cluster needed more hardware.
The problem, precisely
Nobody could say where the idle capacity was, because the scheduler was not producing accounting data anyone trusted.
What we did
Instrumented the queue first — accounting, per-partition utilisation and job-size distribution — then changed policy based on what the data showed rather than expanding the cluster.
Delivered
- Accounting enabled and trustworthy, with per-user and per-project reporting
- Job-size distribution analysis showing a small number of large jobs blocking partitions
- QoS and partition redesign with backfill enabled and a tested preemption policy
- Node health checks automated so bad nodes leave the pool instead of accepting jobs
- User-facing reporting: expected wait time per partition, published rather than guessed
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- days → hours p90 queue time for mid-size jobs
- 60 → 89% node utilisation
- 0 compute nodes purchased to achieve it
- 1 accounting source of truth, no longer disputed
HPC performance is usually a policy problem wearing a hardware costume. If you cannot report utilisation per partition, you are guessing about what to buy.