Utilisation is quoted from the scheduler
The scheduler reports the cards as allocated, so they count as busy. Actual device utilisation is much lower, and nobody can point at the difference.
Solutions · AI & ML
AI startups & ML teams
Expensive accelerators, unstable multi-node training, and no proof of where the time goes.
What makes this sector awkward
The hardware is the most expensive thing in the company and the least measurable. Training runs fail for reasons that look random, inference cost per request is unknown, and the team is under pressure to ship rather than to instrument.
GPU scheduling with gang placement, fabric and storage that keep the accelerators fed, and inference endpoints with a capacity model instead of a guess.
Four situations we see constantly
If two or three of these describe your environment, the audit is worth the money. If all four do, it is overdue.
Multi-node training is flaky in a way nobody can reproduce
It works on four nodes and hangs on eight, or it works until someone else starts a job. The pattern is at the interconnect or the scheduler, and both are being guessed at.
Inference capacity is a guess
Nobody can state requests per second per GPU for the actual prompt and completion length distribution, so both over-provisioning and paging incidents are routine.
The same card is configured differently on different nodes
Driver and runtime versions drifted, and a job that lands on the wrong node behaves differently. Time is lost to a version matrix nobody owns.
Four phases, each useful on its own
Sequenced so that stopping after any phase leaves you better off rather than stranded mid-migration.
- 01
Make utilisation honest
Measure at the device, per user and per job, over a real week. Separate allocated from actually busy, and name the idle tail.
You end up withA utilisation report per team and per job type, with the idle tail quantified.
- 02
Stabilise the version matrix
Collect driver, runtime and fabric firmware versions across every node, then converge on one validated set with deviations documented and excluded from general scheduling.
You end up withOne validated version set, and a scheduler that will not silently place a job on an outlier node.
- 03
Fix placement and collectives
Enable gang scheduling so multi-node jobs allocate all-or-nothing, then benchmark collective operations at your real job sizes.
You end up withA collectives baseline and a deadlock scenario that has been tested rather than theorised.
- 04
Give inference a capacity model
Benchmark with your own prompt and completion length distribution, then size for the latency target rather than for the vendor headline.
You end up withRequests per second per GPU at your target latency, and a scaling rule that survives the next traffic doubling.
The capability behind it
Each discipline has its own page with the technology stack, the acceptance criteria and an explicit scope boundary.
HPC & AI/GPU Cluster Services — Slurm, Kubernetes and Inference
Your GPUs are expensive and mostly idle. We fix the part that wastes them: the scheduler, the fabric, the storage feeding them, and the way jobs queue.
Slurm (SchedMD / NVIDIA builds) · Slinky (Slurm in Kubernetes) · OpenPBS / PBS Pro · IBM Spectrum LSFManaged Kubernetes & Container Platform Services
A cluster you can upgrade without holding your breath, whose etcd backup you have actually restored, and which you could rebuild from code in under an hour.
upstream kubeadm 1.31–1.34 · Kubespray · RKE2 · K3sObservability & Logging Cluster Engineering
Metrics, logs and traces that survive a node loss, alert on things that matter, and cost less than the infrastructure they watch.
Prometheus 2.5x · Thanos · Grafana Mimir · VictoriaMetricsDistributed Storage Cluster Consulting & Operations
Ceph, BeeGFS and MinIO clusters designed around your real failure domains, benchmarked with fio, and proven to survive a disk, a node, and a rack going away.
Ceph (Squid / Reef) · Cephadm + Orchestrator · Rook-Ceph 1.17 · Ceph RGW (S3)How this looks in practice
Research institute · North America
Forty machines, one Git repository, no one logging in by hand
12 min · per-node bring-up, measured on all 40
Read the engagement → B2B SaaS · North AmericaA cluster nobody dared upgrade, rebuilt so upgrades became routine
49 min · measured control-plane upgrade, zero dropped requests
Read the engagement → Logistics technology · EUThree major versions crossed, one workload restarted on purpose
3 · major hops completed in planned windows
Read the engagement →Every case in this library is a representative engagement, written from our delivery patterns and marked as such. We do not present unnamed work as a named customer.
What we measure in this sector
These are the numbers we put in the delivery report. If we cannot measure it, we do not promise it.
- GPU utilisation measured at the device, before and after
- All-reduce bandwidth baseline at real job sizes
- Requests per second per GPU at the target latency
- Cost per training run, per team
What we don't do
Every vendor says yes to everything. Here is where our answer is no, or where someone else needs to be in the room. If this list looks wrong for your situation, tell us — we would rather argue about scope before the contract than after.
- Out of scope
Model architecture, training methodology and research outcomes
We make the infrastructure fast and measurable. What you train on it is yours.
- Out of scope
GPU procurement and hardware sourcing
We do not resell hardware or take margin on it. We will tell you what to buy and what to refuse.
- Requires partner
ML platform vendor integration (experiment tracking, feature stores)
We will run the platform underneath and keep it available. Application-level integration needs the vendor or your own team.
- Included
Scheduling, fabric, storage tuning and utilisation instrumentation
Including the benchmarks you keep as a regression baseline.
AI & ML specifics
Slurm or Kubernetes for training?
By workload shape, not by fashion. Long, tightly-coupled multi-node jobs with gang requirements fit Slurm. Many shorter jobs with service dependencies fit Kubernetes. Most labs end up running both, and we will tell you where that line belongs in your case.
Can you bring up a cluster on hardware we already own?
That is the common case. We start by measuring what the existing hardware can actually do before recommending anything new — sometimes the answer is that the bottleneck is configuration, not silicon.
Do you help with inference cost?
Yes, and usually by measuring first. Batching, quantisation and the serving runtime often matter more than the GPU count, and we will show the numbers before recommending either.
The engineering transfers, the constraints do not
Financial services & fintech Regulated, audited, and unforgiving of downtime. Every change needs evidence. Healthcare & life sciences Imaging volumes grow faster than budgets, and patient data has strict handling rules. Media & entertainment Render and transcode farms starved by storage, with peaks that double overnight. SaaS & internet platforms Multi-tenant isolation, unpredictable growth, and a cloud bill that scales faster than revenue. Research & higher education Slurm on one side, Kubernetes on the other, and a budget committee in the middle. Manufacturing & edge fleets Dozens of small sites, intermittent connectivity, and nobody on site who can debug a node.
Want this looked at against your ai & ml environment?
Thirty minutes, no slides. Bring the awkward constraint — that is usually the interesting part.