Kubernetes DRA: what actually changes for GPU clusters
Dynamic Resource Allocation replaces the device plugin model with something closer to how GPUs really work — but the operational questions it answers are narrower than the marketing suggests. Here is what changes, and what does not.
For most of Kubernetes’ life, a GPU has been modelled as an opaque integer. A node advertises nvidia.com/gpu: 4, a pod requests one, and the scheduler hands over a number. Everything that matters about that GPU — which model, how much memory, which NVLink topology, whether it can be shared — has been invisible to the scheduler.
The device plugin interface was designed for that simplicity, and it worked well until accelerators became varied and expensive enough that treating them all as a count stopped being good enough.
What DRA changes
Dynamic Resource Allocation reverses the model. Instead of a count, a device is described by its attributes, and a workload requests devices by those attributes through a resource claim. The scheduler matches claims against available devices using a driver that understands the hardware.
In practice this means three things change:
Requests become expressive. You can ask for a device with a specific memory size, a device from a specific vendor, or a device with particular interconnect characteristics — rather than asking for “one GPU” and hoping.
The scheduler becomes aware of locality. Because claims describe attributes, the scheduler can group workloads that need to be near each other, which matters for multi-GPU and multi-node training where interconnect topology dominates throughput.
Sharing becomes explicit. Time-slicing and multi-instance GPU are currently configured outside Kubernetes and are invisible to it. Under DRA, a device can declare that it supports sharing, and claims can request a portion rather than the whole.
What does not change
This is the part worth being clear about, because the enthusiasm around DRA has produced a lot of over-claiming.
It does not make your fabric faster. If collectives collapse under all-reduce, DRA will place workloads more intelligently and they will still collapse. Interconnect performance is a hardware and tuning problem, and DRA is a placement mechanism.
It does not solve fragmentation by itself. Attribute-based requests let the scheduler avoid bad placements, but a cluster fragmented across many partially-used GPUs is still fragmented. Better placement reduces the rate at which it happens; it does not fix a cluster that is already in that state.
It does not replace the need to measure utilisation. The gap between “allocated” and “actually computing” is a measurement problem. DRA gives you finer-grained allocation; it does not tell you whether the allocated device is doing anything.
It does not remove the version matrix problem. Driver, CUDA and fabric versions still have to agree across nodes, and a job placed on an outlier node still behaves differently. That is an operational discipline, not an API.
What to do about it
If you are running a GPU cluster today, the practical advice is unglamorous:
Fix the measurement first. Device-level utilisation, per user and per job, over a real week. Without that, no scheduling improvement is verifiable — you cannot tell whether DRA helped or whether the workload changed.
Get the version matrix under control. One validated driver and runtime set across the fleet, with documented deviations excluded from general scheduling. This is worth more than any scheduler feature and it takes a day.
Enable gang scheduling if you run multi-node jobs. A job that holds half the cluster while waiting for the other half deadlocks against a job doing the same thing. This is a scheduling configuration, and it is independent of DRA.
Then evaluate DRA for the specific problem you have. If your problem is that you cannot express “four GPUs on the same NVLink domain”, DRA is the answer and it is a real improvement. If your problem is that utilisation is low, DRA will not fix it.
The honest summary
DRA is a genuine improvement in how Kubernetes models accelerators. It replaces an integer with something that resembles the hardware. If you run heterogeneous GPUs, or you need topology-aware placement, or you want sharing to be a first-class concept rather than a side configuration, it is worth the migration.
If your cluster is one model of GPU, allocated whole, on a fabric that works — you have a placement problem already solved by simpler means, and the effort is better spent on the utilisation question. Which is, in our experience, the question that actually has money behind it.