A model served at the latency the product needed, on the GPUs they already owned
LLM inference cluster
Where they started
An application needed model inference at interactive latency, and the initial single-node deployment could not hold the latency target once concurrency rose.
The problem, precisely
Throughput and latency pull in opposite directions once batching is involved, and the team had no headroom to buy more accelerators.
What we did
Benchmarked the actual request mix first, then sized the deployment around the real prompt and completion length distribution instead of the vendor’s headline numbers.
Delivered
- Benchmark harness reproducing the production request mix, reusable for future model changes
- Tensor-parallel deployment across nodes with the interconnect verified under load
- Continuous batching and KV-cache settings tuned against measured p95 latency
- Autoscaling driven by queue depth and time-to-first-token rather than a proxy metric
- Cost-per-million-tokens report so serving cost is a known number, not an estimate
Technologies used
Measured result
Every number below was measured and reported to the customer. Where we do not have a measured figure, the entry is a fact rather than a percentage — we would rather be precise than impressive.
- 900/s requests per second sustained at target p95
- — time to first token, measured and published internally
- 0 additional GPUs purchased
- 1 benchmark harness, reused for every subsequent model
Serving performance is decided by your request mix, not by the model name. Build the benchmark harness before you choose the deployment topology.