Running a large open-source model on your own hardware is a weekend project. You download the weights, point an inference server at them, send a request, and it answers. The demo works.
Then you put ten users in front of it at once and everything you thought you knew about the system stops being true.
We went through this deploying Qwen 2.5 72B on a client’s own infrastructure. The brief was specific: concurrent API request handling. Not “make it work” — make it work when requests arrive together rather than politely one at a time. What follows is what that actually involves.
Memory is the first wall, and it is not subtle
A 72-billion-parameter model in 16-bit precision needs roughly 144GB just for weights. Before a single request arrives. That is already more than a single 80GB accelerator, so you are either quantising, sharding across devices, or both.
But weights are the fixed cost. The variable cost is the KV cache — the attention keys and values held for every token in every active sequence. It grows with context length and with the number of concurrent requests, simultaneously. A setup that comfortably serves one user with a long document may fall over with twenty users asking short questions, or the reverse, depending entirely on where your traffic sits.
This is the thing teams miss when they size hardware from a single-user benchmark. Your capacity is not a number. It is a surface, and your traffic pattern decides which point on it you live at.
Throughput and latency pull in opposite directions
Batching is how you get throughput out of a GPU. Instead of running the model once per request, you group requests and run them together, and the accelerator does far more useful work per unit of time.
The cost is that batching makes individual requests wait. A request that arrives just after a batch has started either joins the next one or forces a decision about interrupting.
Continuous batching — where sequences join and leave a running batch as they finish rather than waiting for the whole batch to complete — is the standard answer, and it genuinely helps. But it does not remove the trade-off, it just moves it. You still choose a maximum batch size. Larger means better aggregate throughput and worse tail latency. Smaller means the opposite.
There is no correct answer to that dial in the abstract. There is only a correct answer for a given product. Which is why the first question we ask on this kind of work is not about the model.
The question that decides the architecture
Which does your user actually feel — time to first token, or time to last token?
For a chat interface streaming a response, time to first token is nearly everything. The user sees words appearing and perceives the system as fast, even if the full answer takes fifteen seconds. You can batch fairly aggressively and they will not notice.
For a system that returns a structured result — a classification, an extraction, a JSON object another service consumes — nothing is visible until it is all there. Time to last token is the only number that matters, and aggressive batching directly hurts the thing the user experiences.
Most teams never ask this. They tune for a benchmark number, usually tokens per second aggregate, and end up optimising for a metric no user has ever perceived.
Bursty traffic breaks steady-state assumptions
Load tests tend to model traffic as a constant rate. Real traffic is not constant. It arrives in bursts, and the bursts are correlated — everyone opens the tool at 9am, the batch job kicks off at midnight, the customer demo happens on Tuesday.
Two systems with identical average throughput behave very differently under bursts, depending on how they queue. A system that accepts everything and degrades gracefully looks slow. A system that rejects past a threshold looks broken. Which one is right depends on whether a slow answer is worth more to your users than a clear failure — and again, this is a product decision that ends up encoded in infrastructure.
Admission control is unglamorous and it is what keeps a service predictable. Deciding in advance what happens at capacity is better than finding out.
What we would tell someone starting this
Benchmark with your traffic, not with a script. Take a day of real request logs — lengths, arrival times, the burst shape — and replay it. A synthetic benchmark at a constant rate will tell you a number that is real and useless.
Measure p95 and p99, not the average. Average latency hides exactly the behaviour that makes users abandon a product. The tail is the experience.
Decide the failure mode before you need it. Queue, shed, or degrade. Pick one deliberately.
Compare honestly against the commercial API. Self-hosting makes sense for data residency, for per-token cost at genuine volume, or for latency you cannot get otherwise. It does not make sense because it feels more sovereign. Work out your real cost per million tokens, including the engineer-hours to keep it running, and compare. Sometimes the answer is that you should not do this, and that is a good outcome from an assessment.
Instrument before you optimise. Queue depth, batch occupancy, cache hit rate, time to first token and time to last token, separately. Most tuning arguments dissolve once someone can see the numbers.
The part that is not about the model at all
The thing that surprises people is how little of this work is machine learning. It is capacity planning, queueing behaviour, memory budgeting and failure design — the same discipline as any system that has to stay up under load.
We have been doing that since 2013, on healthcare platforms and payment systems where the failure modes were considerably less forgiving than a slow token. The model is new. The engineering problem underneath it is not, and teams that treat it as an ML problem rather than a systems problem tend to find that out expensively.