Why this one matters. Of everything on this site, this is the case study most likely to win an AI engagement. Very few agencies have run a 72B open-source model in production under concurrency, and fewer have written about it. Filling in the sections below is the highest-value hour of content work in the whole redesign.
Context
Our client needed a large language model running on their own infrastructure rather than behind a commercial API.
NEEDS INPUT: why they needed it in-house. The usual reasons are data residency, per-token cost at volume, latency, or a regulatory constraint on sending data to a third party. Which was it?
Problem
Getting an open-source model to answer one question on a workstation is a weekend. Getting it to answer many questions at once, reliably, is the actual engineering.
At 72 billion parameters the constraints bite immediately. The model has to fit in available memory before it serves anything. Every concurrent request competes for the same GPU. Batching raises throughput but costs individual latency. And a serving setup that behaves fine in testing can degrade sharply the moment real traffic arrives in bursts rather than evenly.
Our brief centred on concurrency: the system had to hold up when many requests arrived together, not just when they arrived politely one at a time.
Approach
The work split into three questions:
How the model is served. Configuration and serving architecture, so the model is loaded and addressable as a stable API rather than a script someone runs.
How concurrent requests are handled. Queuing and batching so that simultaneous requests are served without one starving the others.
How it behaves under sustained load. Performance stability, so throughput does not collapse and latency stays predictable as traffic rises.
NEEDS INPUT: the specifics that make this credible. Which serving framework — vLLM, TGI, SGLang, something custom? What hardware? Quantised or full precision? What batching strategy? Each of these is one sentence and each one is worth more than a paragraph of adjectives.
What we built
Deployment and configuration of Qwen 2.5 72B, a serving architecture exposing it as an API, and concurrent request handling tuned for stability under load.
NEEDS INPUT: the component list.
Results
NEEDS INPUT — this is the metric slot that matters most on the entire site. Any one of these would carry the homepage card:
- Concurrent requests sustained
- Tokens per second at that concurrency
- p50 and p95 latency
- Cost per million tokens compared with the commercial API it replaced
Even one number, approximate and client-approved, transforms this page.
Related
Retrieval-Augmented Generation for an Education Platform · AI Engineering