This comes up in nearly every AI engagement, usually early, usually framed as a technical decision. It rarely is. The right answer falls out of how often your content changes and who has to maintain the thing.
The short version
Retrieval when the model needs to know things. Fine-tuning when the model needs to behave a certain way.
That single distinction resolves most cases. If the gap is knowledge — your product, your policies, your documentation, your customers’ data — retrieve it. If the gap is form — a house tone, a strict output structure, a domain vocabulary the model consistently gets wrong — fine-tuning is the tool.
Most mid-market SaaS problems are knowledge problems.
Why retrieval usually wins on operations
Content changes. Your pricing page changes, your policies change, a customer updates a document. With retrieval you update the source and the next answer is current. With a fine-tuned model, that knowledge is baked into weights, and updating it means retraining.
Retrieval also gives you three things that matter more in practice than raw answer quality:
Citation. You can show which document an answer came from. In any regulated or B2B context this is close to non-negotiable, and it is what makes users trust the feature enough to keep using it.
Access control. You filter what a given user can retrieve. A fine-tuned model knows what it knows for everybody — you cannot un-teach it per tenant.
Debuggability. When an answer is wrong you can look at what was retrieved and usually see immediately whether the failure was retrieval or generation. A wrong answer from a fine-tuned model is considerably harder to explain.
When fine-tuning genuinely earns its place
It is not never. Three cases where we would reach for it:
Consistent output structure. If you need reliable, strictly-shaped output at volume, fine-tuning gets you there more cheaply than a long prompt repeated on every call.
Domain language. Specialised vocabulary — clinical, legal, engineering — where the base model consistently misreads terms of art.
Cost at real volume. A fine-tuned smaller model can match a larger model’s quality on a narrow task at a fraction of the per-token cost. This only pays back above a certain volume, and the break-even is usually higher than people assume once you count the retraining work.
Note that none of these are “the model does not know about our product.” That is a retrieval problem wearing a fine-tuning costume.
The option people skip
Before either: make the prompt and the context better.
A surprising share of “we need to fine-tune” conversations end when someone actually looks at what is being sent to the model. Ambiguous instructions. No examples. Retrieved chunks that are too short to carry meaning, or too long and drowning the relevant sentence. No output format specified.
This is unglamorous and it is the highest return per hour of anything in the stack. We look here first, always, and often it is enough.
What we would actually recommend
For a typical mid-market B2B SaaS adding an AI feature over its own content:
- Fix the prompt and the retrieved context first
- Build retrieval properly — chunking that respects document structure, hybrid search rather than pure semantic, and re-ranking if precision matters
- Measure against an evaluation set you wrote before building
- Consider fine-tuning only if the remaining gap is behavioural rather than factual
Most projects stop at step three because the gap has closed. The ones that continue to step four do so for a specific, articulable reason — which is exactly how that decision should be made.
The question that settles it
If you are unsure, ask: when this information changes next month, who updates it and how long does it take?
If the answer is “someone edits a document” you want retrieval. If the answer involves a training run and an evaluation cycle, be certain the benefit justifies putting that on someone’s permanent workload. In a team of ten engineers, it often does not.