Adding an AI feature to a product that already has users

Greenfield AI products get all the attention. Most actual work is the other thing: a product that already exists, already has customers, already makes money, and now needs an AI feature that does not break any of that.

This is harder, and it is harder for reasons that have nothing to do with machine learning.

Your data model was not designed for this

Retrieval needs content in coherent, retrievable units. Your production database has content optimised for transactions — normalised across tables, sliced by tenant, with the useful context spread across six joins and three services.

Getting from one to the other is most of the work. It is an ingestion pipeline: pull the relevant records, assemble them into documents that make sense on their own, keep them current as the underlying rows change, and respect the permission model while doing it.

That last part is the one that catches teams out. If your product has per-tenant or per-role access control — and any B2B product does — your retrieval layer has to enforce exactly the same rules. A retrieval system that can surface another tenant’s document is not a bug you patch. It is an incident.

Latency budgets are already spent

A page that renders in 300ms has a latency budget your users are accustomed to. A model call takes one to several seconds. You cannot simply insert one into the middle of that request and hope.

The options are all architectural, not model choices. Make it asynchronous and show progress. Stream so something appears immediately. Precompute during a quiet moment. Or accept the delay and design an interface that makes waiting feel intentional.

Deciding this early matters because it determines where the feature can live in the product at all. Retrofitting async behaviour into a synchronous flow is a rewrite.

Failure has to be invisible

A new AI product can fail loudly. Users expect rough edges — that is the deal with something new.

An established product cannot. If the AI feature is down, slow, or wrong, the existing workflow still has to complete. That means every AI call needs a defined fallback, the feature needs to degrade rather than block, and the failure path needs testing as seriously as the happy path.

We build this as a circuit breaker: if the model call fails or exceeds its budget, the product continues without it and the user sees the original interface rather than an error. Boring, and it is the difference between a feature people tolerate and one they turn off.

The existing team has to own it afterwards

This is the constraint most external teams ignore, and it determines whether the feature survives.

If the AI feature is built in an unfamiliar stack, with an evaluation process nobody else understands, it becomes an orphan the moment the engagement ends. Six months later the model version is deprecated, nobody is sure how to test a change, and the safest thing is to leave it alone until it can quietly be removed.

So we build inside the existing stack wherever possible, write the evaluation set so the client’s own engineers can run it in CI, and document the decisions rather than only the code. The feature has to be maintainable by the people who will maintain it.

What actually goes well

Retrofits have one large advantage over greenfield: you already know what users do.

You have logs. You know which searches fail, which support tickets repeat, where people abandon a flow. That is a far better foundation for choosing an AI feature than a workshop, because it is evidence rather than speculation.

The best retrofits we have worked on started with someone looking at their own analytics and noticing that a particular question got asked four hundred times a month and answered badly every time. That is a scoped, valuable, measurable feature. It is also the kind of thing you can only find in a product that already has users.

The order that works

1. Find the friction in your own usage data
2. Define what a correct answer looks like, and build the evaluation set first
3. Work out the latency budget and where the feature can live given it
4. Build the ingestion pipeline, respecting the permission model from the start
5. Add the model call, with a circuit breaker and a fallback
6. Ship to a small cohort, measure against the evaluation set, iterate
7. Hand over something the existing team can maintain

Notice that the model appears at step five. That ordering is not a stylistic preference. It is what keeps the feature alive a year later.