What happens when the model changes under you

A feature we shipped in the spring passed every check in its evaluation set the week it launched, and was producing noticeably weaker output four months later without a single line of our code having changed. Nothing had broken in the way software usually breaks. The model underneath it had been updated, and behaviour we had implicitly relied on had shifted just enough to matter to the people using it every day.

This is the failure mode that surprises clients most, and it's the one that most cleanly separates a demo from a system somebody can actually run for years. Traditional software fails when you change it. AI features can fail when you don't, because a dependency you don't control moved underneath a system you thought was finished.

The first thing we do on any production AI system is pin the exact model version rather than pointing at a moving alias that silently tracks the provider's latest release. That single decision removes the worst version of this problem, where a system's behaviour changes overnight with no announcement, no changelog anyone on the project read, and no way to correlate a drop in output quality with anything the team actually did.

What pinning doesn't do is make the problem go away. Pinned versions get retired on a schedule the provider sets, which means pinning converts a gradual, invisible drift problem into a hard deadline problem. That's a genuinely better problem to have, since a deadline can be planned around and drift can't, but it's worth being honest with clients that pinning is a way of controlling when you deal with a model change, not a way of avoiding one.

We build an evaluation set before shipping any AI feature, and the part that matters for this problem is that we keep running it long after launch rather than treating it as a pre-launch gate that gets archived once the feature goes live. A set that runs once, passes, and is never run again tells you the system worked on the day it shipped, which is the least useful moment to know about.

Running that same set on a schedule, and again whenever a provider announces anything about the model in use, is what turns "the output feels a bit worse lately" into a specific, measurable claim somebody can act on. The difference matters enormously in the conversation with the client: a vague sense that quality has slipped invites a debate about whether anything has really changed, while a standing evaluation showing a measurable drop against a fixed baseline moves the discussion straight to what to do about it.

It also protects against the opposite error, which we've hit more than once. A client reports that the system feels worse, the standing evaluation comes back unchanged, and the real cause turns out to be a shift in the kind of input the system is receiving rather than anything about the model at all. Without a baseline to check against, that investigation starts by assuming the model is at fault and can burn a week before anyone questions the assumption.

Model deprecation windows are announced with a fixed end date, and that date has no relationship to a client's budget cycle, release freeze, or the fact that their team is mid-way through something else entirely. We've had a retirement date land squarely in the middle of a client's busiest quarter, when the last thing anyone wanted to hear was that a working feature needed engineering attention.

The practical consequence is that migration work has to be treated as scheduled maintenance with a known trigger rather than an unplanned interruption. We track the announced support window for every model in production across our client work, and we start the migration conversation early in that window rather than near the end of it, because the difference between a migration planned with two months of room and one done under deadline pressure is mostly the difference between a routine piece of work and a stressful one.

Telling a client that a feature which is working needs unbudgeted engineering work is genuinely awkward, and it's much harder if it's the first time the topic has come up. Framed cold, it sounds like a defect being billed as maintenance, and a client who wasn't warned has every reason to hear it that way.

We now put model migration in the original proposal as a named line item with an expected cadence, alongside hosting and monitoring, so the first mention of it happens before anyone has signed anything rather than a year into a working relationship. It's a small change to how a proposal reads and it changes the later conversation completely, because at that point we're executing something the client already agreed to rather than asking for something new.

The framing that lands best, in our experience, is the honest comparison: an AI feature has an ongoing dependency on infrastructure somebody else controls and updates, which is closer to how a business already thinks about a payment provider or a mapping API than how it thinks about custom code. Clients don't usually find that surprising once it's put in those terms. They find it surprising when it's presented for the first time as an unexpected invoice.

Four things, on every project, from the first week rather than added later. The exact pinned model version is recorded in the repository alongside the reason it was chosen, so whoever picks the project up in a year knows what was deliberate and what was default. The evaluation set is wired to run on a schedule rather than on demand, since anything that requires somebody to remember to run it eventually stops being run. The announced support window for the model in use is tracked somewhere a person will actually see it, rather than living in the memory of whoever set the system up. And migration is a named line item in the original proposal with an expected cadence attached.

None of this came from a best-practices article. All four came from watching a system we'd built quietly degrade over several months while everyone involved, us included, assumed it was still doing what it did on launch day. That's an uncomfortable thing to discover late, and the whole point of the four items above is that discovering it late is a choice rather than an inevitability.