The Model Changed Quietly: Drift, Not a Version

From the field: nine days without a deploy, three symptoms

Our customer support assistant answers about 5,200 questions a week: opening an account, withdrawal times, order types, how leveraged trading works. It does not give investment advice. The prompt says this clearly, and until then the model kept that line well. “What is leverage, how is margin calculated?” is an information question. “Should I buy with leverage now?” is an advice question. The model could tell them apart.

When the Thursday message came, we first checked recent changes. Same code, same prompt, same documents in the context. We spent Thursday and Friday looking for something we had broken. On Monday someone opened the provider’s release notes. On Monday 29 June, the model name we used had been moved to a new snapshot. We were using that name, not the dated version.

Then we measured. Three things had changed in the same week, and we were watching none of them:

Of the 411 weekly refusals, 246 were about leverage and short selling, 118 about IPOs and 47 about other topics. The new model was either worse at separating information from advice, or simply more careful. For us it did not matter which. The result was the same: a customer asks how margin is calculated, gets no answer, and ends up with live support.

Part of this is my mistake. In May we built a token dashboard, and it showed the rise in output tokens: daily output tokens were up 58%. But the dashboard alert was on total cost. Input tokens (prompt and context) are most of our cost, so the total only went up 9%, below the threshold. The signal was on the screen. I was looking at it as a bill, not as behaviour.

A model name is not a version

Most providers give two kinds of names. One is an alias: a name that means “the latest model”, and what it points to changes over time. The other is a pinned snapshot: a model with a date in its name, whose behaviour does not change. The alias is handy in a prototype because you always get the newest model. In production the same convenience is a risk. It is like shipping a library with the latest tag.

For us the model name was one line of config, and nobody saw that line as a dependency. After the incident it became this:

# BEFORE - alias: the name stays, the model behind it can change
model: general-model

# AFTER - pinned snapshot: the date is part of the name, behaviour is fixed
model: general-model-2026-03-12
retirement_date: 2026-10-15    # date announced by the provider
next_candidate: general-model-2026-06-29
owner: one name                # the person who tracks the upgrade
# rule: a PR that changes this line must include eval + shadow run results

On Monday 6 July, in the afternoon, we pinned the old snapshot. The next day the refusal rate was back at 1.5%. The fix itself was one line. Finding it took seven days. The gap between those two is what this post is about.

Pinning is not the end. It is a calendar.

A snapshot does not live forever. The provider retires old snapshots on a set date; ours goes in mid-October. So pinning does not remove the problem. It lets you choose the time. The retirement date is in the config, a reminder fires one month before, and the upgrade is done like a deploy: the eval set runs on the new version, a shadow run replays last week’s real questions, someone reads the differences, then we switch.

We are in that process for the new snapshot right now. In the first eval run, two extra sentences in the prompt brought the refusal rate down to 2.1%. That is not the old level yet, so we have not switched.

The second drift: users change too

While we were looking at the model incident, a second thing came up, and I think it matters more. Of the refused questions, 118 were about IPOs. We looked at the share of each question category. In January, IPO questions were 3% of traffic. In June they were 19%. There were three IPOs in a row in June, and customers were asking about the subscription period, share allocation and refund times.

Our eval set was built in January, from January traffic: 120 questions, four of them about IPOs. So we were almost not testing one of the most common topics at all. This gap was there even without the model change. The model change only made it visible.

I call this input drift: same model, same prompt, but different questions. An eval set is a photo. Traffic is a film. As the photo gets older, the eval stays green and users get unhappy. In the hallucination post I wrote about putting signals from the field on top of the eval. Input drift is another side of those signals.

We now have two rules. First, we track the weekly share of each question category. If a category goes above three times its share in the eval set, we get a warning. Second, every month we add 20 questions from last month’s traffic to the eval set and move the 20 oldest ones to an archive.

Watch the distribution, not correctness

You cannot check every day whether each answer is correct. That needs a human or an expensive evaluation. But you can measure the shape of the answers every day, almost for free. If the shape changes, something has changed. We chose four signals:

• Answer length (p50 and p95). Often the first sign of a model update. Going from 92 to 148 words is not a bug by itself, but it tells you something happened.
• Refusal rate. A simple classifier for phrases like “I cannot help” or “I cannot give advice”. This is where the real damage was in our incident.
• Format errors. At the end of each answer, the assistant leaves a small JSON for the ticket classifier. The share of JSON that fails to parse is the cheapest measure of how well the model follows instructions.
• Language. The share of English or mixed answers to Turkish questions. This time it barely moved (0.1% → 0.4%), but in an earlier update it was the first thing to break, so it stays on the list.

Each one is compared with the average of the last 14 days. We kept the thresholds wide on purpose. The goal is not an alert for every small move. It is to catch a shift that goes in one direction for three days:

# every night at 00:30, over yesterday's answers
baseline  = last_14_days_average(exclude=yesterday)
yesterday = measure(date=yesterday)

checks = {
  "length_p50"  : abs(yesterday.length_p50 / baseline.length_p50 - 1) > 0.25,
  "refusals"    : yesterday.refusal_rate > max(2 * baseline.refusal_rate, baseline.refusal_rate + 0.02),
  "format_error": yesterday.format_error > 0.01,
  "language"    : yesterday.english_rate > 0.01,
}

for name, breached in checks.items():
    if breached:
        warn(name, yesterday, baseline, model=active_model_version())  # put the version in the message

Putting the active model version in the warning looks like a small detail. But it is exactly why we lost two days: the answer to “which model is answering?” was in no log. Now the record of every answer includes the model version that the provider returns.

Canary questions: the same 40 every morning

Distribution signals depend on traffic. At the weekend there are fewer questions and more noise. So we also wanted a measure that does not depend on traffic. Every morning at 07:00, a fixed set of 40 questions runs with the production setup: same model, same prompt, same context. We do not expect the same words every day. We check four things:

- id: leverage-margin-01
  question: "How is margin calculated in leveraged trading?"
  expect:
    must_not_refuse: true           # information question, not advice
    must_contain: ["margin", "ratio"] # key concepts
    length_words: [40, 160]         # a band, not an exact number
    format_valid: true              # the JSON at the end must parse

- id: advice-refusal-01
  question: "Should I buy with leverage right now?"
  expect:
    must_refuse: true               # test the OTHER side of the line too

12 of the 40 questions are ones that should be refused. This matters. If you only check “does it answer?”, you will think an update that loosens the line is good news. You need to test both sides of the line.

We tried the canary on the past: we ran the same 40 questions against the old snapshot and the new version from 29 June. The new version refused 7 information questions (the old one: 0), broke the format in 3 answers (the old one: 0), and p50 length went up 55%. So if this check had existed then, we would have seen the change at 07:00 the next morning. That is three days before the support team’s message, and seven days before we found the root cause.

The report goes every morning to that week’s owner of the assistant. It is a five-minute look. We call it the “daily watch”, but it never wakes anyone up. Drift is not an outage; it is a slow shift. Noticing it in the morning is enough, as long as someone notices it every morning.

What did not work for me

• Comparing answers with stored answers by semantic similarity. The idea was nice: if the embedding of today’s answer moves away from yesterday’s, send a warning. In practice the same model gives a slightly different answer to the same question every day. Every morning 5–6 of the 40 questions came out as “different”. Within two weeks nobody read the report. Bands and simple rules are less clever, but people read them.
• Tight thresholds on every signal. In the first week the length threshold was 10%. It fired every day at the weekend. Alert fatigue arrived faster than drift.
• Saying “the provider will announce it”. They did announce it. In the release notes, in one line dated 29 June. A change announced in a place nobody reads was, for us, a change nobody announced.

What to track

Checklist

Conclusion

That Thursday we spent two days looking for a bug in our own code, because we believed that if we had not changed anything, nothing had changed. But part of the system lived on someone else’s calendar, and we were not looking at that calendar.

Pinning the model was one line. The real work is watching what you cannot pin: the questions users ask, the shape of the model’s answers, and 40 answers that should stay the same every morning. You cannot prevent drift. You can only choose how late you hear about it.

The test is simple: if your provider changed the model tonight, who would notice before you? If the answer is “a customer”, you do not have monitoring. You have hope.