Hallucination in Production: Field Signals, Not Evals

From the field: Apple and 0.1%

My first move was to check the eval set. It had this question: “What is the commission on foreign share trades?” The assistant answered it correctly on every run: 0.2%. The customer’s question was different: “If I buy Apple, how much will you take?” The model did not connect Apple with foreign shares. It took the first row of the fee table: Istanbul exchange shares, 0.1%.

The customer had bought $4,800 of shares. At 0.1% that is $4.80; at 0.2% it is $9.60. The difference was small, and we refunded it. The question that really scared me was different: how many other people had it told this?

We took 200 random conversations from the 16,800 of the last two weeks. Two people read them for a full day. There was one question: does the answer contain a claim that contradicts the facts, or a claim with no source?

Two things came out of this. First, the 4% failure rate of the eval set and the 5.5% in the field looked close. That was a coincidence: the errors the set caught and the errors in the field almost never overlapped. Second, four more people in the sample of 200 had hit the local/foreign mix-up. All four read the wrong number. All four said nothing.

The process-step errors were more hidden than the numbers. The assistant had told one customer they could cancel a withdrawal request in the app. There is no such button; after approval, you can only cancel by phone. The customer looked for the button, did not find it, and called the support line. That call reached us as a normal “cancel my withdrawal” request, not as a complaint. Nobody linked it to the assistant.

What can an eval not see?

The mistake was mine, and it was simple: I treated a green eval set as proof that the assistant worked. But an eval set measures regression. It tells you that the questions you knew yesterday are still answered correctly today. It says nothing about the questions you do not know.

We wrote the 50 questions in the set ourselves, in our own words: “foreign shares”, “withdrawal request”, “transfer limit”. Customers do not use these words. They say “if I buy Apple”, “when will my money arrive”, “how much can I send to my bank”. The model finds the right row for our question. It does not find it for the customer’s question.

There is also the question of definition. By hallucination I do not mean the model “making things up”. Nothing was made up in the Apple answer. 0.1% was a real number from the table; it was just the wrong row. The definition that works for us is: a confident claim that contradicts its source, or has no source. With this definition, all three types of error are the same thing. The model has the right information or it does not, but the tone of the answer is the same in both cases.

This does not mean you throw the eval set away. The eval stands at the gate, so that a new prompt or model does not break the old questions. But to see what happens after the gate, you need signals from the field.

Field signals: five sources

No metric measures hallucination directly. A wrong answer looks exactly like a right one: fluent, confident, on time. But something happens after a wrong answer. The customer objects, asks again, or wants a human. We collected those traces:

1. Correction phrase. The message after the assistant’s answer starts with “no”, “wrong”, “that is not right” or “but last time”.
2. Repeated question. The customer asks the same question in different words within two minutes.
3. Handoff to a human. “Connect me to an agent” right after the assistant’s answer.
4. Agent flag. One button in the panel of the support agent who takes over the chat: “The assistant gave wrong information”. Below it, one field: “What is correct?”.
5. Number check. Commission, limit and time numbers in the answer are compared with the fee table (see below).

None of these signals proves an error. A customer can say “no” because the answer was correct but they did not like it. A handoff is often unrelated to the assistant. The job of a signal is to tell you where to look. To learn how useful each one was, we read the flagged conversations for four weeks. Precision here means: of the flagged conversations, how many really had a wrong claim?

Read the table against the baseline row. In a random conversation, your chance of finding a wrong claim is 5.5%. In a conversation with a correction phrase it is 31%, so you find errors six times faster. The repeated question is the noisiest; customers often ask again even when the assistant’s answer was right.

The agent flag: cheapest and most precise

The best signal was already in our hands, and we were not using it. The agent who takes over the chat reads the earlier messages and often sees the assistant’s mistake at first glance. They just had nowhere to write it down. In the first week after we added the button, we got 9 flags. When the support team lead made it a step in the handoff process, it went up to 38 by the third week.

The “What is correct?” field turned out to be more valuable than the button. The sentence the agent writes becomes the expected answer for the new eval item. The person who finds the error also writes the fix.

Number check: comparing the claim with the table

6 of the 11 errors were numbers. Numbers have one good property: the right value lives in a table. Commission rates, withdrawal limits and processing times are in the fee table, and it is the same table the app itself uses. So we can check the number with code, without trusting the model:

# extract numeric claims from the answer: "0.2%", "25 TL", "1 business day"
claims = find_numeric_claims(answer)

for claim in claims:
    # topic: commission, transfer_fee, withdrawal_limit, settlement_time ...
    # market: from the question and the customer's portfolio (Apple -> foreign)
    row = fee_table.find(claim.topic, claim.market)

    if row is None:
        flag(answer, "number_without_source")   # nothing in the table
    elif claim.value != row.value:
        flag(answer, "number_mismatch")         # contradicts the table

if answer.flagged:
    # do NOT send the answer. short message to the customer,
    # hand the chat to an agent. the agent sees the flagged answer
    # and the table row in the panel.
    hand_off_to_agent(chat, reason=answer.flags)

The hardest part of the code is the market field. To know that Apple is a foreign share, we match the question against a list of symbols. If there is no match, no table row is found, and the answer is flagged as “number without source”. So uncertainty goes to a human too.

It has a cost: 29% of the flags are false alarms. About 60 customers a week wait for an agent instead of getting an answer that was actually correct. We made that trade on purpose. A wrong block costs an agent two minutes. A wrong number costs a customer money. Claims that are not numbers, like process steps and menus that do not exist, cannot be checked with code. Sampling catches those.

Sampling: 100 conversations a week

Signals help you find errors faster, but they do not measure the rate. If you only look at flagged conversations, you get a full list every week, whatever the real error rate is. So we split the weekly review into two parts:

Two people do the review: the support team lead and one engineer from our team. About 90 minutes each. The same three questions for every conversation:

1. Does the answer contain a claim that contradicts the facts, or has no source?
2. If so, did the customer act on it? (Did they place an order or withdraw money?)
3. Does this kind of question exist in the eval set?

The second question sets the priority. A wrong claim the customer read and ignored is not as serious as one they placed an order on. I described the masking we use when we open conversation logs for review in the prompt logging post; reviewers do not see the customer’s name or account number.

From the field into the eval set

Every error confirmed in the review goes into the eval set the same week, as a new item. There is one rule: the question is written in the customer’s words. It is masked, but not shortened or corrected. “If I buy Apple, how much will you take?” went into the set exactly like that. If we had rewritten it as “foreign share commission”, we would have added a question the set already had.

In four weeks the set grew from 50 items to 74, and all 24 new items came from the field. On the first run with the new items, the score dropped to 63 of 74. It looked like bad news. In fact it was our first honest measurement. After we added the market distinction to the prompt and fixed the symbol matching, it went up to 71 of 74.

What did not work for me

• Thumbs up / thumbs down. People clicked it in 0.8% of conversations. Most thumbs down were not about wrong answers. They were about correct answers people did not like: “withdrawals are not possible at the weekend” is true, but nobody likes it.
• Asking a second model to check every answer. The cost almost doubled. On the Apple question, the second model approved the first model’s answer. It read the same table in the same wrong way.
• Alerting on signal rates. We set a threshold for the repeated-question rate. It fired on every campaign day, because new users ask everything twice. The signals became input for the review list, not alerts.

What to track

Checklist

Conclusion

Zeynep’s call was about a complaint, but its real value was that it was a sample. Behind it were four customers who saw the same error and said nothing, and 9 more errors the eval set had never seen. Four weeks later we read another 200 conversations with the same method: 4 wrong claims instead of 11 (2%). Not zero, and I do not think it will ever be zero.

What changed is less the assistant itself and more what we can see. We used to learn about an error when a customer found it. Now the support agents, the code and the weekly review find it. Before the customer does.

The test: can you learn about a wrong number your assistant gave yesterday, before anyone complains? If the answer is no, you have an eval set, but you do not know what your assistant says in production.