How do you collect an LLM eval set?

By collecting, not by inventing. Our 60-item set came from four sources: questions users actually asked, runs a user marked as wrong, edge cases (empty result, a range that is too long, an ambiguous question) and a 12-item prompt injection set. Each item records the question, the expected tool, the expected arguments and, where it applies, the expected number. The set starts small; every real failure adds a line. Ours went from 12 to 60 in three months, and that growth was a process, not an accident.

Can you trust an LLM as a judge?

Only within limits. We measured three biases: it prefers longer answers, it likes text produced by models from its own family, and it is sensitive to the order in which two answers are shown. To reduce them we use a written rubric, run comparisons in both directions, and every month compare 20 samples against human scoring. The main rule is different: measure exactly whatever can be measured exactly. Tool choice, arguments and numeric answers are checked by code; the judge is left only for text quality.

What if the eval is green but users complain?

Then the set does not represent reality. Do two things. First, sample production: every week, review 10 random real runs by hand and add whatever the set does not cover. Second, tie the feedback button to the run; every click of 'this answer is wrong' is an eval candidate. For us, more than half of the complaints were not about the model at all: missing data, wrong tool arguments and user expectations. An eval set exists to measure the system, not to clear the model.

Home → Engineering

Eval: A Regression Test, Not a Demo

I changed one word in the system prompt: “give a short answer” became “give a concise answer”. I asked three questions in the demo and all three looked good. Two days later, support wrote to me: volume questions were coming back wrong. I checked: 4 of 27 questions now went to the wrong tool. The only change was that word. A user caught it; I should have.

Sertaç Yıldırım · 21 September 2026, 14:00 · 13 min read

In short

A demo is one example; an eval is a set. Saying “looks good” after three questions is like writing three test lines and calling the app tested.
The set is collected, not invented. Real questions, runs users marked wrong, edge cases, injection texts. Ours grew from 12 to 60 in three months.
Measure exactly what can be measured exactly. Tool choice, arguments and numeric answers are checked by code. 80% of our set works this way; the judge is only for text.
An LLM judge has three biases: it prefers long, it prefers its own family, and it is order-sensitive. A written rubric, two-way comparison and monthly human calibration are required.
A one-word prompt change is a regression. Prompt, model and tool changes all go through the same gate: versioning and eval.
“Green eval, unhappy users” is the set’s fault. Review 10 real runs a week by hand; anything not covered goes into the set.

What are we measuring?

An agent’s answer is not one thing; it is the end of a chain. Each link is measured separately, and they are not equally uncertain:

Link	How it is measured	Exact?
Was the right tool chosen?	Compare with the expected tool name	Exact
Are the arguments right?	Compare date range, id, limit field by field	Exact
Is the number right?	Compare with the expected value (tolerance 0.01)	Exact
Is the required warning present?	Keyword or pattern check	Semi-exact
Is the text clear?	A judge with a rubric, or a human	Uncertain

The rule that follows: shrink the uncertain part, grow the exact part. Our first set asked “is the answer good?” for every item and sent all of them to a judge. Today 48 of 60 items are checked by code, and the judge is left for summary text.

One eval item

- id: volume-01
  question: "how many lots of gold were traded today?"
  expected_tool: volume
  expected_arguments:
    instrument: XAUUSD
    start: TODAY
    end: TODAY
  expected_number: 1284.5      # tolerance 0.01
  required_words: ["lot"]
  forbidden_words: ["roughly", "approximately"]  # no guessing in a numeric answer

Where does the set come from?

Invented questions are easy and useless; real questions are hard and valuable. We used four sources:

Real questions. 27 questions from the first month of chat history; the core of the set.
Failed runs. Every real failure adds a line. Replay helps here: the failed run is already recorded.
Edge cases. Empty result, a 92-day range, an ambiguous question (“how much was withdrawn yesterday” — requested or approved?), a customer that does not exist.
Injection texts. The 12-item set; what it measures is not whether the model is fooled but whether being fooled has a consequence.

Month	Items	Why they were added
Month 1	12	First real questions
Month 2	31	+7 failures, +8 edge cases, +4 injection
Month 3	60	+12 user feedback, +9 new tools, +8 injection

An eval set is not written, it accumulates. Every real failure is a line; a failure you do not add will come back.

From the field: one word, four failures

The incident from the opening happened again after the set existed — but the ending was different. I made the same kind of prompt edit and ran the set before committing:

Eval output

$ agent eval --prompt v9 --model qwen3:4b
  60 items, 2 min 41 s

  tool choice   : 58/60   (previous version: 60/60)   -2  ✗
  arguments     : 59/60   (previous version: 60/60)   -1  ✗
  numeric       : 22/22
  injection     : 12/12
  text (judge)  : 11/12   (previous version: 10/12)   +1

  BROKEN:
    volume-01  expected=volume    got=transactions   (arg: instrument missing)
    volume-04  expected=volume    got=transactions
    kyc-03     argument: start    expected=MONTH_START got=TODAY

Three minutes and I could see it: text quality up one point, tool choice down two. The new word in the prompt had pushed the idea of “transactions” forward and pulled volume questions towards it. I reverted the change, moved the word elsewhere, ran again: 60/60.

The win here is not “the model is good” or “the prompt was bad”. It is seeing the price of a change. If text improves by one point and tool choice drops by two, that is a trade; the difference between making that trade knowingly and making it by accident is what this whole post is about.

The LLM judge: when and how

You cannot measure text quality with code. There are two options: humans (expensive, slow) or a judge (cheap, biased). We mixed the two, but first we had to understand what the judge was actually measuring.

Judge biases

It prefers long. Given two answers with the same content, it picks the longer one.
It prefers itself. It scores text from its own model family higher.
It is order-sensitive. “A or B” and “B or A” can give different answers.
It likes confidence: a correct “I do not know” scores lower than a confident wrong answer.

How to harden it

A written rubric: 4 items, each 0/1. No “general impression”.
Compare in both directions; if they disagree, send it to a human.
Every month, compare 20 samples with human scores; fix the rubric if the gap passes 15%.
Use the judge for text only; never ask it about a number.

The judge’s score is a warning for us, not a gate: if it drops, the commit is not blocked, but it shows up in the PR. The gates are the exact checks: tool choice, arguments, numbers, injection.

When does it run?

Change	What runs	Time
Prompt edit	Full set (60)	~3 min
New or changed tool	Full set + that tool’s items twice	~4 min
Model version	Full set + judge + 20 human-checked samples	Half a day
Flow (orchestrator) change	Full set + 5 end-to-end runs	~10 min
Nightly	Full set + 10 production samples	Automatic

A three-minute gate builds the discipline on its own: nobody says “it is one word, not worth running”, because the opening paragraph of this post is a story the team knows.

“Green eval, unhappy users”

We heard this twice and both times they were right. The reason was the same: the set did not represent reality.

First time: every question in the set was one sentence. Users were asking three questions at once: “how much was approved yesterday, how many are still pending, and what is their total?” We added 9 multi-part questions; 4 were red on the first run.
Second time: the set always ran against full data. A user asked at the weekend, the data was empty, and the agent gave something unreadable instead of “0 lots”. Six empty-result items joined the set.

The lasting fix was sampling: every week, 10 random production runs are reviewed by hand (one click from the trace). Anything the set does not cover goes in the same day. That 20-minute job is the only thing keeping the set connected to reality.

An eval set exists to measure the system, not to clear the model. A green set can be proof that you are asking the wrong questions.

What to watch

Set size and coverage. How many items, which tools? Is there a tool with no items at all?
Score per version. Tool choice / arguments / numbers / injection separately; a single “success percentage” tells you nothing.
Judge – human gap. 20 samples a month; over 15% means fixing the rubric.
Age of the set. How many items were added in the last 30 days? Zero means the set is dying.
Complaints the eval missed. How many complaints were already in the set? High means the thresholds are wrong; low means the set is too narrow.
Eval time and cost. Past three minutes nobody runs it, and a test nobody runs does not exist.

Checklist

Before changing the prompt

Do I have an eval set? How many items?
Where did the items come from: invented, or real runs?
How many are checked exactly by code, and how many go to a judge?
Does each item record the expected tool and arguments?
Are there edge cases: empty result, oversized range, ambiguous question?
Are the injection items in the set?
Is the judge’s rubric written down? Are comparisons two-way?
How often is the human calibration? What was the last gap?
Which changes trigger the set? Is it a gate or a warning?
Is production sampled? How many runs a week are reviewed?
How many items were added in the last 30 days?

Conclusion

I changed one word and broke four questions. The demo did not show it, because the demo was three questions and none of them was among those four. A user found the failure; in between there were two days and an unknown number of wrong reports.

Today the same change is measured in three minutes and I see the result as a trade: text one point better, tool choice two points worse. I make the call. That is all an eval does — it makes the decision visible. It does not improve the model or write the prompt; it tells you the price of a change.

The sentence to remember: if the prompt is code, the eval is its test. If you would not ship code without tests, do not ship a prompt without an eval.