Where should a prompt live: in the repo or in a database?

In the repo. A prompt looks like text but it decides behaviour, which makes it code. In the repo you get four things for free: version history, code review, rollback, and the eval running in the same pipeline. The only fair reason to keep it in a database is that the person editing it is not a developer and the change has to be fast; even then you need a version number, a change record and a rollback button, and going live must still pass the eval. For us, a prompt edited from an admin screen was a deploy with no record.

Should the model version be pinned?

Yes. 'Latest' means a change nobody approved can go live one morning. The model, the prompt and the tool schema produce behaviour together; if one of the three changes silently, you cannot tell what your eval result means. We write the version explicitly, run the new one in a separate environment against the full eval set, then compare it against real traffic in shadow mode, and only then switch. We also keep the old version running for at least a week, because going back should be one configuration line.

What do you look at when rolling back a prompt change?

A threshold written in advance. We have three numbers: if the tool-choice score in the eval drops even by one point, it does not ship; if the negative feedback rate in production doubles, it is rolled back; if average tokens per run rise by more than 30%, it is rolled back. Because the thresholds are written down, there is no debate; when a number is crossed, the rollback is automatic. Rollback is a configuration change, so it takes 2 minutes and the old version is already running.

Home → Engineering

Prompt and Model Versions: Code, Not Text

“I made a small fix to the prompt,” a colleague said on Thursday at 15:40. He typed into the box on the admin screen and saved; production changed that second. When the answers got strange the next day, we asked: what had changed? Nobody knew. The old text was gone, there was no record of who edited it, and no timestamp. The prompt was in a text box; what it did was a deploy.

Sertaç Yıldırım · 21 September 2026, 21:00 · 12 min read

In short

A prompt looks like text but decides behaviour: it is code. Keep it in the repo and you get history, review, rollback and eval for free.
A prompt change is a deploy. An edit from an admin box is a deploy with no record, which makes it the most dangerous kind.
Pin the model version. “Latest” means a change nobody approved going live one morning.
Behaviour comes from three things together: prompt, model and tool schema. All three sit under one “agent version” and go into every run’s trace.
A shadow run is the cheapest insurance. The new version runs on real traffic, its answers never reach users, and the two versions are compared side by side.
Write the rollback thresholds in advance. Three numbers: eval score, negative feedback rate, tokens per run. When a number is crossed it is an action, not a discussion.

Why is a prompt code?

If changing the body of a function needs a deploy, and changing a sentence that changes the same behaviour does not, the difference is not technical; it is habit. The incident in the eval post proved it: one word, four broken questions.

	Prompt in the repo	Prompt in an admin screen
Who changed it	`git blame`	Maybe one log line
What changed	`git diff`	Nothing, unless the old text was kept
Review	A PR	None
Eval	Required in CI	If someone remembers
Rollback	`revert`, 2 min	As soon as someone recalls the old text
Speed	As fast as the pipeline (6 min for us)	Instant

The last row is the only real argument for the admin screen: speed. Our compromise: the prompt lives in the repo, and the pipeline takes 6 minutes. An urgent fix can wait 6 minutes; if something is too urgent for that, the answer is not a prompt edit but a flag in the flow: turn that step off, then fix it properly.

Editing a prompt from a text box is editing a file on a live server. It works once, and then nobody knows what happened.

What exactly do we version?

The prompt alone does not decide the agent’s behaviour. Three things work together and all three are versioned:

agent-version.yaml

agent_version: 14
  prompt:
    system:  prompts/system_v9.md      # sha 3f21a7
    summary: prompts/summary_v4.md     # sha 88c1de
  model:
    name: qwen3:4b
    tag:  q4_K_M                       # NOT "latest"
    sha:  9b7e2c...
  tool_schema: tools/v6.json           # 9 tools
  flow: flows/reconciliation_v3.py
  thresholds:
    eval_tool_choice_min: 60           # 60/60, must not drop
    feedback_rate_max: 0.08
    tokens_per_run_max: 42000

This file has one job: to answer “what changed between yesterday and today” in one place. Every run writes agent_version: 14 into its trace, and when a complaint arrives, that number is the first thing we look at.

Model version: “latest” is banned

Pinning the model tag is the same instinct as pinning a dependency version: so that a change nobody approved does not arrive one morning. It happened to us once: the local model file was updated, the tag stayed the same, behaviour changed, and we spent two days asking “what did we do to the prompt?” Now the file hash is in the version file too, and the agent warns at startup if it differs.

How does a new version ship?

Four steps, in order. None is skipped:

Step	What happens	Pass condition
1. Eval	Full set, in CI	Tool choice and argument scores must not drop
2. Shadow	Real traffic, answers never reach users	48 hours; the differences have been reviewed
3. Partial	20% of traffic (internal users)	3 days; feedback rate under the threshold
4. Full	100%, old version stays up for a week	Rollback is one configuration line

Why is the shadow run the most valuable step?

The eval set is 60 items; real traffic is 200 questions a day and contains sentences the set never imagined. In shadow mode the new version receives the same questions, its answer is stored somewhere and never shown to the user. Then the two answers are compared. Here is the 48-hour summary of our first shadow run:

Outcome	Count	Note
Same tool, same number	371	No issue
Same tool, different text	29	Reviewed by hand: 26 were better
Different tool	7	5 in favour of the new version, 2 against
Error / timeout	3	The model server restarted

The 2 that went the wrong way became eval items, the prompt was fixed, and the shadow ran again. The shadow caught what the eval could not; the partial rollout catches what the shadow cannot. The layers back each other up.

A shadow run tests on real traffic without ever touching a user. The cheapest insurance, and the step people skip most.

Rollback: write the threshold first

The worst moment to decide on a rollback is the moment you have to: everyone is tired and everyone has a theory. So the thresholds live in the version file, written in advance:

The eval tool-choice score must not drop even by one point (a gate before shipping).
The negative feedback rate doubling triggers a rollback (in production).
Average tokens per run rising more than 30% triggers a rollback — we then find out why and try again.

Rolling back is writing agent_version: 13 and reloading the service: 2 minutes. The old version is already running, so no model is downloaded and no prompt is rebuilt. The rule from failure modes applies here too: the way back must be better tested than the way forward.

From the field: three versions, three lessons

Version	Change	Result
v9	One word in the system prompt	Eval tool choice 58/60 → did not ship, was fixed
v11	“Put the numbers in a table” in the summary prompt	29 text differences in shadow, 26 better → shipped
v12	The model file was updated (same tag)	Went unnoticed, two days lost hunting the prompt → hash check added
v13	Tool schema: 23 tools → 9	Eval 71% → 94%; 3 days at partial rollout, no issues
v14	Brakes added to the flow	Tokens down 18%, feedback unchanged

v12 is the most instructive: we did not make the change, but the behaviour changed. “We did not change anything” means nothing if versions are not written down.

Do this

Keep the prompt in the repo and change it through a PR.
Pin the model tag and the file hash.
Tie prompt, model and tool schema to one version number.
Write that version number into every run’s trace.
Run a 48-hour shadow and read the differences by hand.
Write rollback thresholds in advance and keep the old version up.

Do not do this

Make “a small fix” to the prompt from an admin box.
Leave the model tag on “latest”.
Change the prompt and the tool schema at once and read one result.
Skip the shadow step because “the eval is green”.
Debate the rollback decision during the incident.

What to watch

Eval score per version. Tool choice, arguments and numbers separately.
Shadow difference rate. Percentage of questions that pick a different tool; above 2% deserves attention.
Version age. How old is the version in production? Very old means changes are piling up.
Rollbacks: count and duration. Zero means either you are very good or the thresholds are loose.
Unversioned changes. Any run with an empty agent_version in the trace? There should be none.
Pipeline time. Past 10 minutes, people start looking for an admin box.

Checklist

Before shipping a prompt

Where does the prompt live: repo or database?
Who changed it, what changed, when — are all three recorded?
Is the model tag pinned? Is the file hash checked?
Are prompt, model and tool schema under one version number?
Is the version number written into the run trace?
Is the eval a gate in CI, or a warning?
Is there a shadow run? How long, and who reads the differences?
Which users does the partial rollout start with?
Are the rollback thresholds written down? How many numbers?
How long does a rollback take? Have you tried it?
How long does the old version stay up?

Conclusion

That Thursday’s “small fix” was a deploy: no record, no review, no way back. The next day, while we argued about the answers, we had exactly one gap in our knowledge: we did not know what had changed. In software that is unacceptable; with prompts, somehow, it is treated as normal.

Now the prompt lives in the repo, the model tag is pinned, all three are tied to one version number, and that number is in every run’s trace. A new version passes four gates: eval, shadow, partial, full. Rollback takes two minutes. None of this is a new idea; it is all what we have done with services for years.

The sentence to remember: anything that changes behaviour is versioned. If a prompt changes behaviour, the prompt is versioned too; being text does not make it innocent.