Prompt Cache: A Design Decision, Not a Discount

From the field: the cost of one line

The product team’s request was reasonable: the support assistant should call the customer by name. The engineer who took the task chose the most natural place. He put the customer’s name and account type on the first line of the system prompt. His reason was reasonable too: “The model should see the most important information first.” I approved the PR. It went live on Friday at 17:00.

Traffic is low at the weekend, and nobody looked. Monday was a normal day. On Tuesday morning, while preparing the monthly report, I opened the usage page: Monday $450, the Monday before $180. The answers were correct, the error rate was zero, and our latency alert had not fired. Everything worked. It just cost 2.5 times as much.

Finding the cause took an hour. Fixing it took one line. The number of tokens read from the cache had fallen off a cliff at 17:00 on Friday. We moved the customer name to the end of the prompt. The hit rate went from 4% to 61% — but not back to the old 86%. A second cause was hiding in the same release. I will come to it below.

How it works: the fixed prefix

When a model reads a prompt, it computes intermediate results for every token. Prompt caching keeps those results for a short time. If the next request arrives with the same beginning, that part is not computed again. You get two benefits: those tokens are billed at a lower price, and the first token of the answer arrives sooner.

The key word is “beginning”. The match starts at the first token of the prompt and ends at the first different token. If one word in the middle changes, everything after it is processed again, however similar the rest is. The cache is not a dictionary. It is a prefix.

The numbers differ between providers, so let me describe our own case. With our provider, an input token read from the cache costs one tenth of the normal price. The cache is removed if it is not used for a few minutes, and very short prefixes are not cached at all. Some providers also charge extra for writing to the cache. Check your own provider’s documentation; the logic is the same everywhere.

# BEFORE (Friday release) - first line differs for every customer
[1] You are talking to our customer Ayse Kaya. Account: premium.   <-- variable
[2] tool definitions         ~1,100 tokens  # all processed again
[3] instructions             ~  900 tokens  # all processed again
[4] policy summaries         ~1,200 tokens  # all processed again
[5] conversation + question  ~  600 tokens

# AFTER - fixed parts first, changing parts last
[1] tool definitions         ~1,100 tokens  # from cache
[2] instructions             ~  900 tokens  # from cache
[3] policy summaries         ~1,200 tokens  # from cache
    ---- identical on every request up to here ----
[4] customer: Ayse Kaya, premium            # variable, at the end
[5] conversation + question  ~  600 tokens

For the model there is no difference between the two. It still sees the customer name at the end, and it still uses it. We tested this first on 40 conversations and found no difference in the answers. The difference exists only for the cache, and it is 3,200 tokens.

What breaks it: hidden variables

The customer name was the obvious one. The dangerous ones are the variables you add to the prompt without knowing it:

The second cause: the order of the tool list

After we moved the customer name to the end, the hit rate got stuck at 61%. The same Friday release had added a new tool, and the tool definitions were now built from a set. In Python, string hashes use a different random seed in every process. So the same set came back in a different order on every pod.

8 pods and 4 account types: 32 different prefixes. At busy hours each one kept its own cache warm. At quiet hours a prefix could go unused for a few minutes and get removed. We sorted the tool list by name, and the next day the hit rate was 87%. The change: one sorted().

Cost and the first token

Put the four periods side by side and you see the whole effect. Cost per question is shown as an index, with the level before the incident set to 100:

The arithmetic explains why the cost gap is so large. For us, one question uses about 3,800 input tokens and 250 output tokens on average, and an output token costs four times as much as an input token. When 86% of the input comes from the cache, the input bill stays smaller than the output bill. When the cache breaks, input becomes the biggest item on the bill.

Time to first token grows for the same reason: the model reads 3,200 tokens from scratch every time. The user feels it while waiting for the answer to start (in the latency post I explained why the first token matters more than the average). The speed of writing each token does not change; only the start is late. Our latency alert was set at 3 seconds, and 2.3 seconds did not trigger it.

Why it took us four days to notice

After the incident, my question was not “why did it break?” It was “why did it take four days?” The answer was uncomfortable: we had no check that could catch this change. There were three safety nets, and it passed through all three.

• Tests looked at behaviour. The test set asked “does the assistant give the right answer?” It did. No test asked “how much did this answer cost?”
• Alerts looked at errors. The error rate was zero. The latency alert was at 3 seconds. There was no cost alert, because we saw cost as a bill at the end of the month, not as a metric.
• Code review looked at the content. In the PR I checked whether the sentence was right. I did not think about where it was placed. I reviewed the prompt like a piece of text, not like a data structure.

All three have one thing in common: we looked at what the prompt said, not at how it was built. The cache only cares about the second. The design rules and the tracking list below exist to close that gap.

The design decision: build the prompt in layers

After the incident we stopped thinking of the prompt as one text and started thinking of it as layers. Each layer changes at a different rate, and the order follows that rate:

1. Changes with a release: tool definitions, instructions, policy summaries. They only change on deploy.
2. Changes with the customer: name, account type, a summary of open orders. Fixed during a conversation.
3. Changes with every message: conversation history and the new question.

This order brought one more benefit: inside a conversation, the second layer also comes from the cache. By the customer’s third message, the name, the account type and the first two messages are already cached. That only works if you only append to the history. Trimming or summarising old messages makes everything after that point processed again. Long conversations still need trimming, but not on every message: once, at a fixed threshold.

A/B experiments need extra care. When you test a change in the instructions, the fixed prefix splits in two. The experiment arm that gets 5% of the traffic loses its cache often, so its cost goes up for a reason that has nothing to do with the change you are testing. Keep this in mind when you read the result, or you will wrongly conclude that “the new instruction is expensive”.

How cached tokens count against your quota also depends on the provider. When you do the TPM calculation from the rate limit post, check this in your provider’s documentation. Do not guess.

What to track

The alert: if the hit rate is 20 points below the same hour last week, send a notification. With that alert in place on Friday at 17:00, we would have seen the gap after twenty minutes, not four days. We also added one line to the post-deploy checklist: “Did the fixed-prefix hash change? If so, did we expect it?”

What did not work for me

• Pinging to keep the cache warm. The hit rate dropped at night, so we sent a fake request every four minutes. The gain was about $3 a day. The fake requests, their logs and the extra complexity were not worth it. We turned it off.
• Removing personalisation completely. The first reaction was “let us roll back the name”. The name was not the problem; its position was. The feature stayed, only its place changed.
• Tracking one overall hit rate. Our first dashboard put every feature into one number. Most traffic was on the support assistant, so a collapse in the internal tools barely showed in that number.

Checklist

Conclusion

The change we made that Friday evening was not wrong. Calling the customer by name was a good request, and the model did it well. What was wrong was that nobody thought about the position of one line — including me, the one who approved the PR.

We thought of prompt caching as a discount coupon: turn it on and the bill goes down. It is not. It is a decision about how the prompt is built, and it has to be made again with every new line.

The test is simple: if someone adds a line to the system prompt tomorrow, who decides where it goes? If the answer is “nobody”, you have a cache, but you do not have a design.