Token Dashboard: An Early Warning, Not an Invoice

From the field: an $11,400 March

We watched the cost of our support assistant in the provider’s console. The console had one chart: total tokens per day. All through February it was a flat line at about $140 a day, and we looked at it once a month. In March nobody looked, because nothing told us to look — or something did, but nobody was listening.

Explaining the difference took two days. The console knew the total. It did not know the reason. We collected the request logs and split them by hand:

None of the four items was a bug. The transaction history feature was released on purpose. Prompt v4 had improved answer quality. The retry was a reliability decision. The mistake was mine: I did not see that each of these decisions had a price, and that nobody could see that price. Real growth was only $400 of the difference.

Why is the invoice always late?

There are three reasons, and all three are part of what an invoice is. Time: it comes once a month, so an increase that starts on 9 March reaches you on 3 April. Unit: it gives you a total, and four different causes mix into the same line. Owner: the invoice goes to finance, not to the team that spends the money. Finance knows the number is big. They cannot know why.

Token cost differs from other infrastructure costs in one way: it changes the moment the code changes. Server cost goes up after a purchase decision. Token cost goes up after one line in a prompt, one tool output or one retry setting. Those changes go through code review, not cost review. So the monitoring has to live next to the code too.

This was not the first time. In the incident from the prompt cache post, a single line made the cost 2.5 times higher, and we saw it four days later, by chance. That day I learned the lesson about the cache, not about cost itself.

Four tags on every request

The first job was to link every LLM call to an owner. “How much did we spend?” is not a useful question. “What did we spend it on?” is. Four tags are enough for that:

• Feature. Which screen or flow: general_question, transaction_history, order_status… A fixed list; we have six values.
• Segment. Retail, premium, corporate. So you can put cost next to revenue.
• Prompt version. With this tag, we would have seen the $1,300 from v4 on the first day.
• Model. The same feature can run on two models, and their prices differ by four times.

In the first week the tags showed us something unexpected. The premium segment made 12% of the requests but 31% of the cost. Premium customers asked longer questions over more turns. That was not a problem. But now we could discuss “should premium get the bigger model?” with numbers.

We write one event record for every call. We do not estimate the token count; we read it from the usage field in the provider’s response. We also calculate the cost when we write the record, because when prices change, old records must keep the old price:

{
  "ts": "2026-04-14T10:32:07+03:00",
  "request_id": "req_8f2c41",
  "feature": "transaction_history", # fixed list, 6 values
  "segment": "retail",            # retail | premium | corporate
  "prompt_version": "v5",
  "model": "large-2026-02",       # versioned name, not an alias
  "input_tokens": 4210,           # from the provider's usage field
  "output_tokens": 286,           # measured, not estimated
  "cost_usd": 0.0161,             # at write time, price table v3
  "attempt": 1,                   # 2, 3 ... for a retry
  "customer_no": "m_104233"       # ONLY in this record. NOT in metrics
}

We added the attempt field later. To find the $700 retry item in March, we had to search the logs for repeated request_id values. How to limit retries is a separate topic, and I covered it in the rate limit post. Here there is only one concern: the cost of a retry must be visible.

Cardinality: a customer number is not a label

Two metrics go from the event record into Prometheus: token count and cost. The labels are the four tags above, plus a direction (input or output) for tokens. Cardinality means how many different values a label can take, and every combination is a separate time series. The maths is simple: 6 features × 3 segments × at most 3 prompt versions at a time × 2 models × 2 directions = 216 series. Prometheus does not even notice.

In the first attempt I added the customer number too. “Let’s also see who is expensive,” I said. About 38,000 active customers used the assistant, and in two days we reached 210,000 series. Prometheus ran out of memory and the dashboard stopped loading. The metrics of other services lived on the same server.

The rule: split a metric only by things with few, fixed values. The customer number, request ID and conversation ID stay in the event record. “Who were the 20 customers who spent the most yesterday?” is a query, not a panel. I explained where we keep these records and for how long in the prompt logging post; the customer number is masked with the same rule.

Two alerts

A dashboard alone is not enough. We had a dashboard in March too; nobody looked at it. Instead of remembering to look at the dashboard, we wanted the dashboard to call us. We set up two alerts:

# monthly budget $8,100 -> daily budget $270

# 1) DAILY BUDGET: how is the day going?
forecast = spent_today + hours_left * hourly_average_of_last_3_hours
IF forecast > 270 * 1.2
    -> message to the team channel (daytime, no paging)

# 2) SPIKE: per feature, hourly
last_hour = tokens(feature, last 1 hour)
baseline  = tokens(feature, SAME hour over last 7 days, average)
IF last_hour > baseline * 3  AND  last_hour > 200000
    -> message to the feature owner
# the second condition removes noise at night:
# 9 requests instead of 3 is not an alert

Two details matter. First, the baseline is the same hour. If you compare Monday 10:00 with Sunday 03:00, the alert fires every morning. Second, the alert goes to the feature owner, not to on-call. A token increase is not an outage. It is not worth waking anyone at night. But it is not worth waiting a month either. Same day, during working hours, to the person who owns the work.

Since April the spike alert has fired twice. The first one was real: in one release the few-shot block had been added twice. We caught it three hours after the deploy, and it cost $8. The same mistake in March would have run until the end of the month. The second one was a campaign day, and traffic really had tripled. We do not count that as a false alert. The alert should have known the campaign calendar, and now it does.

The input/output ratio: an early diagnosis

The cost tells you how much you spend. The ratio of input to output tokens tells you how you spend it. Here is the March data per feature:

60:1 means you make the model read a lot but ask it for very little. The transaction history feature put a raw list of 90 days of transactions into the prompt. But the customer’s question was usually one line, like “how much commission did I pay this month?”. We replaced the list with a 30-day summary, and we took that summary from the database, not from the model. The cost per request fell to about a quarter. The answers did not change.

The ratio can also break in the other direction, and that is a signal too. If the output suddenly gets longer, the model is either repeating itself or an instruction is pushing it into detail nobody needs. With our prices an output token costs four times more than an input token, so a small shift on this side grows fast. Keeping the fixed prefix in a cache lowers input cost without changing the ratio; I covered that separately in the prompt cache post.

The most expensive 1%

The average cost per request is $0.013. That number tells you nothing. When we sorted requests by cost, the most expensive 1% made 19% of the spend. We looked at that 1% one by one and found three types:

• Long conversations. In a conversation that reaches 40 turns, every turn sends the whole history again. Turn 40 cost 30 times more than turn 1. We limited the history to the last 10 turns plus a summary.
• Pasted documents. A customer pastes the full account statement into the chat. We set an input limit. Above it, the assistant asks “which line are you asking about?”.
• Internal users. Someone from the support team had called the assistant 600 times from a script to test it. No bad intent, but the script had no segment. Internal use is now a separate segment.

There is no average on the dashboard any more. Instead it shows p50, p95 and p99 of the cost per request. We do not look at the average for latency (as I wrote in the latency post), so we do not look at it for cost either.

What to track

What did not work for me

• The budget warning in the provider console. It worked per account, so it still told us a total. Saying “80% of the budget is used” on the 20th of the month creates panic without a reason.
• A weekly cost report by email. People opened it for two weeks. Then it went into a filter. A report does not call anyone. An alert does.
• Estimating token counts on our side. Our own tokenizer estimate counted 15–20% too few tokens for Turkish text. When we switched to the number in the provider’s response, the dashboard and the invoice matched for the first time.

Checklist

Conclusion

The $7,500 difference in March did not come from one big mistake. It came from four reasonable decisions, and none of their prices were visible. The April invoice was $7,700. We raised the budget to $8,100, because the transaction history feature and v4 are changes that really help, and it is normal for them to have a price. What was not normal was learning that price from the invoice.

The real change is not the number. It is the time it takes to learn. In March we learned about an increase after 25 days. In April we learned about the same kind of mistake after three hours.

The test: if one line in a prompt doubled your cost tomorrow, who would tell you, and when? If the answer is “finance, at the end of the month”, you do not have a dashboard. You have an invoice.