Agent Cost: Cost per Task, Not per Token

What is the right unit?

“So much per million tokens” is a price list, not a cost. Cost comes from how many times you have to send those tokens.

cost_per_task = (cost of all attempts) / (tasks completed)

# cloud: cost = input_tokens * input_price + output_tokens * output_price
# own server: cost = gpu_seconds * (hourly amortisation + power) / 3600

# example (our nightly report, before):
#   3 turns * ~39k tokens, 1 failed attempt, 96 s GPU
#   -> 96 s GPU per task (128 s including the failed attempt)

Putting failed attempts in the numerator matters: if a task finishes on the second attempt, its cost is both attempts. Until you separate that, you cannot see the “cheap model that keeps retrying” situation.

Where does it go?

The breakdown of that first 118k-token measurement:

The first row tells the whole story: 60% of the money went to re-sending data that had already been sent. Fixing that was not a model decision but a tool decision — trim the reply, do not send raw lists.

The fixed prefix and caching

For the second item there is a simple rule: put what does not change at the front. If the system prompt, the tool definitions and fixed instructions always sit at the start in the same order, both a cloud provider’s cache and a local model’s KV cache do their job. We had originally put a timestamp on the second line of the system prompt — one line that changed on every request, invalidating the cache for everything behind it. Moving the date to the end saved 9 seconds per run. The change: moving one line.

Cheap model with 3 attempts, or expensive with 1?

In the first post we did this arithmetic simply: the small model answers in 0.5 seconds but gets 17 of 20 right; the big one takes 11 seconds and gets 20/20. Here is the same comparison in cost per task:

The table does not give one answer; it gives an answer per task type. On simple questions the small model is 18 times cheaper; on multi-step analysis the large model is cheaper, because the small one tries two and a half times and still fails a third of the time.

So the decision was not a model choice but a routing decision: the question type is picked from a closed set, simple goes to the small model, multi-step to the large one. After routing, cost per task fell 34% and accuracy went up; hard work is no longer attempted three times on the wrong model.

From the field: three weeks, three fixes

The last row matters: it is the one place we increased cost deliberately. For a job that runs once a month and has to be ready for a meeting, 2.8× the tokens is an acceptable trade. In a flow that runs 400 times a day the answer would have been no. A cost decision is made together with how often the work runs.

Budgets and the kill switch

Measuring cost is not enough; it also has to stop somewhere. Three limits, all in the flow:

RUN_TOKEN_BUDGET  = 120000     # one run
USER_DAILY        = 2000000    # one user, one day
SYSTEM_DAILY      = 60000000   # the whole system

# when crossed:
#   the run stops, the partial result is saved, the user is told why
#   at the system limit: no new runs, they queue
#   night jobs keep priority in the queue

We hit the system limit once: a user had written a script that triggered the same analysis 30 times. Without the limit, the nightly reports would have queued behind it. The value of the kill switch was not protecting money but protecting the other work.

And the value side?

Cost alone means nothing; you have to look at what it buys. For the nightly reconciliation report:

• By hand: one person, 12–15 minutes each morning, five days a week.
• With the agent: 19 seconds of GPU plus 2 minutes of checking.
• Gain: roughly an hour of human time a week, against a maintenance burden and the measurements in this post.

To be honest: automating it (the flow, the evals, the tracing, the brakes) took a few weeks of work. For one report it would not pay off. It pays off because the same skeleton now serves three more jobs. An agent’s cost is per task; its investment is per skeleton.

What to watch

• Cost per completed task. Per task type; a single average misleads.
• Re-sent token ratio. Total ÷ unique. Above 3 means trim.
• Cache hit rate. Low means the fixed prefix is broken; the first suspect is a variable line at the top of the prompt.
• Attempts per task. A task type above 1.2 is on the wrong model.
• Budget triggers. Which limit, how many times, which user?
• Idle GPU / queue time. On your own server this shows the real bottleneck.

Checklist

Conclusion

The day I said “tokens are free”, I was really saying I had not measured. Once I did, the thing I saw was not the model: most of the money went into sending the same data over and over. Fixing it took trimming, repeat protection and moving one line in the prompt; cost per task went from 96 seconds to 19.

Model choice also turned out not to be one decision but a match: simple report questions to the small model, multi-step analysis to the large one. In exactly one place we increased cost on purpose, and we did that by looking at how often the job runs.

The sentence to remember: the expensive part is not the model, it is the repetition. Anyone who wants a smaller bill should look first at how many times they send something, not at which model sends it.