LLM Latency: The First Token, Not the Average

From the field: a green dashboard, an empty bubble

The assistant went live in early January. It sits on the support screen of our mobile app. It takes the customer’s question, looks at their account data and our policy documents, calls a model provider’s API and writes an answer. For the first two weeks nobody complained about speed, because nobody was looking. The dashboard had one number: average response time. I had set the target at 4 seconds. The number was 3.4.

After the screenshot, I opened every request from that week as a distribution. This is what it looked like:

My mistake was simple. I was measuring how long the system took to answer, not how long the customer looked at an empty screen. These are the same thing only if the answer appears all at once — and for us, it did. We had no streaming. Until the model finished a 560-token explanation of commissions, nothing on the screen changed.

The number that really hurt came from somewhere else: 9% of chats were closed before the first answer arrived. When the first answer took longer than 8 seconds, that rate rose to 31%. So the assistant was throwing away its longest, most expensive answers, often before anyone saw them.

Three numbers: TTFT, speed, total

In a classic API, latency is one number: the request goes in, the response comes out. An LLM writes its answer piece by piece, so you need to split the time into three:

1. TTFT (time to first token). The time from the request reaching our server to the model producing its first token. Retrieval, prompt building and the model reading the prompt are all inside it. With streaming, this is how long the user looks at an empty screen.
2. Generation speed (tokens per second). How many tokens the model writes per second after the first one. This is mostly up to the provider and the model. Ours averaged around 80.
3. Total time. Roughly TTFT plus output tokens / generation speed. The second part is often bigger than the first.

Once I wrote this formula down, the 9.6-second p95 explained itself. Long answers had a TTFT of about 2.6 seconds. Writing 560 tokens at 80 tokens per second added another 7 seconds. The model was not slow. The model was talking too much.

t0 = now()                        # request reached our server
first_token_ms = None
output_tokens = 0

for chunk in model.stream(request):
    if first_token_ms is None:
        first_token_ms = now() - t0   # TTFT: time the user stares at an empty screen
    output_tokens += chunk.token_count
    send_to_client(chunk)

total_ms = now() - t0
speed = output_tokens / ((total_ms - first_token_ms) / 1000)   # tokens per second

# a distribution, not an average: write histograms, show p95/p99 on the dashboard
metrics.histogram("llm_ttft_ms", first_token_ms, labels={"flow": "support"})
metrics.histogram("llm_total_ms", total_ms, labels={"flow": "support"})
metrics.histogram("llm_output_tokens", output_tokens, labels={"flow": "support"})
metrics.gauge("llm_token_speed", speed, labels={"model": request.model})

We added a fourth number, and later it became the most important one: first visible word. The mobile client measures it itself, from the moment it sends the question to the moment it shows the first letter. Server-side TTFT does not include the network or the filter I describe below. The number the user sees is the one on the client.

Streaming: saving the first second of the wait

In the first week we did only one thing: we sent the answer to the screen piece by piece. We changed nothing else. The p95 for the first visible word fell from 9.6 seconds to 3.0 seconds. Chats closed before an answer fell from 9% to 4%. Total time stayed the same — the model still wrote answers of the same length in the same time.

On the second day we learned that streaming is not free. We had an output filter on top of the answer. It caught sentences that could read as investment advice, and account numbers that appeared in the answer. The filter waited for the whole answer. With streaming, we had two options: skip the filter (not possible), or wait for the full answer again (then streaming is pointless).

We chose a third way: filter sentence by sentence. As the model writes, we collect text up to a sentence boundary. Each sentence goes through the filter and only then goes to the screen. This adds one sentence of delay to the first word (for us, 250 ms of collecting plus 150 ms of filtering). In return, the answer starts to appear within the first seconds.

Sentence-by-sentence filtering has an honest side effect. If the filter stops the third sentence, the first two are already on the screen. In that case we replace the answer with a safe message, and the customer sees text disappear. In the first month this happened to 0.3% of answers. We accepted it. It is better than skipping the filter, and better than hiding the answer for 10 seconds.

A budget for the first word

After streaming, the first visible word arrived at 3.0 seconds. We set a target of 2 seconds and split those 2 seconds into line items. As soon as we wrote the budget, it was clear that two items used more than twice their share:

ITEM                      BUDGET   BEFORE    AFTER
-------------------------------------------------------
auth + chat history        100 ms    80 ms     80 ms
retrieval (3 sources)      450 ms  1,100 ms   420 ms   <-- sequential -> parallel
prompt building             50 ms    60 ms     40 ms
model first token          800 ms  1,900 ms   800 ms   <-- 6,000 -> 2,200 tokens
first sentence collect     300 ms   250 ms    250 ms
output filter (1 sentence) 200 ms   150 ms    150 ms
-------------------------------------------------------
note: p95 values do not add up; this table is a rough estimate.
      the real measure is the client-side "first visible word" p95

Retrieval used three sources: a policy document search, the account summary service and the customer’s open support tickets. None of them needed the others, but the code called them one after another. When we made the calls parallel, the time dropped to the time of the slowest one: from 1,100 ms to 420 ms. A one-line change that had been in front of us for two weeks.

The model’s first token grew with the size of the prompt. The model has to read the whole prompt before it starts answering, and our prompt was 6,000 tokens. The full list of frequently asked questions was pasted into the system instructions, and on top of that came the last 20 messages of the chat. We removed the list and added only the 3 most relevant items from the search. We cut the history to the last 6 messages. The prompt dropped to 2,200 tokens, and the p95 for the model’s first token went from 1,900 ms to 800 ms.

The real value of the budget was not the lower numbers. It ended the arguments. When someone says “let’s add the last 50 transactions to the account summary for better answers”, the question is no longer “is it a good idea?”. It is “how many milliseconds does it take from the first-token budget, and where do we win them back?”. We still add things, but we measure the cost.

Output length: the knob nobody looks at

After fixing TTFT, the first word arrived at 1.6 seconds, but total time was still long. With long answers, customers watched the text flow for 7 or 8 seconds. I read the answers one by one and saw what the model was doing. It repeated the question in its own words, opened with “great question”, gave the answer, and then summarised the answer as a bullet list. The same information three times.

We made three changes. We added “the first sentence is the answer; do not repeat the question” to the instructions. For topics that need a long explanation, such as a commission breakdown, we asked the model to give a short answer first and then ask “would you like the details?”. And we set a limit of 350 output tokens. The result:

The interesting row is the average. It went from 3.4 to 2.1. Good, but not a revolution. In the same period the first visible word became six times faster, and closed chats fell to less than a quarter. The only number I watched at the start was the one that showed the change least.

Short answers had one side effect. When a customer wants details and types “yes”, a second request goes out, so a long answer now costs a little more in total. But only 18% of customers asked for details. The other 82% were happy with the short answer — nobody had been reading those long paragraphs.

What did not work for me

• Moving to a smaller, faster model. This was my first reflex. TTFT really did drop, but in two weeks the share of chats handed over to a human agent rose from 11% to 16%. A fast wrong answer costs more than a slow right one. We went back and found the speed in the prompt and the output length instead.
• A “typing…” animation. We tried it before streaming. Customers waited 1 or 2 seconds longer, then closed the chat anyway. Three moving dots are not information. The first word is information.
• One timeout for total time. A 15-second timeout cut off long answers that were flowing well, and it made a request that never started wait for 15 seconds. We split it in two: if the first token does not arrive in 5 seconds, cancel and show “we are busy right now, connecting you to an agent”. Once the stream has started, allow up to 30 seconds.

What to track

One rule helped us when we set the targets: engineering does not set them alone. “First word p95 under 2 seconds” is a product decision. We wrote it together with the support team lead, and next to it we wrote who does what when it is broken. I explained this idea in detail in the SLO and error budget post.

Checklist

Conclusion

The customer in that screenshot waited 11 seconds, and we saw no part of those 11 seconds. The dashboard was green because it looked at the system’s average. The customer was looking at their own wait.

None of the fixes were big: streaming, parallel calls, a shorter prompt, shorter answers. The big thing was starting to look at the right number. Once we had it, what was slow became obvious on its own.

The test: how many seconds does your customer wait for the first letter on the screen, and can you say that number today? If you cannot, you are not measuring speed. You are only taking an average.