Context: The Right Summary, Not a Longer Window

What is in the window?

The agent loop sends a stack to the model on every turn. This is what piles up in it:

Watch the fourth row. When a tool returns 2,300 rows, those 2,300 rows are sent again on every turn until the end of the run. In an 8-turn run you pay for the same data 8 times, and 7 of those times it does nothing.

Why a longer window is not enough

“The window is 200k tokens, 2,300 rows fit” is a true sentence that leads to a wrong conclusion. Fitting is not the same as working. Three things I measured:

1. Accuracy falls as the window fills

I repeated the labelling job from the previous post — 34 withdrawals — while deliberately filling the window. Same model, same prompt, same data; the only variable is how full the window is:

Up to 70% there is no serious loss; after that the drop is sharp. When I looked at the records it got wrong, most of them depended on data sitting in the middle of the stack. The instruction at the start and the question at the end stay strong; what is in between drowns.

2. Cost snowballs with turns

The window is re-sent every turn. A 10k-token stack in an 8-turn run is 80k tokens. If the stack also grows across turns, the total rises close to the square of the turn count. In that night job, about 71k of the 118k tokens was data being sent again after it had already been sent.

3. Latency loses the user

The last column of the table: a four-fold gap between 9 and 34 seconds, and the only difference is a full window. In a chat window, 34 seconds means the user switches tabs.

Where to cut: three places, in this order

1. Tool output: the biggest win

Never sending raw data to the model is always cheaper than sending it and summarising later. Our list_transactions tool used to return every row. It does not any more:

# BEFORE: 2,300 rows, ~46,000 tokens
return {"transactions": [row for row in query()]}

# AFTER: summary + small sample + continuation key, ~380 tokens
return {
  "count": 2300,
  "total_amount": 1841250.40,
  "currency": "USD",
  "sample": first_n(query(), 5),      # 5 rows, just to show the shape
  "unmatched": unmatched(),           # 13 rows: what the job is about
  "next": "page=2"                    # the model can ask for detail
}

The rule: a tool returns what the model needs in order to decide; it is not a data store. If detail is needed, there is a second call. This one change took the night job from 118k to 31k tokens per run. (The tool contract itself is a post of its own.)

2. Run state: on disk, not in the window

If the agent’s “where was I” lives in the window, it disappears when the window is cleared. We keep run state in a separate record:

{
  "run_id": "rec-2026-09-19-01",
  "step": 3,
  "decisions": ["date range: 18.09 00:00-23:59",
                "tolerance: 0.01 USD"],
  "open_questions": ["WD-24817 appears in two records"],
  "completed": ["file_read", "matched"],
  "identifiers": ["WD-24817", "WD-24902", "CUST-7741"]
}

What enters each turn is not this record but a short translation of it: 6–8 lines. If the run stops halfway, it continues from here; even if the window is reset, the work is not lost. The nice part is that a person can read it too: whoever looks in the morning sees where it stopped in 8 lines.

3. Summarising: the last resort

If the window still nears its limit, old turns are reduced to a single note. There are two rules here, and both were learned the hard way:

• Identifiers, decisions and open questions are carried word for word. They are not rephrased. In my first attempt the model wrote WD-24817 as WD-24871, and the agent went on to review the wrong record. Numbers travel outside the summary, as a list.
• The first answer after a summary gets checked. Summarising is the most fragile moment of a run. Ours was clearly less accurate right after a summary, so we ordered the flow to make the first step after a summary a verifiable one: a count, or an identifier check.

From the field: turn 13 of the night job

In the orchestrator post, turns 13–15 of that 17-turn night were the ones where the agent started a summary, stopped and fetched data again. I read those turns again, this time from the context side:

The mistake here is mine, not the model’s: I put decisions in the same place as data. “Date range is 18.09 00:00–23:59” is a decision and holds for the whole run; the 2,300-row list was used once and finished. Put them in the same stack and the data buries the decision.

Why “just use a bigger window” is not the answer

Moving to a model with a larger window does not change how fast you fill it; it only delays the moment. All three problems stay: quality drops in the middle, cost is paid again every turn, latency grows. Our 4-billion-parameter model had a small window to begin with, and that forced discipline on us early. When I later tried a large-window model, the same mistakes appeared later but in exactly the same shape. Window size is a budget; a bigger budget does not make spending discipline optional.

What to watch

• Tokens per run, broken down by turn. You see which turn jumps; the jump is always a tool reply.
• Window fill, p95. Above 70%, quality loss has started.
• Re-sent token ratio. Total tokens ÷ unique tokens. Above 3 means the stack is carried too long.
• Number of summaries, and error rate right after one. The most fragile moment of a run; measure it.
• Tool reply size, p95 per tool. This number names your fattest tool.
• Time to first answer. The only number the user sees.

Checklist

Conclusion

Two sessions gave two different answers to the same question. The difference was not the model, not the prompt, not the data: it was the window. In the full window the right record was there, but it sat among 2,300 rows and the model could not pick it. In the clean window the same record was one of 13 rows.

We did three things, in order: trimmed the tools, moved run state out of the window, and made summarising a last resort. Tokens per run went from 118k to 31k, time to first answer from 34 seconds to 11, and label accuracy from 24/34 to 31/34. None of it was a model change.

The sentence to remember: context is a desk, not a warehouse. Watch what you put on it. A bigger desk does not fix the mess; it only lets you carry more of it.