The Agent’s Trail: A Chain of Decisions, Not Logs

Why a flat log is not enough

Three things flow at once in an agent run: the model’s turns, the tool calls, and the orchestrator’s steps. A flat log lines them all up on one strip. The problem is that our questions are tree questions, not strip questions:

• “What data was this answer based on?” → look at the children of the span that produced it.
• “Why was this tool called?” → look at the span’s parent.
• “Where did the time go?” → compare sibling spans.
• “What changed between yesterday and today?” → put two trees side by side.

None of these is answered by reading “line 217”. The problem in one screen, not six log files applies here exactly; the only difference is that step boundaries replace service boundaries.

What we record

trace: rec-2026-09-21-01        (nightly reconciliation, 38 s)
├─ span: step-1 fetch_data       6.1 s
│   ├─ tool: read_bank_file      2.2 s   arg#a91f  reply 480 B
│   └─ tool: list_transactions   3.8 s   arg#7c02  reply 512 B  count=2300
├─ span: step-2 match            0.4 s   (code, no model)
├─ span: step-3 label           21.3 s
│   └─ model: qwen3:4b  in 4,812 tok  out 233 tok  turn 1
└─ span: step-4 write_summary   10.6 s
    └─ model: qwen3:4b  in 1,340 tok  out 412 tok  turn 1

The fields we keep on each span, in the order we found them useful:

What we do not record

• Raw customer data. Names, IBANs and identity numbers are masked. Record ids stay readable; without them you cannot debug.
• The whole tool reply. First 500 characters plus a hash. If you need all of it, the data is still at its source.
• All of the model’s intermediate text. The decision and the first sentence of the reasoning is enough; the rest solves a problem once every 14 days and eats disk daily.

Retention: detailed traces for 14 days, summary metrics for 13 months. The audit log is separate and longer: every side-effecting call with full arguments, in its own place, with its own access rules.

From the field: 40 minutes, then 12 seconds

The first time, I went like this: download the log, find the timestamp, pick out the tool calls by eye, compare arguments by hand, see that the date range was off by one day. 40 minutes.

After the tracing was in place, the same complaint came in:

1. The user’s “wrong” button carries the trace id; I opened the link.
2. In the tree I looked at the list_transactions span under step-1: argument summary start=2026-09-19, where it should have been 2026-09-20.
3. The parent span carried the version of the code that produces date ranges; a fix had shipped that day.

12 seconds. The real win was seeing in seconds that the error was not in the model. Without tracing, half of those 40 minutes had gone into reading the model’s answer and wondering whether it had misunderstood.

Replay

The trace holds the hash and the first 500 characters of each tool reply, and for failed runs we keep the full reply for 48 hours. That makes replay possible: same inputs, stored tool replies, no real tool calls.

$ agent replay rec-2026-09-21-01 --model qwen3:4b
  step-3 label: 13 records -> 11 correct (previous run: 11)
  result: identical

$ agent replay rec-2026-09-21-01 --model qwen3:8b --prompt v7
  step-3 label: 13 records -> 13 correct
  result: different (2 records fixed)

What it gives you:

• It splits the error. If the same error appears with stored replies, it is the model; if it does not, the tool or the data changed.
• The cheapest version comparison. On a real run, without calling real systems.
• No side effects. Writing tools are disabled in replay mode, so you never re-run a step that moves money.
• A source for the eval set. Real failed runs are the most valuable test data you have.

Tying feedback to the trail

The “this answer is wrong” button in the chat window used to increment a counter and nothing else. It got 20–30 clicks a month and we could not review any of them, because we did not know which run they belonged to. After we added the trace id to the button:

The last row matters: more than half of the feedback had nothing to do with the model. “The model is bad” is the default diagnosis of teams without traces.

What to watch

• Steps and turns per run. Tells you whether the flow is written down.
• Duration per step, p50/p95. Shows where the slowness is; the model is not always guilty.
• Tool argument error rate. Our biggest category; it deserves its own chart.
• Brake hits by type. Which brake, how many times?
• Feedback → root cause distribution. Monthly; it tells you where the work should go.
• Sampling rate and storage size. Observability costs money too; ours is about 11 KB per run.

Checklist

Conclusion

That day I spent 40 minutes inside 400 lines of log and what I found was a date range. The model had nothing to do with it; I spent half an hour on it because I could not tell whether it did. That is exactly the value of observability: telling you quickly who is not guilty.

What we built is not new technology; it is the trace–span structure we have used in microservices for years, adapted to an agent. The only additions are agent-specific fields: argument summary, tokens, prompt version, brake. Plus replay, which we use most of all.

The sentence to remember: an agent’s trail is the chain of its decisions. If you do not record the chain, all you have is the result, and you cannot tell from a result what to fix.