The Agent Called It Twice: Idempotency, Not Autonomy

Why does it repeat?

The problem from how many times does a message arrive applies here exactly: when the answer to a network call does not arrive, you cannot know that the call did not happen. There are two possibilities and they look identical: the request never went out, or it went out and the answer never came back. With an agent, one more layer sits on top of this classic problem: the side that decides is a model, and a model that is unsure tends to try again.

From the field: a 32-second approval

Here is that Friday in detail. The agent was running a job over 6 withdrawals that had queued overnight: “approve the ones that match the rule set” (test environment, no real money).

What the model did here is not wrong: no answer came, so it tried again. I would have done the same. What was wrong is that the same work was accepted a second time. Leaving that to a model “being careful” is like leaving a two-thread race to “careful code”: it works for a while, and then it does not.

Who generates the key?

Our first attempt at a fix was to add an idempotency_key field to the tool schema and let the model fill it. Over two runs it produced two different keys — one wd-24817-approve, the other WD24817_approve_2. Nothing was protected; we just got two rows in the database.

The right way: the key is never shown to the model. The orchestrator builds it from the natural key of the work and adds it to the call itself.

# natural key: the fields that make this piece of work unique
def idempotency_key(tool_name, args, business_day):
    natural = "|".join([
        tool_name,                 # approve_withdrawal
        args["record_id"],         # WD-24817
        business_day.isoformat(),  # 2026-09-20
    ])
    return sha256(natural).hexdigest()[:32]

# same work -> same key; different work -> different key
# the model neither sees nor fills this field

Why the business day? We may deliberately want to approve the same record again a week later; if the key stayed the same forever, a legitimate second operation would be blocked too. Pick the window from the nature of the work: for us, one business day was the right answer.

Server side: what should the second call return?

A key on its own does nothing; what matters is how the receiving side behaves. The same rule as in the delivery guarantees post:

def approve_withdrawal(record_id, _key):
    seen = idempotency_table.find(_key)
    if seen:
        # IMPORTANT: not an error, the first result itself
        return {**seen.result, "repeat": True}

    with transaction():
        # unique index: of two concurrent requests, one fails here
        idempotency_table.insert(_key, status="running")
        result = payment_service.approve(record_id)
        idempotency_table.complete(_key, result)
    return result

Two details were learned the hard way:

• Do not return an error to the second call. Our first version returned 409 Conflict; the model read it as “it failed”, tried a third time and then told the user “I could not approve it”. It had been approved. The right answer is the first result; the "repeat": true field is only for the log.
• Uniqueness belongs in the database. “Check first, then insert” lets both concurrent calls through. Without a unique index this is a race condition, and it is far more likely once you have parallel sub-agents.

Irreversible work: two stages

Idempotency solves “do not do it twice”. It does not solve “this should never have happened”. For sending money, sending email, deleting records and calling external systems, we split the tool in two:

# stage 1: the agent only produces a plan (no side effect)
plan_approve_withdrawal(record_id)
# -> {"record_id": "WD-24817", "amount": 1250, "currency": "USD",
#     "rule": "under_limit", "key": "a91f...", "status": "awaiting_approval"}

# stage 2: code executes after approval; the model never calls this
# approval rule lives in the flow:
#   amount < 500 USD and KYC complete  -> automatic
#   everything else                     -> human approval

The plan goes into a queue, and the queue is drained by the flow, not the agent. Three benefits:

• If the agent decides wrongly, the cost is a queue row; undoing it means deleting a line.
• The approval rule is in code, so it can be read and tested. “Ask for approval above $500” in a prompt cannot be tested.
• Even someone who tricks the agent through prompt injection can only get a row written into a queue; they cannot move money.

The list of things you cannot undo

We keep a page with exactly that name. Every side-effecting tool goes on it and answers three questions: can it be undone, who approves it, and what happens if it repeats?

The real value of the list showed up when adding tools: whoever writes a new side-effecting tool cannot open a PR without answering those three questions. If there are no answers, it stays a read-only tool.

What to watch

• Idempotency hits. How many calls came back as “already done”? Zero means the protection has never been tested; a rising number means you should look at the timeout.
• Side-effecting calls per run. More than expected means the model is writing when it should be reading.
• Timeout rate and tool duration p99. This was the root cause of our incident: p99 was 31 seconds, the limit 30.
• Waiting time in the approval queue. If it grows, human approval has become the bottleneck; revisit the threshold.
• Automatic approval rate. If it climbs, the threshold may have loosened; review it monthly.
• Audit log completeness. Does every side-effecting call carry the run id, the key, the arguments and the result?

Checklist

Conclusion

That Friday, the agent did nothing wrong: no answer came, so it tried again. The system was wrong, because it accepted the same work a second time. This whole post is an old subject in new clothes: the network is unreliable, calls repeat, uniqueness belongs in the database. The only new part is that the caller doing the repeating is now a model, and you cannot know in advance when it will repeat.

After the fix: code generates the key, the second call returns the first result, and any step that moves money is split into plan, approve and commit. The agent still retries — we see “repeat: true” in the log three or four times a month — but it is now a log line, not an incident.

The sentence to remember: an agent’s autonomy is worth exactly as much as its tools’ idempotency. Without the second, the first is not a feature; it is an open risk.