Idempotency Keys: Making Agent Tool Calls Safe to Retry

An agent decides to refund order 8123. The HTTP request times out. The agent's retry loop fires the same tool call again — and the refund runs twice. Nothing about the second request knows the first one happened. The vendor's dashboard shows two refunds, the agent's transcript shows one tool call that "eventually succeeded," and the difference comes out of someone's incident review.

This is not an agent bug. Retries are correct behavior; the agent loop should retry transient failures. The bug is that the tool's API has no memory of attempts. Idempotency keys are that memory, and adding them to an agent-facing tool API is cheaper and safer than any amount of retry-policy tuning. This article walks through the full contract: what the client sends, what the server stores, where keys come from, and what this pattern does not fix.

What a retry actually repeats

Agent loops retry more aggressively than human-driven clients. A typical harness wraps every tool call in a policy like: retry on timeout, connection reset, 429, and 5xx, with exponential backoff, up to three attempts. That policy is right — models stall, gateways flap, and a tool call that gives up on the first blip makes the whole agent flakier.

But every one of those retries re-sends the same intent. If the first attempt reached the backend and the failure happened on the response path — which is the most common timeout shape — then the operation already ran. The retry creates a second execution of an operation the system believes happened once.

The classic distributed-systems framing applies unchanged: with retries, delivery is at-least-once, and at-least-once means duplicates are possible. The question is not whether a duplicate will occur but what happens when it does. For reads, nothing. For create_invoice, you get two invoices.

The idempotency key contract

The contract fits in three sentences. The client attaches a unique key to every logical operation. The server records the key when it first sees it and stores the response it produced. If the same key arrives again — in flight or hours later — the server returns the stored response instead of running the operation a second time.

Stripe's Idempotency-Key header is the canonical example, and the shape is worth copying even for internal APIs:

POST /v1/refunds HTTP/1.1
Idempotency-Key: 7f9d2c1e-4b8a-4f6e-9a21-0c5d8e2f7b44
Content-Type: application/json

{ "order_id": "8123", "amount": 4900, "reason": "damaged" }

Three properties matter in that one header:

• Same key, same operation. The key identifies the logical refund of order 8123 — not "a refund," not "this session."
• Different keys are different operations. Two keys for the same payload means two refunds. This is a feature: a second, deliberate refund gets its own key.
• The key survives the client. It is generated before the first attempt and re-sent on every retry, unchanged.

Generating keys on the agent side

The agent harness is the right place to generate keys, because the harness owns the retry loop. The rule: one key per logical operation, stable across retries of that operation, fresh for each new operation. Deriving it from the agent's own call identity keeps that honest:

import uuid

def execute_tool(tool_call):
    # tool_call.id is stable across retries of THIS call and unique
    # per call the model makes — exactly the key's contract.
    idempotency_key = f"agent-{tool_call.id}"
    return http.post(
        f"{API_BASE}/refunds",
        json={"order_id": tool_call.args["order_id"],
              "amount": tool_call.args["amount"]},
        headers={"Idempotency-Key": idempotency_key},
    )

Two failure modes to design out here. First, generating the key inside the retry closure — every attempt mints a fresh key and you have built a duplicate machine with extra steps. Second, keying on the model's transcript position or a timestamp: both drift across retries and across harness restarts. A per-call UUID minted once, before the first attempt, is boring and correct.

One nuance for agent systems specifically: if the model can issue what is logically the same operation twice (two refund calls for order 8123 in one session), those get different keys and will both execute. If that should be impossible, that is an approval-gate or ledger problem — a separate mechanism, described in the earlier piece on command ledgers. Idempotency keys protect one logical operation against transport-level duplication; they are not a business-rule engine.

Server side: the replay window

The server implementation is a single table with a unique constraint, and the discipline to insert before doing the work:

CREATE TABLE idempotency_keys (
  key           TEXT PRIMARY KEY,
  endpoint      TEXT NOT NULL,
  request_hash  TEXT NOT NULL,
  response_code INT,
  response_body BYTEA,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

The request handler becomes: insert the key; if the insert conflicts, return the stored response; otherwise run the operation, store the response, and return it.

def handle_refund(request):
    key = request.headers["Idempotency-Key"]
    try:
        db.execute(
            "INSERT INTO idempotency_keys "
            "(key, endpoint, request_hash) VALUES (%s, %s, %s)",
            (key, "/v1/refunds", hash_payload(request.body)),
        )
    except UniqueViolation:
        prior = db.one(
            "SELECT response_code, response_body FROM idempotency_keys "
            "WHERE key = %s", (key,))
        if prior.response_code is None:
            return 409  # original still in flight — retry with backoff
        return prior.response_code, prior.response_body

    result = run_refund(request.json)          # the real work
    db.execute(
        "UPDATE idempotency_keys SET response_code = %s, "
        "response_body = %s WHERE key = %s",
        (result.status, result.body, key),
    )
    return result.status, result.body

The 409 branch matters more than it looks. Two concurrent first attempts — the exact race that load-balancer retries and per-link failovers produce — will both try to insert. Only one wins; the loser must learn the operation is in progress and back off, not assume it failed. Returning the stored response for a request whose original is still executing would fabricate a result out of thin air.

Fingerprint mismatch is a bug, not a retry

The request_hash column exists to catch a client that reuses a key for a different payload. That is never a legitimate retry — it is a client bug, and the worst response is to silently replay the old result, because the caller believes new work happened.

So compare hashes on replay and fail loudly:

if prior.request_hash != hash_payload(request.body):
    return 422, ("idempotency key reused with a different payload; "
                 "mint a new key for a new operation")

Stripe returns idempotency_error in this case. The name matters less than the behavior: a key/payload mismatch must be impossible to mistake for success. In agent systems this error is a gift — it means the harness's key derivation is broken, and it surfaces in the transcript where an engineer will actually see it.

Expiry and retention

Keys need a lifetime, or the table grows forever and eventually every INSERT pays for a history nobody replays. The window should cover your worst realistic retry horizon: the longest backoff sequence, plus queue delays, plus a human investigating the next morning. Twenty-four hours is the common default and a fine starting point; payment-adjacent operations often keep seven days to survive weekend incident timelines.

A nightly job is enough:

DELETE FROM idempotency_keys
WHERE created_at < now() - INTERVAL '7 days';

The expiry also defines the honest answer to "how safe is this?" — retries are deduplicated within the window. A retry that arrives after expiry executes for real. Size the window to make that event unreachable by your retry policy, and document the boundary for callers instead of implying forever-safety.

What idempotency keys do not fix

The pattern has a precise shape, and pretending it covers more is how systems get bitten:

• Different operations on the same entity. Two refund calls with two keys are two refunds. Business-level deduplication ("one refund per order per day") is a different control at a different layer.
• Side effects beyond your boundary. If the handler calls an email vendor after writing the database, a crash between those steps replays the email on retry unless the vendor has its own idempotency story. The key deduplicates your API, not the internet.
• Ordering. Keys make repeats of one operation harmless; they say nothing about two different operations racing each other.
• Reads, mostly. Retrying a read is already safe unless it is an expensive read worth caching under the same mechanism — which is a legitimate but separate use.

Testing: replay the same request twice

The contract is testable in a few lines, and it belongs in CI next to the tool's other tests:

def test_idempotent_replay(client):
    key = "test-key-1"
    body = { "order_id": "8123", "amount": 4900 }
    r1 = client.post("/v1/refunds", json=body, headers={"Idempotency-Key": key})
    r2 = client.post("/v1/refunds", json=body, headers={"Idempotency-Key": key})
    assert r1.status_code == 200
    assert r2.status_code == 200
    assert r1.json() == r2.json()
    assert count_refunds("8123") == 1

The last assertion is the one that catches real regressions — the responses can match while two refunds exist if someone moved the insert after the work. A second test asserts the fingerprint path: same key, different body, expecting the loud error rather than a replay.

For agent harnesses, extend the replay test to the transport layer: make the first attempt return a 503 after the backend committed, and verify the retried call returns the committed result exactly once. That is the precise scenario idempotency keys exist for, and it is fully testable with a fault-injecting proxy in front of the tool.

The checklist

• One key per logical operation, minted before the first attempt, stable across retries.
• The server inserts the key before doing the work; concurrent losers get a retryable in-flight response, never a fabricated one.
• Replays compare a payload fingerprint; mismatches fail loudly instead of replaying.
• Retention window sized past the worst retry horizon, enforced by a cleanup job, documented to callers.
• A two-identical-requests test in CI asserts one side effect — not just two matching responses.

An agent that can retry without fear makes strictly better decisions: it stops treating transient failures as verdicts. Idempotency keys are how you buy that, for the price of one header and one table.