Implementing Idempotency Keys in APIs to Prevent Duplicate Actions

Introduction

If you expose APIs that trigger work, a dropped network connection can turn a single user action into several HTTP requests. In production this happens constantly: clients retry on timeouts, users tap buttons twice, mobile networks reconnect, and load balancers replay traffic. The endpoint may be technically correct and still create duplicate side effects.

A request is considered idempotent when repeating it multiple times has the same effect as once. GET is usually safe to retry, but POST is not by default. An idempotency key is the practical pattern that lets you make POST-style writes tolerate retries without duplication.

Why retries create real-world duplicates

A retry happens for many legitimate reasons. The client sends a request, never receives a response, and assumes failure. A second request is issued just in case. If the first request succeeded, you now have two side effects.

A minimal failing flow

app.post('/checkout', async (req, res) => {
  const { cartId, cardToken } = req.body

  const order = await orders.create({ cartId, cardToken })
  await billing.capturePayment(order.id, cardToken)

  res.status(201).json({ orderId: order.id })
})

There is no protection here for duplicate submissions. If this handler runs twice, you get two charges and potentially two fulfilled orders.

Typical duplicate triggers

• Manual double-submit from UX latency
• SDK retry when transport or timeout happens
• Gateway/reverse-proxy retries after ambiguous upstream response codes
• Client-side offline/online reconnect races
• Retry loops caused by exponential backoff

Even if you debounce UI and disable buttons, you still need server-side protection because retries can arrive from third-party systems you do not control.

Idempotency flow and state model

The most reliable model is to persist request identity before doing irreversible side effects:

1. Client sends a stable Idempotency-Key per business action.
2. Server checks if key exists for this route.
3. If key is cached with a completed response, return that response.
4. If key is in progress, decide whether to wait, reject, or treat as conflict.
5. If new key, reserve it, run business logic, persist result, and return the response.

sequenceDiagram
  autonumber
  participant C as Client
  participant A as API Server
  participant S as Storage

  C->>A: POST /payments (Idempotency-Key: abc-123)
  A->>S: SELECT key state
  S-->>A: not found
  A->>S: INSERT key with status=in_progress
  A->>A: process payment/order creation
  A->>S: UPDATE key with status=completed and response
  A-->>C: 201 + response payload
  Note over C,A: Retry with same key
  C-->>A: POST /payments (Idempotency-Key: abc-123)
  A->>S: SELECT key state
  S-->>A: status=completed + cached response
  A-->>C: same 201 payload

Recommended storage fields

A simple table supports most implementations:

CREATE TABLE idempotency_records (
  key_value      TEXT PRIMARY KEY,
  route          TEXT NOT NULL,
  request_hash   CHAR(64) NOT NULL,
  status         TEXT NOT NULL CHECK (status IN ('in_progress', 'completed', 'failed')),
  http_status    INTEGER,
  response_body  JSONB,
  created_at     TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  completed_at   TIMESTAMPTZ
);
``

Keep `request_hash` to prevent the same key being reused with different payloads.

{ad-block}

## Practical Node.js implementation pattern

A robust server pattern is to require the idempotency key, check it transactionally, and only commit side effects once the key is reserved.

```js
app.post('/checkout', async (req, res) => {
  const key = req.header('Idempotency-Key')
  if (!key) return res.status(400).json({ error: 'Idempotency-Key required' })

  const payloadHash = crypto
    .createHash('sha256')
    .update(JSON.stringify(req.body))
    .digest('hex')

  const record = await db.query(
    `SELECT status, response_body, request_hash
       FROM idempotency_records
      WHERE key_value = $1 AND route = $2`,
    [key, '/checkout']
  )

  if (record.rows.length) {
    if (record.rows[0].request_hash !== payloadHash) {
      return res.status(409).json({ error: 'Idempotency key reused with different payload' })
    }
    if (record.rows[0].status === 'completed') {
      return res.status(record.rows[0].http_status).json(record.rows[0].response_body)
    }
    if (record.rows[0].status === 'in_progress') {
      return res.status(409).json({ error: 'Request already processing' })
    }
  }

  await db.query(
    `INSERT INTO idempotency_records (key_value, route, request_hash, status)
     VALUES ($1, $2, $3, 'in_progress')`,
    [key, '/checkout', payloadHash]
  )

  try {
    const order = await createOrder(req.body)
    const payment = await capturePayment(order)
    const responseBody = { orderId: order.id, paymentId: payment.id }

    await db.query(
      `UPDATE idempotency_records
          SET status = 'completed', http_status = 201,
              response_body = $1, completed_at = NOW()
        WHERE key_value = $2`,
      [JSON.stringify(responseBody), key]
    )

    return res.status(201).json(responseBody)
  } catch (error) {
    await db.query(
      `UPDATE idempotency_records SET status = 'failed' WHERE key_value = $1`,
      [key]
    )
    throw error
  }
})

Important implementation details

• Use a unique key route pairing, not just key value, so different endpoints cannot collide.
• Keep keys short-lived (for example 24h-7d) with periodic cleanup.
• Return a stable cached response structure, not just a success boolean.

DELETE FROM idempotency_records
WHERE completed_at < NOW() - INTERVAL '14 days';

Client responsibilities and retry strategy

Servers cannot prevent every race alone. Clients still need sane behavior:

const headers = {
  'Content-Type': 'application/json',
  'Idempotency-Key': crypto.randomUUID(),
}

async function requestWithRetry(payload, attempt = 0) {
  try {
    const res = await fetch('/api/checkout', {
      method: 'POST',
      headers,
      body: JSON.stringify(payload),
    })

    if ([408, 425, 429, 500].includes(res.status) && attempt < 3) {
      await new Promise((r) => setTimeout(r, 200 * 2 ** attempt))
      return requestWithRetry(payload, attempt + 1)
    }
    return res
  } catch (err) {
    if (attempt < 3) {
      await new Promise((r) => setTimeout(r, 200 * 2 ** attempt))
      return requestWithRetry(payload, attempt + 1)
    }
    throw err
  }
}

Keep the same Idempotency-Key across retry attempts for the same logical action. If the key changes, each retry becomes a new operation and the server cannot safely de-duplicate.

Failure modes and what to watch for

You avoid duplicates, but you can still run into subtle bugs:

1. Different payload, same key: reject with 409 so clients fix their transport logic.
2. Long-running handlers: requests may stay in_progress for seconds; if a client times out mid-flight, retrying should return the same in_progress result or a clear conflict message.
3. Concurrent processing: multiple app instances can race on the same key; always rely on database uniqueness and atomic updates.
4. Storage outages: if idempotency storage fails, fail fast and disable writes rather than accepting duplicate writes.

A useful compromise is to apply idempotency only on critical mutation routes (POST /payments, POST /orders, POST /transfers) and keep fast, low-risk routes simpler.

Conclusion and Next Steps

Idempotency keys are not just a retry workaround; they are a correctness feature for distributed systems. They make your API resilient to client and network behavior outside your direct control. Start with a tiny scope: one high-value endpoint, one key format, one persistence table, and one retry policy.

Then expand the pattern by adding:

• structured metrics around hit/miss/conflict states
• observability for in_progress durations
• shared cache integration for read-through response replay

If you want strong reliability for write APIs, the idempotency key pattern is a high-impact change with a relatively small code footprint.