The Anatomy of Duplicate Requests in Distributed Systems

In distributed architectures, network failure is not an anomaly—it is a guarantee. When a client sends a POST /v1/payments request, the network can drop packets at two distinct stages: on the way to the server, or on the way back after the server has already executed the business logic. Because HTTP clients and upstream API gateways automatically trigger retry policies upon receiving connection timeouts or TCP resets, a single user tap can translate into multiple identical mutation requests hitting your backend instances concurrently.

Without a robust idempotency architecture, retries lead to devastating double-spend anomalies, duplicate orders, or corrupted financial ledger balances. Simply relying on unique database constraints is insufficient because mutations often coordinate multiple subsystems: billing third-party APIs (like Stripe), decrementing inventory, and appending event records.

The Idempotency Lifecycle State Machine

A production-ready idempotency framework requires a deterministic state machine for every unique request identifier (commonly supplied via the Idempotency-Key HTTP header). A mutation passes through three distinct phases:

  • STARTED (or IN_PROGRESS): The server claims execution ownership and locks the idempotency key with a finite Time-To-Live (TTL).
  • COMPLETED: The business logic and database transaction have committed successfully. The HTTP status code and response payload are permanently stored.
  • FAILED: The transaction failed with an unrecoverable business error or application exception, releasing the lock or storing a deterministic failure response.

If a concurrent request arrives while the status is STARTED, the server must reject it immediately with an HTTP 409 Conflict or pause and poll until the active transaction completes.

Layer 1: Atomic Acquisition via Redis Lua Script

Distributed locks using standard SETNX commands are vulnerable to race conditions if the lock acquisition and payload inspection are split into multiple network round-trips. We solve this by executing an atomic Lua script directly inside Redis. This script checks whether an idempotency key exists, inspects its state, and either locks the key or returns the pre-computed cached response.

-- KEYS[1]: Idempotency cache key (e.g., 'idemp:orders:req_8f14c0a')
-- ARGV[1]: Current execution token (unique worker UUID)
-- ARGV[2]: Lock TTL in seconds (e.g., '120')

local current = redis.call('GET', KEYS[1])

if not current then
    -- Key does not exist: claim lock and mark as IN_PROGRESS
    local payload = cjson.encode({
        status = 'IN_PROGRESS',
        owner = ARGV[1],
        created_at = redis.call('TIME')[1]
    })
    redis.call('SET', KEYS[1], payload, 'EX', tonumber(ARGV[2]))
    return { 'ACQUIRED', '' }
else
    -- Key exists: decode JSON and inspect state
    local data = cjson.decode(current)
    if data.status == 'COMPLETED' then
        return { 'COMPLETED', current }
    elseif data.status == 'IN_PROGRESS' then
        return { 'LOCKED', current }
    else
        return { 'FAILED', current }
    end
end

Executing this logic atomically inside Redis guarantees zero race windows. If two identical requests hit two different API replicas within a single millisecond, exactly one replica receives ACQUIRED, while the second receives LOCKED and can immediately abort or yield.

Layer 2: Persistent Storage and Atomic Transitions in PostgreSQL

Redis is volatile and should serve as an execution barrier and ephemeral response cache. The definitive source of truth must reside within PostgreSQL. We design a dedicated idempotency_keys table that couples the operation directly to the application database transaction.

CREATE TABLE idempotency_records (
    key VARCHAR(255) PRIMARY KEY,
    user_id UUID NOT NULL,
    request_path VARCHAR(255) NOT NULL,
    request_hash CHAR(64) NOT NULL,
    status VARCHAR(32) NOT NULL DEFAULT 'IN_PROGRESS',
    response_code INTEGER NULL,
    response_body JSONB NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_idempotency_user_created 
ON idempotency_records (user_id, created_at DESC);

The request_hash column stores an SHA-256 digest of the incoming HTTP request payload. If an attacker or buggy client sends the same Idempotency-Key with a mutated payload (for example, changing the payment amount from $10 to $1000), the backend detects the hash mismatch and immediately throws an HTTP 422 Unprocessable Entity.

End-to-End Orchestration in Node.js

Here is how the atomic coordination integrates into an Express or Fastify middleware workflow:

import crypto from 'node:crypto';
import { db } from './database';
import { redis } from './redis';

export async function handleIdempotentOrder(req, res) {
  const idempotencyKey = req.header('Idempotency-Key');
  if (!idempotencyKey) {
    return res.status(400).json({ error: 'Missing Idempotency-Key header' });
  }

  const userId = req.user.id;
  const redisKey = `idemp:${userId}:${idempotencyKey}`;
  const workerId = crypto.randomUUID();
  const requestHash = crypto.createHash('sha256').update(JSON.stringify(req.body)).digest('hex');

  // 1. Evaluate Redis Lua script
  const [verdict, cachedPayload] = await redis.eval(
    LUA_IDEMPOTENCY_SCRIPT,
    1,
    redisKey,
    workerId,
    120 // Lock TTL in seconds
  );

  if (verdict === 'COMPLETED') {
    const cached = JSON.parse(cachedPayload);
    res.set('X-Cache-Lookup', 'HIT');
    return res.status(cached.response_code).json(cached.response_body);
  }

  if (verdict === 'LOCKED') {
    return res.status(409).json({
      error: 'Conflict: A request with this idempotency key is actively processing'
    });
  }

  // 2. Open PostgreSQL transaction for the business logic
  const client = await db.connect();
  try {
    await client.query('BEGIN');

    // Insert idempotency state with ON CONFLICT safety
    await client.query(
      `INSERT INTO idempotency_records 
       (key, user_id, request_path, request_hash, status)
       VALUES ($1, $2, $3, $4, 'IN_PROGRESS')`,
      [idempotencyKey, userId, req.path, requestHash]
    );

    // Execute business mutation: charge customer, insert order
    const orderResult = await executeOrderCreation(client, req.body);
    const responsePayload = { orderId: orderResult.id, status: 'processed' };
    const statusCode = 201;

    // Transition record to COMPLETED
    await client.query(
      `UPDATE idempotency_records
       SET status = 'COMPLETED',
           response_code = $1,
           response_body = $2,
           updated_at = NOW()
       WHERE key = $3`,
      [statusCode, JSON.stringify(responsePayload), idempotencyKey]
    );

    await client.query('COMMIT');

    // 3. Update Redis cache with permanent result (24h TTL)
    const finalCache = JSON.stringify({
      status: 'COMPLETED',
      response_code: statusCode,
      response_body: responsePayload
    });
    await redis.set(redisKey, finalCache, 'EX', 86400);

    return res.status(statusCode).json(responsePayload);
  } catch (err) {
    await client.query('ROLLBACK');
    // Release the Redis lock immediately on application error
    await redis.del(redisKey);
    throw err;
  } finally {
    client.release();
  }
}

Edge Cases and Production Pitfalls

1. Zombie Locks from Process Crashes

If an API worker crashes (such as via an Out-Of-Memory termination) midway through executing business logic, the Redis key must have an aggressive TTL (e.g., 60–120 seconds). Never acquire an idempotency lock without a TTL. When the TTL expires, incoming retries can re-claim the lock. In PostgreSQL, clean up abandoned IN_PROGRESS rows via a background sweeper job that marks records older than 15 minutes as FAILED.

2. Payload Mutation Attacks

Clients might accidentally reuse an idempotency key generated for an earlier, different request. Comparing the stored SHA-256 request_hash against the incoming body prevents silent state corruption. Always validate that the current user context (user_id) matches the original record owner.

3. The Distributed Commit Dilemma

Notice that the database transaction commits before the Redis key is updated with the completed payload. If the Redis write fails due to a network glitch, the database remains consistent. The next client retry will see the lock expired in Redis, hit the database table, find the COMPLETED record, backfill Redis, and return the saved response safely.

Key Takeaways

  • Network timeouts are ambiguous: Never assume a failed HTTP request means business logic did not execute on the backend.
  • Atomic Lua scripting prevents race conditions: Split-second concurrent retries must be intercepted at the caching layer before reaching expensive database locks.
  • Validate payload hashes: Pin each idempotency key to a cryptographic hash of the request payload to catch corrupted or reused client keys.
  • PostgreSQL is the source of truth: Keep your definitive execution records inside ACID database transactions; treat Redis purely as an acceleration and lock barrier.