Distributed Systems3 min read

Retries are not reliability without idempotency

Why networks make duplicate work inevitable, and how idempotency keys turn unsafe retries into a deliberate protocol.

Ayomide OlajideSoftware engineer

A repeated key returns the stored result. New operations reserve a unique key before applying a durable effect. External providers also need idempotency.

The request succeeded. The client did not hear it.

Imagine a checkout request that reaches the server, charges a card, and then loses its response during a network interruption. The client retries. Without a protocol, the system has two valid requests and one very unhappy customer.

Retries are not exceptional in distributed systems. Timeouts, connection resets, process restarts, and load balancer decisions can all make the caller uncertain about whether work completed. A retry policy must therefore be paired with a way to recognize the same logical operation.

Make the operation nameable

An idempotency key is a client-generated identifier for one logical command. The server stores the key with the request fingerprint and final result. A second request with the same key returns the stored result; a second request with different parameters is rejected. For effects inside one database, commit the key and the protected mutation in the same transaction. An external charge needs the provider’s idempotency protocol as well; a database rollback cannot undo a payment. Scope keys to the authenticated customer and define retention explicitly.

A repeated key returns the stored result. New operations reserve a unique key before applying a durable effect. External providers also need idempotency.

// Pseudocode: scope keys to the authenticated customer.
const operation = await operations.reserveOrRead({
  customerId, key, fingerprint: fingerprint(input)
}) // UNIQUE(customerId, key); commit before contacting provider
if (operation.fingerprint !== fingerprint(input)) throw new ConflictError()
if (operation.status === "complete") return operation.response

// Concurrent retries use the SAME provider key and payload.
const result = await provider.charge(input, {
  idempotencyKey: operation.providerKey
})
await operations.complete(operation.id, result)
return result

Recover after a partial failure

If the provider charges successfully but the process crashes before saving the result, retry with the same provider key or reconcile against provider status. Never create a new key just because the response was lost. A reservation is not proof of success; expose pending and completed states separately.

Concurrent callers need a unique constraint or equivalent atomic reservation, a stable request fingerprint, and a documented in-progress response. Keep reconciliation within the provider’s key retention window. Once that window expires, resolve the uncertain outcome before attempting another charge.

Know what you cannot promise

Exactly-once execution is rarely available end to end. You can often provide exactly-once effects within one storage boundary, or at-least-once delivery with idempotent consumers. External providers may require their own idempotency key. A queue can redeliver a message; that is a feature for durability, not a bug to hide.

A common mistake is retrying every error. Retry timeouts and transient 503 responses with bounded exponential backoff and jitter. Do not retry validation failures, authorization failures, or a known business conflict.

Practical conclusion

Treat retries as part of the API contract. Name operations, persist their outcomes, validate request fingerprints, cap attempts, and instrument duplicate detection. Reliability is not the absence of failure; it is predictable behavior when failure is normal.