Every distributed system engineer learns this lesson the hard way: networks are unreliable, and at-least-once delivery is the only guarantee that cloud payment gateways, SMS aggregators, and message queues can practically provide.
When a webhook times out, Stripe or Chapa will retry the event three seconds later. If your webhook handler blindly increments account balances or triggers order fulfillment without idempotency safeguards, you will double-charge or double-credit your users.
Idempotency keys at the database layer
Idempotency cannot rely on memory caches that reset when a pod restarts. It must be enforced at the database level where relational constraints provide absolute guarantees.
We persist an idempotency_keys table in PostgreSQL with a unique constraint on (key, consumer). When a webhook arrives, the worker attempts an atomic INSERT ... ON CONFLICT DO NOTHING. If zero rows are inserted, the worker recognizes this event as a duplicate and immediately returns HTTP 200 without executing side-effects.
const record = await prisma.idempotencyKey.createMany({
data: [{ key: eventId, scope: 'payment_webhook' }],
skipDuplicates: true,
});
if (record.count === 0) {
// Duplicate event received; acknowledge safely without re-executing
return { status: 'already_processed' };
}Atomic transactions over memory locks
Distributed Redis locks are popular, but they add operational overhead and can fail silently under network partitions or clock drift. For state mutations involving transactions or inventory, PostgreSQL transactions with row-level locks are vastly superior.
By wrapping the idempotency key verification and the domain balance mutation inside a single ACID database transaction, we guarantee that either both succeed or both rollback completely if an unexpected error occurs.
Dead-letter queues that tell the truth
Background queue workers using BullMQ and Redis will inevitably encounter transient network failures when calling third-party APIs. We configure exponential backoff retries with jitter to prevent thunderous herds from overwhelming external systems.
When all retries are exhausted, jobs are moved to a dead-letter queue with the full error stack, request payload, and timestamp. The job status tells you the unvarnished truth, making manual inspection and one-click replay straightforward without losing data.