Webhook Idempotency: Preventing Duplicate Payments and Actions

The charge.succeeded webhook arrives, your handler charges nothing itself but credits the customer’s wallet, and the balance goes up. The provider’s delivery timed out waiting for your response even though the credit committed, so it retries. The same event lands again, and the wallet is credited twice. Nothing malfunctioned: the network was slow once, and everything downstream did exactly what it was told. Idempotency is the discipline that makes “process this event again” produce the same result as processing it once, so that a normal, expected duplicate cannot turn into a double credit.

Duplicate delivery is normal

Webhook systems are built on at-least-once delivery, not exactly-once. Exactly-once delivery over an unreliable network is, in the general case, impossible — so providers and forwarders choose the safe failure mode: if they are unsure a delivery succeeded, they send it again. That means duplicates are not an anomaly to be stamped out upstream; they are a guaranteed input your handler must expect.

The classic path to a duplicate is the timeout:

1. The provider sends the event.
2. Your handler receives it and does the work — commits the database write.
3. Responding is slow (a GC pause, a slow load balancer, a lock), and the provider’s timeout fires before your 200 comes back.
4. The provider records the delivery as failed and retries.
5. The retry arrives, and a naive handler does the work a second time.

Retries are one source; manual replays are another; a provider resending after a transient error on its side is a third. You cannot prevent duplicates at the source, so you neutralise them at the handler.

The idempotency key: which event is this?

To recognise a repeat, you need a stable identifier that is the same across every delivery of the same logical event and different for genuinely different events. Almost every provider gives you one:

• Stripe puts an event id (evt_...) in the body and repeats it on retries.
• GitHub sends an X-GitHub-Delivery UUID header.
• Most others carry an event ID in the payload or a delivery ID in a header.

Use the provider’s event identifier as your idempotency key. Do not synthesise one by hashing the body — two legitimately distinct events can share a body (two identical refunds a minute apart), and a hash would wrongly collapse them. Do not use a timestamp; it is neither unique nor stable. The provider’s event ID is the contract: same event, same ID, every time.

Uniqueness lives in the database

You can check “have I seen this event?” in application code, but that check is only as reliable as the storage behind it — and application-level checks race. The only place that can enforce “this key is processed at most once” under concurrency is a database uniqueness constraint. It is a single atomic decision made by the one component that serialises writes, so two concurrent deliveries of the same event cannot both win.

Everything else in this guide is built on that constraint. A dedicated table for processed events, keyed uniquely on the idempotency key, is the enforcement point. Application logic decides what to do with the outcome; the constraint decides the outcome.

A processed-events table

Keep deduplication state in its own table rather than overloading a status column on a business table. It stays simple to reason about and to prune.

CREATE TABLE processed_events (
    -- The provider's event id, used verbatim as the idempotency key.
    event_id     TEXT        NOT NULL,
    -- Namespace by source so ids from different providers never collide.
    source       TEXT        NOT NULL,
    status       TEXT        NOT NULL DEFAULT 'processing'
                             CHECK (status IN ('processing', 'done', 'failed')),
    received_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    completed_at TIMESTAMPTZ,
    PRIMARY KEY (source, event_id)
);

-- Supports pruning old keys by age (see retention below).
CREATE INDEX processed_events_received_at_idx
    ON processed_events (received_at);

The composite primary key (source, event_id) is the uniqueness constraint doing the real work. The status column lets you distinguish “a delivery is mid-flight” from “a delivery finished”, which matters for the transaction strategy below.

Transaction boundaries: where correctness is won or lost

There are two sound strategies. Both hinge on making the key-claim and the work agree about whether the event happened.

Strategy A — claim and work in one transaction

Do the deduplication insert and the business write in a single database transaction. If the whole handler’s side effect is a database write, this is the simplest correct design: either both the claim and the effect commit, or neither does.

BEGIN;
    INSERT INTO processed_events (source, event_id, status, completed_at)
    VALUES ('stripe', 'evt_9f2c81aa', 'done', now())
    ON CONFLICT (source, event_id) DO NOTHING;
    -- If that inserted a row, do the business write here in the same tx.
    -- If it conflicted (0 rows), skip the write — already processed.
COMMIT;

Because the claim and the effect share a transaction, a crash between them is impossible: there is no “credited the wallet but forgot to record the event” window. The limitation is scope — it only covers side effects that live in the same database. An external charge (a call to a payment API) cannot be rolled back by ROLLBACK.

Strategy B — claim first, then work, with status tracking

When the side effect is external (charging a card, calling a third-party API, enqueuing a job), you cannot wrap it in the database transaction. Instead, claim the key first, then do the work, then mark it done:

1. INSERT ... ON CONFLICT DO NOTHING with status = 'processing'. If it conflicts, another delivery owns this event — stop.
2. Perform the external side effect.
3. Update the row to status = 'done'.

This introduces a real edge case you must design for: if the process dies between steps 1 and 3, the row is stuck in processing, and a later retry will see the claim and skip — even though the work never finished. Handle it with a rule: a processing row older than a sensible timeout is treated as reclaimable, so a retry after the timeout re-attempts the work. This is why the external side effect itself should also be idempotent where the provider supports it — for example, Stripe’s own Idempotency-Key on the charge request, so re-attempting the charge cannot double it.

Neither strategy is universally right. Strategy A is correct and simple for database-only effects; Strategy B is what you reach for the moment an external system is involved, at the cost of handling the crash-in-the-middle case explicitly.

Race conditions and distributed workers

The reason application-level “SELECT then INSERT” is wrong is the window between the two statements. Two deliveries of the same event, handled by two workers at the same moment, both SELECT (neither sees a row), both decide to process, and both do the work. The uniqueness constraint closes that window because the decision and the write are the same atomic operation:

• Both workers run INSERT ... ON CONFLICT DO NOTHING.
• Exactly one insert succeeds (one row affected); it owns the event and does the work.
• The other conflicts (zero rows affected); it knows another worker has the event and backs off.

This holds no matter how many workers or machines are running, because the database, not the application, arbitrates. You do not need a distributed lock, a queue with exactly-once semantics, or coordination between workers — a unique constraint plus “check rows affected” is the whole mechanism. That property is what makes it safe to scale handlers horizontally.

A production handler in Go

Here is Strategy B wired up with database/sql and PostgreSQL. It claims the key, processes only if the claim was won, and marks the row done — checking rows affected at each decision point rather than trusting a prior read.

package webhook

import (
	"context"
	"database/sql"
	"errors"
	"fmt"
	"net/http"
)

type Handler struct {
	db *sql.DB
}

// errAlreadyProcessed signals that another delivery already owns this event.
var errAlreadyProcessed = errors.New("event already processed")

// claim inserts the idempotency key. It returns errAlreadyProcessed when the
// event has been seen (or is in flight and not yet reclaimable).
func (h *Handler) claim(ctx context.Context, source, eventID string) error {
	// ON CONFLICT DO NOTHING makes the insert a no-op for a known key; the
	// RETURNING clause yields a row only when this call actually inserted one,
	// so a conflict produces sql.ErrNoRows — an unambiguous "someone else has it".
	const q = `
		INSERT INTO processed_events (source, event_id, status)
		VALUES ($1, $2, 'processing')
		ON CONFLICT (source, event_id) DO NOTHING
		RETURNING event_id`
	var got string
	err := h.db.QueryRowContext(ctx, q, source, eventID).Scan(&got)
	if errors.Is(err, sql.ErrNoRows) {
		return errAlreadyProcessed
	}
	return err
}

func (h *Handler) markDone(ctx context.Context, source, eventID string) error {
	const q = `
		UPDATE processed_events
		   SET status = 'done', completed_at = now()
		 WHERE source = $1 AND event_id = $2`
	_, err := h.db.ExecContext(ctx, q, source, eventID)
	return err
}

func (h *Handler) markFailed(ctx context.Context, source, eventID string) {
	// Best-effort: drop the claim so a retry can re-attempt the work.
	const q = `DELETE FROM processed_events
	            WHERE source = $1 AND event_id = $2 AND status = 'processing'`
	_, _ = h.db.ExecContext(ctx, q, source, eventID)
}

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	ctx := r.Context()

	// Verify the signature over the raw body BEFORE trusting the event id.
	event, err := verifyAndParse(r) // returns the event id + typed payload
	if err != nil {
		http.Error(w, "invalid signature", http.StatusUnauthorized)
		return
	}

	switch err := h.claim(ctx, "stripe", event.ID); {
	case errors.Is(err, errAlreadyProcessed):
		// Duplicate (retry or replay). Acknowledge so the sender stops resending.
		w.WriteHeader(http.StatusOK)
		return
	case err != nil:
		// The database is unavailable: return 5xx so the sender retries later.
		http.Error(w, "storage unavailable", http.StatusServiceUnavailable)
		return
	}

	if err := h.process(ctx, event); err != nil {
		h.markFailed(ctx, "stripe", event.ID)
		http.Error(w, "processing failed", http.StatusServiceUnavailable)
		return
	}

	if err := h.markDone(ctx, "stripe", event.ID); err != nil {
		// Work committed but marking done failed. Log loudly; the row is stuck
		// in 'processing' and will be reclaimed by the age rule. See below.
		fmt.Println("mark-done failed:", err)
	}

	w.WriteHeader(http.StatusOK)
}

Three details make this production code rather than a toy. First, it returns 503 (not 200) when the database is down, so a genuine failure is retried rather than silently swallowed. Second, a duplicate returns 200 — the sender must be told to stop, or it will keep retrying an event you have already handled. Third, it deletes the claim on processing failure so a retry can try again, instead of leaving a poisoned processing row that blocks recovery forever. The reclaim-by-age rule from Strategy B is the backstop for the one gap left — a crash after process but before markDone.

How replay behaves against an idempotent handler

This is the payoff. Once the handler above is in place, replaying a delivery is safe by construction:

• Replay an event that completed: the claim hits the existing done row, conflicts, and the handler returns 200 without repeating the side effect. A clean no-op.
• Replay an event that never completed: no row (or a reclaimable processing row), so the claim succeeds and the work runs — exactly the recovery you wanted.

Idempotency is what turns replay from a risky manual operation into a routine one. It is also why the replay safety guide treats an idempotent handler as the precondition for batch replay: the handler, not the operator’s judgement, is what prevents duplication. HookWatch’s replay — single or batch — re-sends the exact captured payload, so an idempotent handler sees a byte-identical event body and the dedupe key matches perfectly.

Key retention and expiry

The processed-events table grows forever if you never prune it, but you cannot prune blindly — deleting a key means the next delivery of that event will be treated as new and reprocessed. The trade-off:

• Retain keys at least as long as the longest window in which a duplicate can arrive: the provider’s maximum retry horizon plus any window in which you might manually replay. If a provider can retry for three days, deleting keys after one day reopens the duplicate hole.
• Retain them not much longer than you need, to keep the table and its index small.

A scheduled job that deletes rows older than the safe horizon (using the received_at index) keeps the table bounded without reintroducing duplicates. Pick the horizon from your providers’ documented retry windows, not a round number that felt safe.

Where this fits

Idempotency is one of three habits that together make webhook processing reliable: sensible retries so transient failures recover, idempotency so those recoveries cannot duplicate, and signature verification so the events you dedupe are genuine. A handler that returns 200 while silently dropping work is a different, quieter failure — covered in delivered but not processed. On the capture side, a delivery record with a full attempt history — like the one HookWatch keeps per delivery — is what lets you confirm, after an incident, that a duplicate delivery genuinely resolved to a single effect rather than two. Build the constraint first; the rest is bookkeeping around it.