How to Replay Failed Webhooks Safely

You have found the failed deliveries, deployed the fix, and now you want to push the missed events through. The temptation is to select every failure and press replay. Resist it for one more minute. A replay is not a log line being reprinted — it re-runs a real business event against your live system, with whatever side effects that event carries. Replaying a batch you have not thought about is how a recovery turns into a second incident. This guide is about replaying deliberately.

Replay is not retry

Retry and replay both re-send a delivery, but they are different operations owned by different actors, and conflating them causes mistakes.

Automatic retries are the system’s own recovery: they extend the failed delivery’s attempt history in place, and they only help when the failure was transient. Replay is what you reach for when retries were exhausted or were never going to succeed — the handler had a bug, a secret was wrong, a dependency was down for longer than the retry budget. You fix the cause, then replay.

The distinction matters for auditing. In HookWatch, a retry mutates the original delivery’s state as its attempts progress, whereas a replay never touches the original record — it is stored as a separate replay with its own attempt, pointing back at the delivery it came from. After the dust settles you can still see exactly what failed, when, and what you did about it.

When manual replay is the right tool

Replay earns its place in a specific situation: the event was valid and important, but your side could not process it at the time, and the cause is now fixed. Good candidates:

• A handler bug rejected a class of payloads (the null-field case, an unhandled event type). You shipped a guard and want the rejected events reprocessed.
• A signing secret was wrong or a rotation was mis-timed, so valid events were refused as forgeries. Verification is fixed; the events are real.
• A downstream dependency (database, queue, third-party API) was down past the retry window, so deliveries exhausted their attempts.

Replay is the wrong tool when the failure is still live — replaying into a handler that is still broken just produces the same failure with a fresh timestamp. Fix first, confirm the fix against one captured payload, then replay.

The core risk: duplicate side effects

The captured event does not know it already partly happened. If the original delivery ran halfway before failing — charged a card, then timed out before writing the order — a replay runs the whole handler again. Without protection, that is a second charge.

This is the single most important fact about replay: a replay is safe exactly to the degree your handler is idempotent. An idempotent handler recognises the event’s identifier, sees it has already been processed, and returns success without repeating the effect — so replaying it is a harmless no-op. A non-idempotent handler repeats every side effect on every replay. If you are unsure which yours is, assume it is not idempotent and treat every replay as capable of duplicating whatever the handler does. The durable fix is to make the handler idempotent; see webhook idempotency.

Check the current state of the world first

A failed event is a snapshot of an intent from the past. By the time you replay it, the world may have moved on, and replaying blind can undo or duplicate work that already resolved another way.

• The subscription.updated you are about to replay may have been superseded by a later subscription.cancelled you did process. Replaying the older event could resurrect a cancelled subscription.
• The order the event refers to may have been created manually by support while the webhook path was broken. Replaying now duplicates it.
• The customer or resource the event points at may have been deleted, so the replay will fail anyway — better to know before you fire a batch.

Before replaying, ask what the correct end state is now, not what it was when the event first arrived. Query your own system for the referenced resource and decide whether reprocessing this event still moves you toward the right state.

Idempotency keys make replay boring

The reason to invest in idempotency is that it turns replay from a dangerous operation into a boring one. When your handler keys each event by its stable identifier — the provider’s event ID — and records which identifiers it has already processed, replay behaviour becomes predictable:

• Replaying an event that did complete: recognised as already processed, returns success, no side effect. A safe no-op.
• Replaying an event that did not complete: no record of it, so it processes normally — which is exactly what you wanted.

With idempotency in place you can replay freely, because the handler, not your judgement about which events are safe, is what prevents duplication. Without it, every replay is a manual risk assessment.

Single replay versus batch replay

HookWatch supports both replaying one delivery and replaying many at once, and the difference in blast radius is large.

• Single replay is your probe. Use it to confirm the fix works end to end and that the downstream effect happened once. It is also the right granularity for high-stakes events you want to watch individually.
• Batch replay is for volume once you trust the outcome — dozens or hundreds of low-risk events after you have verified the pattern on one. Scope the batch tightly (a single endpoint, a single event type, a bounded time window) rather than replaying “all failures” indiscriminately.

The safe sequence is always the same: one, verify, then the rest. Batch replaying before you have verified a single delivery multiplies whatever mistake you have not noticed yet.

Payload and headers: what is preserved, what may change

A replay is only trustworthy if it re-sends the event faithfully. HookWatch replays the exact captured payload — the original body bytes, unmodified — so your handler sees the same event it would have seen originally.

Headers are a different matter, and it is worth being precise because handlers often get this wrong. Do not assume a replayed request is byte-identical to the original in its headers. A replay mechanism may legitimately regenerate transport-level headers: a fresh timestamp, updated forwarding headers, and in particular signatures may be recomputed or differ from the original. That has a concrete consequence:

• A handler that verifies the signature over the raw body will still pass, because the body is preserved.
• A handler that rejects requests whose signature timestamp is older than a few minutes may reject a replay of a delivery from an hour ago — the timestamp tolerance check, not the signature itself, is what fails.

Design the handler so it does not assume header byte-identity across a replay. Verify the body, and if you enforce a timestamp freshness window, decide deliberately how it should behave for a legitimately replayed event.

Auditability and authorisation

Replay is a privileged action with real-world effects, so who can do it, and who did, both matter.

• Record who replayed what. Because a replay is a separate record linked to the original delivery, and HookWatch keeps an audit log, you can answer “who reprocessed this event, and when” after the fact. That trail is what lets you reconstruct an incident and prove that a duplicate charge came from a specific replay rather than a provider resend.
• Limit who can replay. Not everyone who can read deliveries should be able to replay payment events into production. Treat replay as a permission, scoped by workspace role, the same way you would gate any action that can move money. The fewer people who can fire a batch replay, the smaller the surface for an expensive mistake.

A pre-replay checklist

Before you replay anything with side effects, walk this list in order. If any step fails, stop.

1. The root cause is fixed and deployed. Replaying into a still-broken handler reproduces the failure.
2. The fix is verified against a captured payload — you have confirmed the exact failing event now processes correctly, ideally in staging.
3. The handler is idempotent for these events, or you have accepted and understood the duplication risk for each one.
4. You have checked current state — the resource still exists and has not been resolved another way that a replay would undo or duplicate.
5. Scope is bounded — one endpoint, one event type, a defined time window. Not “all failures”.
6. You are replaying one delivery first, and you know exactly which downstream effect to check to confirm it ran once.
7. You have verified that single replay end to end before touching the batch.
8. Replay is authorised — you have the permission to do it, and the action will be recorded against you.

Work through it and replay becomes a controlled recovery step rather than a gamble. For the mechanics of finding and diagnosing the failures in the first place, see the debugging walkthrough; for the quick version of this list, there is a short replay checklist too. The recovery itself lives on the replay side of HookWatch — replays are immediate, leave the original delivery untouched, and can even be pointed at an alternative target URL when you want to reprocess into a different destination.