Webhook Retry Best Practices: Backoff, Jitter and Attempt Limits

A single failed HTTP request is not an emergency. What turns it into one is the response: a component that reacts to failure by hammering the destination as fast as it can. Whether the component retrying is a webhook provider delivering to your endpoint, or a forwarder like HookWatch re-sending a captured event to your destination, the same rules decide whether retries heal the failure or amplify it. This guide is about designing a retry policy that recovers deliveries without making an outage worse — and about being honest that retries reduce loss, they do not eliminate it.

Why naive retries are dangerous

The instinct after a failed request is to try again immediately. At the scale of one delivery that is harmless. Across a fleet it is how a brief blip becomes a sustained outage.

• Retry storms. Your destination returns 503 for two seconds during a deploy. Every in-flight delivery fails at once, every sender retries at once, and the destination comes back up straight into a wall of retried traffic plus the new traffic that arrived meanwhile. It falls over again, and the cycle repeats.
• Thundering herd. Even without an outage, if every failed delivery waits the same fixed interval and retries on the same tick, retries arrive synchronised. You have converted a smooth stream into periodic spikes.
• Wasted work on permanent failures. Retrying a request that can never succeed — a malformed payload your handler will always reject — burns capacity and delays the deliveries that could have succeeded.

The goal of a retry policy is to spread load out in time, give a struggling destination room to recover, and stop retrying the moment it is clear retrying cannot help.

Backoff: fixed, linear, exponential

Backoff is the delay a sender waits before the next attempt. The three common shapes differ in how fast that delay grows.

Concretely, with a 30-second base:

Exponential backoff is the default choice for a reason: it responds to a persistent failure by rapidly reducing pressure, while still retrying quickly the first time or two when the failure is most likely transient. In practice most policies also cap the delay (an exponential curve reaches hours quickly), so the real schedule is “exponential up to a maximum, then fixed at that maximum”.

Jitter: why identical schedules are a bug

Backoff decides how long to wait. Jitter decides how much to vary that wait per delivery, and it is what actually breaks up the synchronised spikes.

Imagine a thousand deliveries all fail at 14:00:00 and all use a clean 30-second backoff. Every one retries at 14:00:30 — the destination gets a thousand requests in the same instant, which is exactly the burst that knocked it over. Add a random offset to each delay and those thousand retries smear across a window instead of landing together.

The simplest and most effective form is to pick each delay uniformly from a band around the base, for example the base delay plus or minus twenty percent. That is enough to decorrelate senders that failed at the same moment. Jitter is not a nicety bolted onto backoff; without it, backoff still produces thundering herds at each retry tick.

Attempt limits and dead-letter handling

Retries cannot run forever. Two things bound them:

• Maximum attempts — a hard count. After it, the delivery is done being retried.
• A total time budget (optional) — stop retrying once the event is older than some window, because a stale event may no longer be worth delivering.

The number that is right depends on the event. A few retries over a few minutes covers the overwhelming majority of transient failures — deploys, brief network blips, momentary overload. Pushing attempts into the dozens rarely recovers anything the first handful missed; it mostly delays the moment you admit the delivery has failed.

When attempts are exhausted, the delivery must not simply vanish. Route it to a dead-letter destination — a queue, a table, a status you can query — so a human or a scheduled job can decide what to do. A delivery that quietly stops being retried with no record is an event you have lost without noticing. Dead-lettering turns “lost” into “parked, visible, and recoverable by replay”.

Which failures are retryable

Retrying only helps when the failure might succeed on a later attempt. Retrying a failure that is deterministic just wastes attempts and postpones the dead-letter.

The dividing line is roughly: 5xx, timeouts, and transport errors are the destination’s problem and may pass; 4xx is the request’s problem and will not. Retrying 4xx is the most common way to waste a retry budget — the payload that was rejected as invalid stays invalid no matter how many times you send it.

429 is the important exception inside the 4xx range: it is not “this request is wrong”, it is “this request is fine, just not now”. Retry it, but slowly, and read Retry-After. If it is your own endpoint returning 429, the fix may be on the receiver instead — see handling webhook rate limits.

The Retry-After header

When a destination returns 429 or 503, it may include a Retry-After header telling you when to come back — either a number of seconds or an HTTP date:

HTTP/1.1 429 Too Many Requests
Retry-After: 120
Content-Type: application/json

{ "error": "rate_limited" }

If the header is present, it overrides your computed backoff — the destination knows its own recovery window better than your exponential curve does. Waiting less than it asked for just earns another 429. If the header is absent, fall back to your normal backoff-with-jitter. Either way, cap how long you will honour a very large Retry-After so a misconfigured destination cannot park a delivery for a day.

A retry decision function

Encoding the rules above in one place keeps the policy consistent. Here is a compact decision function that both classifies the outcome and computes the delay:

package retry

import (
	"math/rand"
	"net/http"
	"time"
)

type Outcome struct {
	StatusCode int           // 0 if the request never got a response
	RetryAfter time.Duration // parsed from Retry-After; 0 if absent
	Err        error         // transport error, if any
}

// retryable reports whether this outcome is worth another attempt.
func retryable(o Outcome) bool {
	if o.Err != nil {
		return true // connection refused, reset, DNS, timeout
	}
	switch o.StatusCode {
	case http.StatusRequestTimeout, // 408
		http.StatusTooManyRequests: // 429
		return true
	}
	return o.StatusCode >= 500 // any 5xx
}

// delayFor returns how long to wait before attempt number `attempt`
// (1-based), honouring Retry-After when the destination sent one.
func delayFor(attempt int, o Outcome, base, max time.Duration) time.Duration {
	if o.RetryAfter > 0 {
		if o.RetryAfter > max {
			return max
		}
		return o.RetryAfter
	}
	// Exponential: base * 2^(attempt-1), capped at max.
	backoff := base << (attempt - 1)
	if backoff > max || backoff <= 0 {
		backoff = max
	}
	// Full-band jitter: +/-20% of the computed backoff.
	jitter := time.Duration(rand.Int63n(int64(backoff)/5*2+1)) - backoff/5
	return backoff + jitter
}

The two functions are deliberately separate: retryable answers “should we try again at all”, and delayFor answers “if so, when”. A caller loops while retryable holds and attempt is below the limit, sleeping delayFor between tries, and dead-letters when either condition ends the loop. Because retries can deliver the same event more than once, the handler on the receiving end must be idempotent — see an idempotent handler.

Retries and duplicates

A retry is, by definition, a second delivery of the same event. The nastiest case is the timeout: the destination received the request, processed it, and was slow to respond, so the sender timed out and retried — and now the work runs twice.

This is why a retry policy is only half a reliability story. The other half lives on the receiver: every handler that a retry can reach must tolerate seeing the same event twice and produce the same result. Retries make deliveries more likely to land; idempotency makes it safe for them to land more than once. Ship them together, or retries will trade lost events for duplicated ones.

Observability: you cannot tune what you cannot see

A retry policy you cannot observe is a policy you cannot trust. At minimum, keep per delivery:

• Every attempt with its timestamp, the delay that preceded it, and the outcome (status or transport error).
• The final disposition: delivered, still retrying, or dead-lettered.
• Aggregate signals: retry rate over time, and how many deliveries reach the last attempt.

A rising retry rate is an early warning that a destination is degrading before it fully fails. A spike in dead-lettered deliveries is a page. Without the attempt history you are left guessing whether a failure was a one-off that the next retry fixed or a persistent problem eating your budget. If you would rather not build that recording yourself, HookWatch keeps a numbered attempt history per delivery and can alert on failure spikes through Slack or an outbound webhook.

How HookWatch’s retry policy maps to these ideas

HookWatch retries on the forward leg — when it forwards a captured webhook to your destination — and its per-endpoint policy is a direct expression of the principles above:

A worked schedule for the defaults, with max_attempts = 3:

The backoff array holds a third entry (600) that the default limit of three attempts never reaches; raising max_attempts extends the schedule into it. When attempts are exhausted the delivery is recorded as failed rather than silently dropped, so you can inspect it and, once the destination is healthy, recover it by replaying the captured payload.

Putting it together

A sound retry policy is a short list of decisions applied consistently: exponential backoff with a cap, jitter on every delay, a small attempt limit, retries confined to 5xx/timeouts/transport errors and 429, respect for Retry-After, a dead-letter path for the exhausted, and enough observability to see it working. None of it promises perfect delivery — it promises that transient failures recover on their own and that permanent ones surface quickly instead of hiding in an infinite retry loop. When something does slip through, working the failure the way a systematic debug does tells you which bucket it fell into. Pair the policy with an idempotent receiver and a way to replay what still slips through, and you have covered both the failures retries can fix and the ones they cannot.