How to Debug Failed Webhooks: A Step-by-Step Guide

A webhook fails somewhere on a path with many owners: the provider’s delivery infrastructure, DNS, TLS, your load balancer, your framework’s body parser, your signature check, your handler, your database, your queue. Debugging efficiently means walking that path in order instead of guessing. This guide works from the outside in — the same order a request travels.

What counts as a failed webhook

From the provider’s point of view, a delivery failed if it did not receive a 2xx response within its timeout. That is the whole definition. It covers several very different situations:

• The request never arrived (DNS, network, firewall).
• The request arrived but was rejected before your code ran (TLS, auth, routing).
• Your handler ran and returned an error (4xx or 5xx).
• Your handler ran but took too long, and the provider gave up waiting.

There is also the inverse case, which providers cannot see: your endpoint returned 200 but the event was never actually processed. That failure mode is invisible in provider dashboards and deserves its own treatment — see webhook delivered but not processed.

Keep the two perspectives separate while debugging:

Step 1: Confirm the request left the provider

Before touching your own stack, check the provider’s delivery log (Stripe’s event detail, GitHub’s Recent Deliveries, or the equivalent). Three things matter:

1. Was a delivery attempted at all? If not, the event type may not be subscribed, or the endpoint may be disabled on the provider side after repeated failures.
2. What response did the provider record? A status code, a timeout, or a connection error — each sends you down a different branch below.
3. How many attempts were made? One failure followed by a successful automatic retry needs no action. Exhausted retries do.

Step 2: Rule out DNS, TLS, and connectivity

If the provider recorded a connection error rather than an HTTP status, the request never reached your application. Work through the transport:

• DNS — does the hostname resolve publicly, and to the address you expect? A stale record after a migration is a classic silent killer.
• TLS — an expired or incomplete certificate chain fails the handshake before any HTTP happens. Check with openssl s_client -connect host:443 and confirm the full chain is served; some providers also enforce a minimum TLS version.
• Firewalls and allowlists — if you restrict inbound traffic by IP, confirm the provider’s published egress ranges are current. Providers change them.
• Load balancer / reverse proxy — a 502 or 503 recorded by the provider often means your proxy was up but the upstream application was not.

A quick discriminator: send a request to the endpoint yourself from outside your network. If your curl works but the provider’s deliveries fail, suspect IP filtering or provider-specific TLS requirements rather than your application.

Step 3: Read the status code and response body

Once the provider shows an HTTP status, the code narrows the failure class immediately:

The response body is just as valuable as the code — it is whatever your endpoint returned at the moment of failure: a stack trace, a validation error message, a framework error page. Read it before reading anything else; it frequently names the exact line that broke.

Step 4: A worked example

Here is a realistic failed delivery, the kind you would find in a delivery log. The provider sent an order event:

POST /webhooks/orders HTTP/1.1
Host: api.example.com
Content-Type: application/json
User-Agent: Provider-Webhooks/1.0
X-Provider-Event-Id: evt_9f2c81aa
X-Provider-Signature: t=1753228800,v1=6ffbb59b2300aabb...
Content-Length: 486

{
  "id": "evt_9f2c81aa",
  "type": "order.completed",
  "created": 1753228800,
  "data": {
    "order_id": "ord_20481",
    "amount": 4900,
    "currency": "AUD",
    "customer": null
  }
}

And the response your endpoint returned:

HTTP/1.1 500 Internal Server Error
Content-Type: application/json
Content-Length: 92

{
  "error": "TypeError: Cannot read properties of null (reading 'email')"
}

The diagnosis is all here: the signature header is present, so the request passed the transport and reached your handler. The handler assumed data.customer was always an object, and this event carried null — likely a guest checkout. The fix is a null guard; the recovery is a replay of this exact payload once the fix is deployed.

Step 5: Timeouts

If the provider recorded a timeout, your handler did too much work before responding. Most providers wait somewhere between 5 and 30 seconds — treat the exact budget as provider-specific and assume it is short.

The reliable pattern is acknowledge fast, process later: validate the signature, persist or enqueue the raw event, return 200, and do the real work asynchronously. Anything on the synchronous path — a slow third-party API call, a lock-contended database write, a cold serverless start — eventually causes a timeout under load.

Timeouts are also the most common cause of duplicate processing: the provider gave up waiting, but your handler finished anyway, and the retry delivers the same event again. If that scenario stings, your handler needs idempotency.

Step 6: Request parsing and signature failures

A 400 before your business logic usually means the body never parsed: a Content-Type your framework does not expect, malformed JSON after a proxy modified the body, or a size limit. Compare the raw captured request against what your parser accepts.

A 401/403 usually means signature verification failed. The common causes, in order of likelihood:

1. Verifying a parsed and re-serialised body instead of the raw bytes.
2. The wrong signing secret — especially after a rotation, or a test-mode secret against live events.
3. Encoding mismatches (hex vs base64) or a missing header prefix.
4. Clock skew breaking a timestamp tolerance check.

The full treatment, with working code in three languages, is in how to verify webhook signatures.

Step 7: Application exceptions, databases, and queues

When the handler itself throws, standard debugging applies — but webhooks add a wrinkle: the triggering input arrived from outside and may be gone. Recentre the investigation on the captured payload:

• Application exceptions — match the response body’s error and timestamp against your error tracker. Field-shape assumptions (like the null customer above) top the list; providers evolve payloads.
• Database failures — unique-constraint violations on a duplicate delivery, exhausted connection pools under a webhook burst, and lock timeouts all surface as 5xx. The constraint violation is special: it often means your idempotency layer worked but returned the wrong status; deduplicated events should return 200.
• Queue failures — if the handler enqueues work, a broker outage turns into webhook failures. Decide deliberately whether a failed enqueue should return 5xx (so the provider retries) or be absorbed; silent absorption is how events vanish.

Step 8: Reproduce, then replay

To reproduce a failure, re-send the exact captured request — same body bytes, same relevant headers — against a local or staging instance of your handler. Reconstructed payloads hide the bug precisely when the bug is a payload you did not anticipate.

Once the fix is deployed, replay the failed delivery in production. Replay one delivery first, verify the downstream effect happened exactly once, and only then process the backlog. Replaying re-runs a real event with real side effects, so it has its own safety checklist — see how to replay failed webhooks safely.

Diagnosing intermittent failures

A delivery that fails once an hour is harder than one that fails every time. Patterns worth checking:

• Correlated with load — bursts of events exhausting pools or hitting rate limits. Compare failure timestamps against delivery volume.
• Correlated with deploys — a few seconds of connection refusals on every deploy looks like random flakiness until you overlay deploy times.
• One instance of many — a single bad host behind the balancer fails a fraction of requests. Response headers that identify the instance settle it fast.
• Payload-shaped — only events with certain optional fields fail. Diff the bodies of failed vs successful deliveries of the same event type.

Intermittent failures are where an attempt history earns its keep: a delivery that failed at 14:02 and succeeded on the automatic retry at 14:04 tells you the failure was transient — and the two attempts give you the before/after to compare.

Production debugging checklist

Work down this list in order; each line assumes the ones above it came back clean.

1. Provider attempted delivery, to the URL you expect, for the event type you expect.
2. DNS resolves publicly; TLS handshake completes with a full chain.
3. No firewall or IP allowlist is dropping the provider’s current egress ranges.
4. The response status and body from the failed attempt have been read, not guessed.
5. Signature verification uses the raw body and the correct current secret.
6. The handler responds inside the provider’s timeout; slow work is off the request path.
7. Application logs around the failure timestamp are matched to the delivery by event ID.
8. Database and queue dependencies were healthy at the time of failure.
9. The handler is idempotent, so recovery by retry or replay is safe.
10. The fix is verified against the captured payload, then the delivery is replayed — one first, then the rest.

If you want the capture side of this handled for you, point an endpoint at HookWatch: every delivery keeps its request, response, and attempt history, and failed deliveries can be replayed individually or in bulk once your fix ships.