Webhook Delivered but Not Processed: Common Causes

The provider dashboard is all green. Every delivery shows 200. And yet the customer’s subscription is still inactive, the order never shipped, the balance never updated. This is the failure mode that provider dashboards cannot see: the request was delivered and acknowledged, but the event was never processed. Nothing retries, because from the sender’s side nothing went wrong. This guide walks the gap between “accepted” and “handled”, where events die inside it, and how to trace one through your pipeline.

Delivery is not processing

A 2xx response answers exactly one question: did the HTTP request reach your endpoint and get accepted? It says nothing about whether the business effect occurred. Those are two different systems with two different failure modes, and conflating them is why these incidents go unnoticed for days.

The last row is the trap. Once you return 200, the provider considers the event done and will never send it again. Every failure after that byte is yours to detect and recover. The mirror image of this problem — deliveries the provider records as failed — is covered in how to debug failed webhooks; this guide is about the ones it records as succeeded.

The gap opens when you acknowledge early

The advice to “acknowledge fast, process later” is correct — synchronous processing causes timeouts, and returning 200 quickly then doing the work asynchronously is the right pattern. But it is also precisely what creates this failure class. The instant you return 200 before the work is done, you have told the provider “I’ve got this” while the work is still ahead of you. Everything that can go wrong between that 200 and the committed side effect now goes wrong silently, because retries are off the table.

That trade is worth making — you cannot hold a provider’s connection open for a slow job. The point is to make it deliberately: know that acknowledging early transfers responsibility for durability from the provider to you, and build the detection and recovery that responsibility implies. The sections below are the places the handoff leaks.

Where events die after the 200

The queue publish failed after you replied

The classic ordering bug: return 200, then publish to the queue. If the broker is briefly unavailable, the publish throws — but the response already went out. The event is gone, and the provider thinks it succeeded. Publish before you acknowledge, and only return 200 once the event is durably persisted or enqueued:

@app.post("/webhooks")
def webhooks():
    event = verify_and_parse(request)
    try:
        queue.publish(event)          # durable before we acknowledge
    except QueueError:
        return "", 503                # let the provider retry
    return "", 200                    # only now is it safe to acknowledge

Returning 503 here is the right call: the enqueue genuinely failed, so a provider retry is exactly what you want. Absorbing the error and returning 200 anyway is how events vanish.

The background worker crashed mid-job

The event made it onto the queue, a worker picked it up, and the process died — OOM killed, deployed over, panicked — before the job committed. Whether the event survives depends entirely on your queue’s acknowledgement model. If the worker acks the message before doing the work, a crash loses it. Ack after the side effect commits, so a crash returns the message for redelivery. Pair at-least-once redelivery with an idempotent handler and a crash costs you a retry, not an event.

A transaction rolled back after acknowledgement

The handler ran, logged “processed order ord_20481”, returned 200 — and then the surrounding database transaction rolled back on a deadlock or constraint violation. The log line lies: it recorded intent, not commitment. If you emit the success log or the ack before COMMIT returns, a rollback leaves you believing an event was processed that was not. Confirm the commit succeeded before you record success anywhere.

A catch-all swallowed the exception

try:
    process(event)
except Exception:
    pass  # <- events die here, in total silence

A bare except that logs nothing turns every downstream failure into a 200 with no trace. Its milder cousin logs at debug in a system that ships info and up, which is the same thing in practice. If a handler can fail, the failure must be loud: logged with the event ID at a level you actually collect, and surfaced so someone notices. Silent absorption is the difference between an incident you fix in an hour and one you discover in a customer complaint.

The event type was silently ignored

Routing on event type with no default case drops anything unexpected:

switch (event.type) {
	case 'invoice.paid':
		return handlePaid(event);
	case 'invoice.payment_failed':
		return handleFailed(event);
	// a new event.type falls through here and is silently discarded
}

When the provider adds an event type, or you subscribe to one nobody wired up, it lands in the void. Make the default explicit: log the unhandled type with its event ID at a visible level, so “we don’t handle this yet” is a decision you can see rather than a hole you cannot. The same applies to filtering — if you drop events by type or condition, log what you dropped and why.

The payload changed shape

Providers evolve payloads: a field goes from string to object, an optional field appears, an enum gains a value. A parser that assumed the old shape throws deep in processing — after the 200, inside a worker, where no one is watching. The guest-checkout null that breaks data.customer.email is the archetype. Treat inbound payloads as untrusted external input: validate against a schema at the edge of processing, and when validation fails, route the event to a dead-letter store instead of dropping it.

Duplicate detection dropped a real event

Idempotency and deduplication are necessary — the provider will redeliver, and your handler must not double-process. But over-aggressive dedup causes the opposite failure: keying on something too coarse (a customer ID, a timestamp truncated to the minute) makes two genuinely distinct events look identical, and the second gets silently discarded as a “duplicate”. Deduplicate on the provider’s unique event ID, nothing coarser, and log every drop so a suppressed duplicate is visible rather than assumed. The webhook idempotency guide covers how to pick the key.

You cannot debug what you cannot see

Every cause above shares one root: the event failed somewhere with no record that it should have succeeded. The fix is not any single code change — it is observability across the whole path. You need to be able to answer, for any given provider event, “how far did it get?” That means a durable record at each hop:

• The delivery was received and passed verification.
• It was enqueued (or persisted) — with the event ID.
• A worker picked it up.
• The side effect committed — with the event ID again.

If any hop is missing for an event, you have located the failure to a single stage. If no hop records anything, you have found the real problem: you are flying blind, and the first task is instrumentation, not a fix.

Correlation IDs and the provider event ID

Two identifiers make the path traceable end to end. The provider’s event ID (evt_9f2c81aa, msg_..., the GitHub delivery GUID) is stable across the provider’s own retries and lets you join your logs back to the provider dashboard. A correlation ID you assign at the edge threads the event through your internal systems — the HTTP handler, the queue message, the worker, the downstream calls. Log both at every hop:

const correlationId = crypto.randomUUID();
logger.info('webhook.received', { eventId: event.id, correlationId, type: event.type });

await queue.publish({ ...event, correlationId }); // carry it onto the message

// later, in the worker:
logger.info('webhook.processed', { eventId, correlationId, orderId });

Now one query on a correlation ID returns the event’s entire journey, and one query on the provider’s event ID reconciles your system against theirs. Without them, “did we process evt_9f2c81aa?” is an archaeology project.

Walking one event through the pipeline

When something delivered but did not process, do not theorise — trace one specific event through every stage in order, and stop at the first hop with no record.

1. Find the delivery. Get the provider’s event ID from the dashboard and confirm the provider recorded a 2xx. If it did not, this is a delivery failure, not a processing one — switch to the debugging guide.
2. Confirm receipt. Search your logs for that event ID at the edge. No entry means it never reached your code despite the 200 — suspect a proxy, a load balancer returning 200 itself, or a health-check path shadowing the route.
3. Confirm durability. Was it enqueued or persisted? A receipt log but no enqueue log points at a publish-after-acknowledge bug or a broker outage.
4. Confirm pickup. Did a worker claim it? Enqueued but never picked up means a stalled consumer, a dead worker pool, or a poison message blocking the queue head.
5. Confirm commit. Did the side effect commit? Picked up but no commit is a swallowed exception, a rollback, a payload-shape crash, or a downstream failure — read the worker logs around that correlation ID.
6. Confirm exactly-once. If it committed, did it commit once? A dedup mistake or a double-consume shows here.

Each step assumes the ones above came back clean. Whichever step first has no record is where the event died, and the cause list above tells you what usually lives at that stage.

Closing the gap

Detecting these failures needs a record of the inbound event that outlives your own process. A HookWatch endpoint keeps the full request — headers, body, and the provider’s event ID — for every delivery, so “did evt_9f2c81aa arrive, and what exactly was in it?” is answerable even when your own logs came up empty at step two. When failures cluster on one endpoint, HookWatch groups them into an incident so a pattern is visible rather than scattered across days. And once you have fixed the handler, you can replay the exact captured payload to reprocess what was lost — see replaying failed webhooks safely and, for the handler property that makes replay safe, the retry best practices guide. The green 200 is a receipt for delivery, not a guarantee of processing — build the record and the recovery that the gap between them requires.