Build vs Buy: Should You Build a Webhook Retry System?

Receiving a webhook looks like a solved problem for about a week. You write a route that accepts a POST, parses the body, does the work, and returns 200. When the occasional delivery fails you add a retry loop. Both fit on one screen. The trouble is that neither is the system you actually end up needing — they are the visible tip of a system that grows every time a payload arrives in a shape you did not plan for. This guide is about deciding, honestly, whether that system is worth building yourself.

There is no universally correct answer. A team with an existing platform group and a data-residency requirement should probably build. A three-person startup integrating one payment provider should probably not. Most of the value is in knowing which situation you are in before you commit, so the sections below are organised as: what looks simple, what accumulates, when each choice is right, and a decision matrix you can hold against your own constraints.

What looks simple at the start

The minimum viable receiver really is small:

app.post('/webhooks', async (req, res) => {
  const event = req.body;
  await handle(event);
  res.sendStatus(200);
});

Add a retry and it is still small:

async function withRetry(fn, attempts = 3) {
  for (let i = 0; i < attempts; i++) {
    try {
      return await fn();
    } catch (err) {
      if (i === attempts - 1) throw err;
      await sleep(1000 * 2 ** i);
    }
  }
}

If your integration is one low-stakes internal feed and a failed event can simply be re-triggered by hand, you may be done here — and that is a legitimate place to stop. The rest of this guide is about what happens when it is not.

What accumulates

Each item below starts as a small addition and becomes a component you maintain. None of them is hard on its own. The cost is that you need most of them at once, they interact, and they never stop needing attention.

Event persistence

The retry loop above holds the event in memory. If the process restarts mid-retry — a deploy, an out-of-memory kill, a crash — the event is gone, and the provider has already recorded a 2xx or exhausted its own retries. So the first real requirement is durable storage: write the raw event before you acknowledge it, so a restart can pick it back up. Now you own a table, a schema, and a migration path for it.

Retries and scheduling

An in-process loop that sleeps between attempts blocks a worker and dies with the process. A real retry system needs the schedule to live in durable storage — a next_attempt_at timestamp and a worker that polls for due rows. That means exponential backoff so you do not hammer a struggling destination, jitter so a thousand deliveries that failed at once do not all retry at the same instant, and a cap on attempts so a permanently broken destination does not retry forever. It also means distinguishing retryable failures (a timeout, a 5xx, a dropped connection) from permanent ones (a 4xx — retrying a validation error just wastes attempts). The reasoning behind each of those choices is its own topic; see webhook retry best practices.

Distributed locking

The moment you run more than one worker — and you will, for throughput and for surviving a single instance dying — two of them can pick up the same due event. Now you need a way for exactly one worker to claim a row: a SELECT ... FOR UPDATE SKIP LOCKED, an advisory lock, or a queue with visibility timeouts. Get it subtly wrong and the same event forwards twice under load, which is precisely when it is hardest to notice.

Idempotency

Retries, at-least-once delivery from the provider, and the double-forward risk above all converge on the same requirement: processing the same event twice must not double its effect. That means a dedupe key (usually the provider’s event ID), a uniqueness constraint, and a decision about what a duplicate returns. This is the single most common source of “we charged them twice” incidents, and it is easy to implement in a way that looks right and races under concurrency. The full treatment is in webhook idempotency.

Replay tooling

Sooner or later a bug means a batch of events was accepted but processed incorrectly, and you need to run them again after the fix ships. If you only stored a “processed / failed” flag, you cannot — you need the original payload bytes, and you need an operator-facing way to re-send them: one at a time to verify, then in bulk. Building replay after the fact, during the incident that made you want it, is the worst time to build it.

Tenant isolation

If you serve multiple customers, every one of the components above gains a tenant dimension. Events must be scoped so one customer can never see or replay another’s payloads. Retry budgets and rate limits should be per-tenant so one noisy tenant does not starve the others. Access to the operator tooling has to respect tenant boundaries too. Isolation bugs in a webhook store are data-leak bugs, because the payloads frequently contain other people’s personal information.

Secret storage

Verifying inbound signatures means holding a signing secret per source, and often per tenant. Those secrets should be encrypted at rest, rotatable without downtime (which means accepting the old and new secret during an overlap window), and never logged. Rotation-without-downtime in particular is a feature people discover they need the first time a secret leaks and they cannot rotate it without dropping live traffic.

Observability

When a delivery fails at 3am, the on-call engineer needs to answer “what arrived, what did we send back, and how many times did we try” without shipping a debug build. That means the stored request and response, an attempt history, and a way to search it. Providers’ own dashboards truncate bodies and retain limited history, so the copy that matters is the one on your side. Wiring metrics, logs, and a searchable delivery view is a project in itself.

Retention and storage growth

Webhook payloads are small individually and enormous in aggregate. A busy endpoint can store millions of rows a month, each with a full request and response body. Left alone, that table grows without bound, slows every query, and eventually becomes its own operational problem. So you need a retention policy, a job that enforces it, and a deliberate decision about how long payloads live — which is also a compliance decision (below).

Compliance considerations

Webhook payloads routinely carry personal data: email addresses, names, order details, sometimes more. Storing them makes your webhook system part of the scope of whatever privacy obligations you operate under. That raises general questions worth answering deliberately — how long payloads are retained, whether sensitive fields should be redacted before storage, who can access them, and how a deletion request reaches them. These are considerations, not a checklist, and the right answers depend on your jurisdiction and the data you handle; the point is that building the store means owning them.

Operational burden

Everything above has to be run, not just written. That is on-call rotation for a component that fails at the provider’s schedule rather than yours, database migrations on a hot table, dependency upgrades, capacity planning for spikes, and the slow accretion of edge cases that each seemed like a one-line fix. The build cost is a one-off; the operational cost is forever, and it is the part teams consistently underestimate because it does not show up in the initial estimate.

When building makes sense

Building is the right call more often than a vendor’s marketing suggests. Concrete cases where an in-house system is genuinely the better choice:

• Unusual requirements a product cannot express. If your retry policy, routing, or transformation logic is core to your business and does not fit a configurable product, building buys you exactly the behaviour you need.
• You already run the infrastructure. A team with a mature platform group, an existing durable queue, a secrets manager, and an on-call culture is adding a component to a machine that already exists — the marginal cost is far lower than for a team starting from zero.
• Data-residency or isolation constraints. If payloads legally or contractually cannot leave your infrastructure — or cannot pass through a third party at all — an in-house system may be the only option that satisfies the requirement.
• One internal integration with low stakes. A single feed between two of your own services, where a lost event is a shrug and a manual re-trigger, does not justify adopting anything. The 20-line receiver is the correct engineering.
• Cost sensitivity at high, predictable volume. At very large and steady volume, usage-priced products can cost more than the amortised infrastructure and engineering to run your own — if you already have the engineering. Model it honestly, including the on-call and maintenance time, before assuming either direction is cheaper.

When a managed product makes sense

The mirror image is equally concrete. A managed webhook receiver tends to win when:

• The webhook is not your product. You integrate Stripe, GitHub, or Shopify to support something else you are actually building, and every hour spent on retry scheduling is an hour not spent on that.
• You need it working this week. Adopting a product is measured in hours to a first reliable delivery; building the equivalent is measured in weeks before it is trustworthy, and longer before it has replay and observability.
• The reliability bar is high and the team is small. Backoff, jitter, dedupe, isolation, and secret rotation are all easy to get subtly wrong. A product has already made those mistakes and fixed them.
• You want the debugging and recovery tooling included. Full request/response capture, an attempt history, failure analysis, incident grouping, and safe replay are a large fraction of the total system, and they are the parts most often deferred in a build until an incident forces the issue.

A decision matrix

Neither column is a winner. Read each row against your own constraints and count which side you land on more often — and weight the rows that matter to you, rather than tallying them equally.

A useful tie-breaker: the visible receiver — the POST handler and the retry loop — is perhaps a tenth of the finished system. Most of the cost is in persistence, isolation, idempotency, replay, observability, and the operational burden of running all of it. Estimate the whole system, not the tip, and compare that against the integration cost of a product.

A middle path

Build versus buy is not always binary. Two common blends:

• Buy the ingestion and recovery layer, own the business logic. Let a product handle capture, retries, isolation, and replay; keep your handler — the code that actually acts on an event — entirely yours. This is the most common split, because the handler is the part that is genuinely specific to you and the plumbing is the part that is not.
• Build now, with an exit in mind. If you build, keep the boundary between ingestion and handling clean, so that swapping the ingestion layer later does not mean rewriting your business logic. The teams that regret building are usually the ones whose handler grew tangled into the plumbing.

If you land on the buy side — or want to compare a concrete managed option against the in-house estimate you just made — the companion page HookWatch vs building in-house walks the same trade-offs against a specific product. HookWatch records every delivery’s full request, response, and attempt history, retries retryable failures with jittered backoff, and lets you replay the exact captured payload once a fix ships, which covers the accumulating middle of this guide without you building it. That is the point of the exercise, in both directions: decide with the whole system in view, not the first screenful.