Debugging Webhooks in Production: A Practical Guide

Webhook failures in production are stressful. Unlike synchronous API calls where you get immediate feedback, webhook issues can go unnoticed for hours—leading to lost data, frustrated customers, and late-night debugging sessions.

Common Webhook Failure Patterns

Before diving into debugging, understand the typical failure modes:

Connection Failures

The webhook sender can't reach your server:

• DNS resolution failures: Your domain isn't resolving
• Connection timeouts: Firewall blocking, server unreachable
• SSL/TLS errors: Certificate expired or misconfigured

Response Failures

Your server receives the request but returns an error:

• 5xx errors: Application crashes, database issues, resource exhaustion
• 4xx errors: Authentication failures, validation errors, missing routes
• Timeout: Your handler takes too long to respond

Silent Failures

The trickiest category—your server returns 200 but something's wrong:

• Partial processing: Some operations succeed, others fail
• Queue backlog: Events accepted but never processed
• Data corruption: Malformed payloads causing downstream issues

The Debugging Workflow

Follow this systematic approach when webhooks fail:

Step 1: Check the Sender's Dashboard

Most webhook providers show delivery status:

# Stripe CLI - view recent webhook attempts
stripe events list --limit 10

# Check a specific event's delivery attempts
stripe events retrieve evt_1234567890

Look for:

• HTTP status codes returned
• Response body from your server
• Timing of retries

Step 2: Inspect Your Logs

Add structured logging to your webhook handler:

app.post('/webhook', async (req, res) => {
  const requestId = crypto.randomUUID();

  console.log({
    type: 'webhook_received',
    requestId,
    headers: req.headers,
    bodySize: JSON.stringify(req.body).length,
    timestamp: new Date().toISOString()
  });

  try {
    await processWebhook(req.body);

    console.log({
      type: 'webhook_processed',
      requestId,
      success: true
    });

    res.status(200).json({ received: true });
  } catch (error) {
    console.error({
      type: 'webhook_failed',
      requestId,
      error: error.message,
      stack: error.stack
    });

    res.status(500).json({ error: 'Processing failed' });
  }
});

Step 3: Replay the Webhook

Once you've identified the failing payload, replay it:

# Using curl to replay a saved payload
curl -X POST https://your-api.com/webhook \
  -H "Content-Type: application/json" \
  -H "X-Webhook-Signature: sig_xxx" \
  -d @failed_webhook.json

# Stripe CLI replay
stripe events resend evt_1234567890

Step 4: Trace the Full Request Path

When the issue isn't obvious, trace the entire path:

1. Load balancer logs: Did the request arrive?
2. Application logs: Did your code execute?
3. Database logs: Did queries succeed?
4. External service logs: Did downstream calls complete?

Debugging Tools

Request Inspection

Capture the full request for analysis:

const fs = require('fs');

app.post('/webhook', (req, res) => {
  // Save request for debugging
  const debugData = {
    timestamp: new Date().toISOString(),
    headers: req.headers,
    body: req.body,
    ip: req.ip
  };

  fs.appendFileSync(
    'webhook_debug.log',
    JSON.stringify(debugData) + '\n'
  );

  // Continue processing...
});

Health Check Endpoint

Add a dedicated health check for webhook infrastructure:

app.get('/webhook/health', async (req, res) => {
  const checks = {
    database: await checkDatabase(),
    redis: await checkRedis(),
    memory: process.memoryUsage(),
    uptime: process.uptime()
  };

  const healthy = checks.database && checks.redis;
  res.status(healthy ? 200 : 503).json(checks);
});

How HookWatch Simplifies Debugging

Debugging webhooks without proper tooling is time-consuming. HookWatch provides:

• Full request logging: Every header, body, and response captured
• One-click replay: Resend any webhook with a single click
• Delivery timeline: See every attempt with timing and status
• Payload search: Find webhooks by content, not just ID
• Real-time alerts: Know about failures before your users do

Stop spending hours hunting through logs. Let HookWatch give you complete visibility into your webhook pipeline.