Monitoring Webhooks at Scale: Lessons from Processing Millions

At HookWatch, we process millions of webhooks every day. Here's what we've learned about monitoring webhooks at scale.

Key Metrics to Track

Delivery Rate

Track the percentage of webhooks successfully delivered on the first attempt:

• Healthy: > 95% first-attempt success
• Warning: 90-95% first-attempt success
• Critical: < 90% first-attempt success

Latency

Monitor how long webhook processing takes:

• Time to first byte: Network latency
• Total processing time: End-to-end duration
• Queue depth: Backlog of unprocessed webhooks

Error Rates by Type

Categorize failures to identify patterns:

• Connection errors (server down)
• Timeout errors (slow processing)
• 4xx errors (client issues)
• 5xx errors (server issues)

Alerting Strategy

Immediate Alerts

• Endpoint returning 5xx for > 5 minutes
• Delivery rate drops below 80%
• Queue depth exceeds threshold

Daily Digests

• Failed webhooks summary
• Retry statistics
• Latency trends

Dashboards We Use

Real-time Dashboard

• Live event stream
• Current queue depth
• Last 5 minutes success rate

Historical Dashboard

• 7-day delivery trends
• Error breakdown by type
• Top failing endpoints

Debugging Failed Webhooks

When things go wrong, you need:

1. Full request/response logs: See exactly what was sent and received
2. Retry history: Track all delivery attempts
3. Payload inspection: View the webhook body
4. One-click replay: Resend failed webhooks

Scaling Tips

Horizontal Scaling

• Use multiple webhook receivers behind a load balancer
• Process webhooks asynchronously with a queue
• Implement proper health checks

Database Optimization

• Index by endpoint ID and timestamp
• Partition tables by date
• Archive old data regularly

Caching

• Cache endpoint configurations
• Cache signature verification results
• Use Redis for rate limiting

Building a reliable webhook infrastructure at scale requires careful monitoring and quick debugging tools. HookWatch provides all of this out of the box.