Webhook reliability: questions to ask any vendor

A webhook that quietly stops arriving is worse than no webhook at all, because your team keeps trusting the data. The damage shows up as orders that never reach the ERP, leads stuck in a spreadsheet, or a payment sitting at "pending" for three days — and it lands first on the ops and finance people who reconcile by hand, long before the CTO hears about it. Whether a vendor is genuinely dependable usually comes down to three unglamorous things: retries, dead letters and replay. Here is what to ask before you sign.

What does "reliable" mean beyond "we send webhooks"?

Reliability means the vendor assumes delivery will fail and has a documented plan for those failures. In practice that means at-least-once delivery with retries, an HMAC signature on each request so you can verify the source, a unique event ID you can deduplicate on, and a stated timeout — typically 5 to 10 seconds — after which an attempt counts as failed. Ask what happens on a 500 versus a 410: a well-built sender stops retrying permanently on a 410 and backs off on a 500. Also ask whether delivery is ordered. Most systems do not guarantee it, so your handler must tolerate an "updated" event arriving before "created", keyed on the event timestamp.

What retry schedule should you actually ask for?

Ask for a documented schedule that spans at least 24 hours, with exponential backoff and jitter — not "we retry a few times". Plenty of vendors retry three times inside 15 minutes, which survives a pod restart but not a firewall change on a Friday evening or a payment gateway outage on a Sunday. A defensible schedule looks like 30 seconds, 2 minutes, 10 minutes, 1 hour, 6 hours and 24 hours: six attempts across a day.

Worked example. Take a mid-sized B2B order flow at 40,000 webhook events a month, with 1 percent failing on first attempt because of deploys, timeouts and transient errors. That is 400 events. A sender that gives up after 15 minutes might recover 250 of them and drop 150 without telling anyone — about five lost order-status updates a day. The same sender on a 24-hour schedule recovers nearly all of them. The difference is one line in the contract. If you are building the receiving side yourself, the queueing and reconciliation design matters as much as the sender; our notes on integration architecture cover that pattern.

Why is a dead-letter queue more important than retries?

Because a dead-letter queue turns "we don't know what we lost" into "here is the list". A DLQ is where the sender parks events that exhausted every retry, and the quality of that queue separates an accountable vendor from an apologetic one. Ask four things: can you see the dead-lettered payload along with the exact HTTP response body, is there a retention window (30 days is common, 90 is better), do you get an alert or a daily digest when the count is non-zero, and can you export the whole queue as CSV or JSON. Without those, your only recovery path is emailing support and hoping someone reads it before month-end close.

How does replay work, and what should you be able to demand?

Replay should let you re-deliver events by event ID or time range, to the same or a new endpoint, without the vendor recreating the data by hand. Demand self-serve replay from the dashboard rather than a support ticket — during an incident, a four-hour ticket turnaround is the whole outage. Ask about the lookback window (7 days is a common floor, 30 is comfortable), rate limits on replay (a cap around 100 events per second is normal), whether it re-sends everything in a window or only failed events, and whether replays land in an audit log. Your handler must stay idempotent, because replay will duplicate whatever it touches.

What questions should go into your evaluation?

Put these to the vendor in writing, and score the answers instead of the demo:

• What is the exact retry schedule, in seconds and hours, and how many attempts in total?
• Is there a dead-letter queue, what is its retention, and how are we alerted?
• Can we replay by event ID and by time range, ourselves, from the dashboard?
• What is the per-endpoint timeout, and do you auto-disable endpoints that fail repeatedly?
• Do you sign payloads, with which algorithm, and how are signing secrets rotated?
• Do you guarantee at-least-once or at-most-once, and is ordering preserved?
• What is the documented throughput ceiling per endpoint, and what happens above it?
• How long are event logs kept for debugging?

Check the answers against your own volume, not the demo's. And if you run the sending side in-house rather than buying it, your server setup becomes part of the reliability story — retry workers and a DLQ table need somewhere to live.

Where to start

Pick the one integration you cannot afford to lose — usually payment status or order sync — and ask its vendor for the retry schedule, dead-letter policy and replay window in writing this week. If the answers are vague, treat that as a design constraint and build a nightly reconciliation job against the vendor's list API; it will catch more than any promise. We build webhook-heavy platforms at Maysan Labs, and this is the standard we hold vendors to.