Idempotency in Payment Webhooks: A Safe Migration

Your payment gateway sends the same webhook twice, and your order service ships the order twice. It hits any team that has wired Razorpay, PayU, Cashfree or Stripe callbacks straight into business logic, usually at the worst possible moment — a festival sale, a retry storm, or a gateway failover nobody told you about. The fix is well understood. Migrating a live system to it without a bad deploy is the part that goes wrong.

Why do payment webhooks get delivered more than once?

Duplicate delivery is normal gateway behaviour, not a bug you can escalate away. Payment providers retry on timeouts, on non-2xx responses, and on slow responses they assume failed; they also fan out the same event to every configured endpoint, and some replay events after incidents or on request.

The uncomfortable part is that your endpoint cannot tell a retry from a new event by looking at it. Gateways do send an event identifier (Razorpay's x-razorpay-event-id, Stripe's id), but many older integrations ignore it because the first version worked fine in testing. Retries typically arrive within minutes, then hours, and a few arrive days later. Any handler that is safe only on first delivery is already unsafe; you just have not been charged for it yet.

What is the minimum correct idempotency design?

A correct design is a persisted dedupe record keyed on the gateway event ID, written in the same transaction as the business change. If the insert fails because the key already exists, the handler exits successfully with a 200 and does nothing else.

The important detail is atomicity. A common half-fix is to check a processed_events table, then do the work — that leaves a race window where two concurrent retries both pass the check. Instead, insert the event row first with a unique constraint on (gateway, event_id), and let the constraint be your lock. If the insert throws a duplicate error, stop.

Three supporting rules keep it honest:

• Return 200 for duplicates, not 4xx. A 4xx tells the gateway the delivery failed and invites more retries.
• Store the raw payload and a hash of it. Same event ID with a different body means something upstream is wrong and you want an alert.
• Make side effects either idempotent or outbox-driven. Sending email, firing a WhatsApp message or calling a courier API should read from an outbox table, not run inline mid-handler.

How do you add deduplication without a big-bang release?

You add it in shadow mode first: write every incoming event to the dedupe table, log what you would have skipped, but change no business behaviour. Run that for a week or two and you get real numbers instead of guesses.

In our experience on Indian gateway traffic, a mid-volume merchant sees roughly 0.3-2% of webhook deliveries as duplicates in a normal month, and that figure can jump several times over during a failed deploy or a gateway incident. Shadow mode tells you your own number, and more usefully it tells you which handlers are silently fragile — for example, an order that got created twice but was caught by a downstream uniqueness check, or one that was not.

Worked example: suppose you process 40,000 webhook deliveries a month and shadow mode flags 600 as duplicates. Ninety percent will be harmless retries of an already-successful write. The remaining 60 are the ones that matter, and reading them is the actual deliverable of phase one — not code.

What order should the rollout follow?

Roll out per handler, smallest blast radius first, with the dedupe check live and the business logic unchanged. Notifications and analytics endpoints go first because a skipped duplicate there costs nothing.

Then move payment-status handlers, which usually only flip an order state and are naturally safe to guard. Order creation and refund initiation go last; those are the ones with money attached and the ones you want to switch over on a quiet weekday morning with someone watching the dashboard.

A workable sequence:

• Week 1: add the dedupe table, write-only logging, no behaviour change.
• Week 2: review shadow data, fix any handler that is not safe to replay.
• Week 3: enable enforcement on notification and analytics endpoints.
• Week 4-5: enable on payment status and refund handlers, one at a time.
• Week 6: enable on order creation, then remove the shadow flags.

Keep the old code path deployable for a fortnight after each step. A feature flag that can revert a single handler is worth more than a perfect design document.

How should you test idempotency before it reaches production?

Test by replaying real captured payloads, not by writing a unit test that calls your handler twice in a loop. The loop test passes on code that will still fail under concurrency and under gateway-signature edge cases.

Capture a set of production payloads (scrubbed of PII), replay them against staging in three patterns: strictly sequential, two at once, and the same event with a mutated body. Assert on final database state — one order, one refund row, one ledger entry — rather than on log lines. Do the same against a genuinely slow endpoint by adding a 30-second delay, because that is exactly the condition that makes gateways retry in the first place.

If you run a queue in front of your handlers, the same tests belong at the queue consumer too. Deduplication at the HTTP layer alone does not protect you when a worker crashes after processing but before acknowledging. For queue and worker topologies that survive this, see our notes on how we approach system architecture.

Which operational habits keep it working after launch?

Keep a monthly report of duplicate counts by endpoint and alert when a single event ID arrives more than five times, because that pattern usually means your endpoint is slow or erroring rather than the gateway being noisy. Prune dedupe rows on a schedule — gateway event IDs are not needed forever, and 90 days covers every retry window we have seen in practice.

Verify signatures before you touch the dedupe table; a forged event should never occupy a key. And log the gateway's timestamp alongside yours, so that when a finance lead asks why a refund fired twice, you can answer in minutes rather than reconstructing the day from nginx logs. If you are running this on your own infrastructure, server and deployment practices matter as much as the handler code — a dedupe table on a single unbacked-up database is its own incident.

Where to start

Start with shadow mode on one endpoint this week: a table, a unique index, and a log line. It costs an afternoon and gives you the data to plan the rest. If you would rather not do the production cutover alone, we at Maysan Labs have migrated payment handlers on live systems before and are happy to talk through the sequencing.