Webhook idempotency and replays: build the handler that ships nothing twice

"Did the webhook fire?" is the wrong question. The right one is "if it fires three times, does anything happen three times?"

Every team integrating webhooks asks us the same thing first: how do I make sure I receive the event? That is the easy half. The hard half is the one nobody asks about until a customer gets two shipping notifications, or an account gets credited twice, for a single invoice.paid. Webhooks are an at-least-once delivery channel. Plan for the "least" being more than one.

Why the same event arrives twice

A redelivery is not a bug on our side or yours. It is the normal cost of a system that would rather send an event twice than lose it.

• Your endpoint took longer than the delivery timeout to respond, so we recorded the attempt as failed and retried — even though your handler had already done the work.
• Your endpoint returned a 5xx after the side effect ran but before it returned 200.
• A deploy, a load-balancer hiccup, or a brief network partition dropped the response on the floor.
• You replayed the event yourself from the dashboard while debugging, and the original delivery also landed.

In all four cases the event id is the same. The payload is byte-for-byte the same. The only thing that changed is that your handler ran twice. Whether that is harmless or catastrophic is entirely a property of your handler, not ours.

delivery · at-least-once

Three lines of defense, in order

There is a temptation to solve this with one clever trick. Don't. Use three layers, each cheap, each catching what the previous one let through.

1. Verify the signature before you trust a single byte

Every halfin delivery carries an X-Halfin-Signature header. Compute the HMAC over the raw request body — the exact bytes on the wire — and compare it, in constant time, against the header. Do this before you parse the JSON, before you read the event id, before you touch your database.

The order matters. If you parse first and verify second, you have already let an unauthenticated payload reach your JSON decoder. Worse, re-serializing the parsed object to "rebuild" the signed bytes will fail the moment key order or whitespace differs from what we signed. Sign the bytes, verify the bytes. The signing secret that backs this is your webhook secret — keep it server-side, rotate it like any other credential, and never log the raw value. The full mechanics, including constant-time comparison and the timestamp window, live in the verify-a-webhook-signature guide. If you only read one thing before going live, read that.

A handler that dedupes perfectly but skips signature verification will faithfully process a forged event exactly once. Idempotency without authentication is a well-organized way to get robbed.

2. Dedupe on the event id, not the business id

Once the signature checks out, look at the envelope's event id. Each delivery — including a redelivery of the same business event — is identified so you can recognize a repeat. Keep a table of event ids you have already fully processed. If the id is already there, return 200 and stop. You are done. That is the whole point of the layer.

The classic mistake is deduping on the underlying business object instead:

// WRONG — collapses distinct events on the same invoice
if (await seen(payload.invoice_id)) return ok();

// RIGHT — one row per delivered event
if (await seen(event.id)) return ok();
await markSeen(event.id);

Deduping on invoice_id looks correct until a single invoice legitimately emits more than one event you care about — invoice.underpaid followed later by invoice.paid, or invoice.paid followed by invoice.deposit_reversed. Collapse those onto the invoice id and you will silently drop the second real event while congratulating yourself for being idempotent. The event id is the deduplication key. The business id is not. (This is the same discipline as the idempotency key you send on writes back to us — recognize the operation, not the resource.)

3. Make the side effect itself idempotent

This is the layer people skip, and it is the one that actually saves you. Layers one and two reduce duplicate processing; they do not eliminate the race. Two copies of the same delivery can arrive close enough together that both pass the "have I seen this id?" check before either writes the row. Under enough traffic, this will happen.

So make the effect safe even if it runs twice:

• Crediting an account? Write the credit keyed by event id with a unique constraint, and let the second insert fail the constraint instead of adding a second balance row.
• Fulfilling an order? Transition state by a guard — WHERE status = 'awaiting_payment' — so the second run updates zero rows.
• Sending an email or a downstream notification? Stamp it with the event id and check before sending; a duplicate send is the one effect you cannot claw back.

The mental test we use: if I take any event delivery and replay it ten times, the observable end state must be identical to replaying it once. If that holds, redelivery is no longer a hazard. It is a non-event.

effect · exactly-once outcome

Respond fast, work async

A subtle source of duplicates is your own latency. If your handler does the signature check, then the dedupe lookup, then a slow third-party call, then the database write, all before returning 200, you widen the window in which we time out and retry. Acknowledge fast: verify, dedupe-check, enqueue, return 200. Do the slow fulfilment on a worker that is itself keyed by event id. The webhook endpoint's job is to accept the fact, not to finish the work.

This also keeps you honest about which events you act on. The events we deliver are concrete and named — invoice.confirming, invoice.paid, invoice.overpaid, invoice.underpaid, invoice.expired, invoice.late_deposit, invoice.deposit_reversed, balance.credited, payout.completed, payout.failed. There is no generic "something happened" event, and there is no invoice.activated. Switch on the event type explicitly and ignore the ones you don't handle; an unknown type should be a logged 200, never a 500 that triggers a retry storm. The full list and envelope shape are on the webhooks product page.

A test that actually finds the bug

The hardest duplicate-delivery bug is the one your test suite is green on, because most suites send each event exactly once. Ours doesn't. For every event we emit in a test, we run an "evil twin": deliver it, then deliver the identical bytes again, then deliver it a third time interleaved with the first still in flight. If the end state after three deliveries differs from the end state after one, the integration is not done — no matter how clean the happy path looks.

You can run the same drill against your own endpoint before launch. Capture one real signed delivery, then curl the exact bytes at your URL five times in a tight loop. Count the side effects. The number you want is one.

The short version

• Webhooks are at-least-once. A redelivery is normal, not an incident.
• Verify the HMAC over raw bytes first — authentication gates everything else.
• Dedupe on the event id, never the invoice or payout id.
• Make the side effect idempotent anyway, because two deliveries can race past the dedupe check.
• Acknowledge fast, fulfil on a worker, switch on real event types only.

Get these in order and a redelivered webhook does exactly what it should: nothing you can see. For the verification half of this — the part that has to be airtight before any of the dedupe logic earns its keep — read verify the webhook signature before acting.

R. Adeyemi, halfin payments engineering