Reconciling mass payouts without tears

The question that decides whether reconciliation is calm or catastrophic is one you answer before the run, not after: "If I send this line twice, does the recipient get paid twice?"

A mass payout is not a single event. It is 5,000 small events that happen to share a button. Each one can succeed, fail, retry, or — the case that ruins your afternoon — succeed on a request your client never saw the response to. The first time you run a large fan-out, you will hit at least one of these. The trap is treating the run as atomic when it is anything but.

Here is the pattern we ship and recommend, in the order the problems actually show up.

The request that times out but went through

You POST a batch of lines, the connection drops at second 28, and you get nothing back. Did the lines land? You don't know. The naive instinct is to re-send. The naive instinct is how one trader gets paid twice and your treasury eats the difference.

This is the entire reason idempotency_key exists, and it is worth being precise about what it is on the halfin API: a snake_case field in the request body, one per payout line — not an Idempotency-Key header, not a run-level token. The only headers a payout request carries are X-API-Key and Content-Type. The idempotency is per line, in the body, because the line is the unit that can be retried in isolation.

{
  "amount": "40.00",
  "currency": "USDT_TRC20",
  "destination": "T...",
  "idempotency_key": "payrun-2026-04-08:affiliate-8831"
}

The key is yours to mint, and the shape matters. Derive it from something stable and meaningful in your own ledger — the run ID plus the recipient ID, as above — not a random UUID you generate at send time. A random key is uniquely useless on retry: a retried request mints a fresh random key, the server sees a new key, and you've defeated the entire mechanism. A deterministic key means "if you've already accepted a line with this exact identity, return that line; don't create a second one." Send the same line ten times after a timeout and you get one payout and ten identical responses.

One discipline that pays for itself: build the keys before you start sending, store them in your run table, and send from that table. Then a crashed batch resumes by re-sending every line — the accepted ones no-op, the unaccepted ones go through, and you never have to reason about "where did I get to."

There are no payout batches as an API object, by the way. A mass payout is a fan-out over single payouts — many POST /api/v1/payouts calls, each independently keyed. That's a feature for reconciliation: every line has its own ID, its own state, and its own webhook. You are never reconciling an opaque blob.

one run · 5,000 independent lines

The state you're reconciling toward is "released," not "sent"

A payout you create does not immediately move money. It enters pending approval, and it is released from the dashboard. This is deliberate: a fat-fingered run of 5,000 lines is a held queue you can inspect, not 5,000 irreversible on-chain sends. Your reconciliation model has to account for the gate. A line sitting in pending-approval is not stuck and is not failed — it is waiting for a human, and your books should show it as exactly that.

So the lifecycle you reconcile against has more rungs than "sent / not sent":

• created — accepted by the API, idempotency key recorded, waiting on approval
• released — approved in the dashboard, now broadcasting
• completed — confirmed on chain, money is gone and credited to the recipient
• failed — it will not complete; the funds did not leave

The only two terminal states worth writing into your ledger as final are completed and failed. Everything else is in-flight, and in-flight is not a number you book.

The webhook is the truth; the API response is a receipt

The response to your POST tells you the line was accepted and gives you its ID. It does not tell you the money arrived — it can't, because at that instant nothing has confirmed on chain. People wire reconciliation to the POST response, mark the line "paid," and then spend a week explaining to finance why "paid" and "actually on chain" disagree.

Truth arrives later, over webhooks. Two events close a payout line:

• payout.completed — the line confirmed on chain. This is your signal to mark the line done in your ledger.
• payout.failed — the line will not complete. This is your signal to mark it for retry or investigation.

Two rules make these reliable, and they're the same rules that make any halfin webhook reliable:

1. Verify the HMAC signature over the raw request bytes before you act on the event. Not over the parsed JSON — key order and whitespace make parsed bytes ambiguous. An unverified payout.completed is an instruction from an attacker to close a line you should keep open.
2. Process by event identity, not payload identity, and make handling idempotent. You will receive the same event more than once; that is a property of at-least-once delivery, not a bug. Marking a line completed twice must be a no-op. (We went deep on the surviving-everything version of this in designing webhooks that survive everything — the same discipline applies here.)

The shape of a correct handler is small: verify signature, look up the line by its payout ID, transition it to a terminal state if it isn't already there, ignore if it is. No arithmetic, no clever diffing. The webhook carries the verdict; your job is to record it once.

The recovery query, because webhooks are a delivery mechanism, not a database

Webhooks are an accelerant, not the source of record. Endpoints go down, deploys drop in-flight deliveries, a bad config blackholes an hour of events. If your books depend on having received every webhook, your books are one outage away from wrong.

So the closing move of every run is a reconciliation pass that does not trust your inbox: walk your run table, and for every line that is not yet terminal, ask the API for its current state. The payout endpoints are the authority; the webhook just got you the news faster. List the run's lines, compare each against what you've recorded, and resolve the deltas:

• Line completed on the API but still open in your ledger → you missed the webhook. Close it.
• Line failed on the API → mint a fresh line with a new idempotency key (a new attempt is a new identity) and re-send. Re-using the failed key just returns you the failed line.
• Line created/released and not yet terminal → it is genuinely still in flight. Leave it open and re-poll on your next pass.

This is the loop that lets you go home: send from a keyed table, let webhooks close most lines in near-real time, then sweep the stragglers against the API. We wrote the full operational version of this — including the SQL-shaped state machine and the back-pressure on the polling sweep — in how to reconcile crypto payments. It is the guide we hand every new integrator before their first run over a thousand lines.

What goes wrong when you skip a step

Every painful mass-payout incident we've watched reduces to one of four missing pieces:

• No per-line idempotency key → a retry double-pays. The fix is a body field, not a heroic deploy.
• Booking off the POST response → "paid" in your ledger, nothing on chain. The POST is a receipt; the webhook is the verdict.
• Acting on unverified webhooks → a forged payout.completed closes a line you still owe. Verify the HMAC on raw bytes, always.
• No reconciliation sweep → one dropped webhook leaves a line open forever, and you find it during an audit instead of during the run.

None of these are exotic. They are the boring four, and getting them right is the difference between a payout run you click and forget and a payout run you spend the next morning untangling.

Operating rule

Mint the keys before you send, send from the keyed table, treat payout.completed / payout.failed as the only truth — after you've verified the signature — and close every run with an API sweep that doesn't trust your inbox. Do that and a 5,000-line run reconciles itself while you watch. Skip any one of them and you'll reconcile it by hand, twice.

R. Adeyemi, halfin payments engineering