Most payout integrations are written against the happy path and discover the rest in production. A transfer API is easy to call and genuinely hard to operate, because money moves asynchronously and the network underneath can be slow, partial or temporarily unavailable. These four areas are worth settling before the first live transaction.
Idempotency is not optional
Any request that moves money must carry a key that you generate and the provider stores. If a request times out, you cannot know whether it was received — and without a key, your only options are to retry and risk paying twice, or not retry and risk not paying at all.
- Derive the key from something stable in your own domain: an invoice ID, or a payroll run plus the beneficiary.
- Generate it before the first attempt, not at call time — a random value regenerated on retry defeats the entire mechanism.
- Check the provider's validity window is longer than your longest retry schedule.
- Persist it with the record; it is how you ask what happened to a request whose response you never saw.
- Give a corrected, genuinely different payment a new key, or the provider will return the original result and your correction will silently do nothing.
What the provider does with the key
Understanding the server side makes the client side obvious. The provider stores the key with the result of the first request. A second request with the same key does not execute — it returns what the first one produced.
| Scenario | What the provider does | What you should do |
|---|---|---|
| First request, succeeds | Executes, stores the result | Record the outcome |
| Retry after a timeout | Returns the stored result | Treat it as the original outcome, not a new one |
| Retry after a genuine failure | Returns the stored failure | Do not assume a retry will fix it — read the reason |
| Same key, different payload | Usually rejects as a conflict | This is a bug in your code; alert on it |
| Retry past the window | Treats it as a new request | This is how duplicate payments happen |
The last row is the one that catches people. An idempotency window is finite, and a retry schedule that outlives it converts your safety mechanism into a second payment.
Treat the status lifecycle as a state machine
A payout is not a boolean. It is typically created, queued, processed, then settled, failed or returned — and a transfer can fail after apparently succeeding, when the beneficiary bank rejects it and sends the funds back.
Model those states explicitly rather than storing a flag, and write down which transitions are legal. A reversal arriving for a transfer you already marked complete is normal. Code that cannot represent it will either crash or silently lose the event.
Guarding transitions
Because events arrive out of order, every transition needs to be checked against the state you currently hold rather than applied blindly. The rule is simple: an update that would move the record backwards is discarded, not applied.
- Define a rank for each state, so 'later' is a comparison rather than a judgement.
- On any update, compare the incoming state's rank with the stored one.
- Apply it only if it moves forward, or if it is one of the explicitly allowed reversals like settled to returned.
- Log the discarded ones rather than dropping them silently — a flood of out-of-order events is a signal worth seeing.
- Never let two concurrent updates read-modify-write the same record without a lock or a conditional update.
Verify webhooks, and make them replayable
Webhooks are the only way to learn about events that happen minutes or hours after your request returns. Two rules matter above the rest.
Verify the signature before acting
An unverified webhook endpoint is an instruction to change payment state that anyone who finds the URL can send. Verify first, act second, with no exceptions for testing.
- Compute the signature over the raw request body, before any JSON parsing — re-serialising changes the bytes and breaks the comparison.
- Use a constant-time comparison, not string equality.
- Reject anything whose timestamp is outside a tolerance window, which closes off replay.
- Keep the signing secret out of the codebase and rotate it on a schedule.
Assume duplicates and reordering
Providers retry, so duplicate deliveries are expected rather than exceptional, and events arrive out of order — a settlement can land before the acknowledgement it logically follows. Make the handler idempotent and guard each transition against the state you already hold.
Persist the raw payload before processing it. When you need to reconstruct an incident, the stored event is the only reliable record of what you were actually told.
Return fast, process later
A handler that does its work inline before responding will eventually be slow, and a slow handler causes the provider to time out and retry — multiplying load at exactly the moment the system is already struggling.
Verify, persist, enqueue, respond. The actual state change happens in a worker. This keeps the response time flat regardless of what the event triggers downstream.
Reconcile against the provider, not yourself
Your database tells you what you intended. The provider's statement tells you what happened. Those two drift, and the gap is where losses hide.
- Pull a daily statementMatch it line by line against your ledger, keyed on the reference you supplied.
- Alert in both directionsPresent in one and absent from the other matters whichever way round it is. A payment the provider made that you have no record of is the more alarming case.
- Reconcile the amounts, not just the presenceA line that exists in both but for a different amount is a worse problem than one that is missing, and a presence check will not find it.
- Build it before you need itThe first time you reconcile should not be during an incident, when nobody is sure whether the tooling itself is right.
The same discipline applied to a batch run is covered in the bulk payout guide, where the statement rather than the API response is what tells you whether people were actually paid.
Test the failures, not just the successes
A sandbox that only returns success teaches you nothing. Before launch, deliberately exercise the cases that will actually occur.
- An insufficient-balance response mid-run.
- An invalid beneficiary account.
- A timeout during the request, followed by a retry with the same idempotency key.
- A duplicate idempotency key, to confirm you get the original result rather than a second payment.
- A webhook with a bad signature, which must be rejected.
- A settled payment that is later returned.
Making the failures reproducible
Testing these once by hand proves the code worked that afternoon. Keeping them as automated tests is what stops a refactor six months later quietly removing the guard.
- Record real provider responses, including the failures, and replay them in tests rather than hand-writing fixtures that drift from reality.
- Test the webhook handler directly with a crafted payload and a deliberately wrong signature — this needs no provider at all.
- Assert on the resulting state, not on the function returning without throwing.
- Include an out-of-order pair in the suite: settlement before acknowledgement, asserting the record ends in the right state.
Observability you will want at 2am
The time to decide what you can see is before the first incident, not during it. Payment integrations fail in ways that are invisible from ordinary application metrics, because nothing errors — things simply stop progressing.
| Signal | Why it matters | Alert when |
|---|---|---|
| Count by state, over time | A pile-up in one state is the earliest symptom | Pending count grows for a sustained period |
| Age of the oldest pending payment | Catches stalls a count would hide | Beyond the rail's normal settlement window |
| Webhook delivery rate | Tells you if your endpoint fell over | Drops sharply against the recent baseline |
| Retry rate | The first sign of upstream trouble | Crosses a threshold against normal |
| Reconciliation break count | The only true measure of correctness | Any unexplained break persists past one cycle |
The second row is the one most often missing and the most useful. A count of pending payments looks healthy right up until you notice the same forty have been pending since Tuesday.
Frequently asked questions
What makes a good idempotency key?
Something stable from your own domain — an invoice ID, or a payroll run identifier plus the beneficiary. It has to be generated before the first attempt and reused unchanged across every retry, which rules out anything random created at call time.
What happens if I retry past the idempotency window?
The provider treats it as a new request and executes it. That is one of the more common routes to a duplicate payment, which is why your retry schedule has to be shorter than the provider's window.
Do I still need polling if I have webhooks?
Yes. Webhooks assume your endpoint is reachable and healthy; deployments, certificate expiries and firewall changes break that. A periodic sweep over anything still in a non-terminal state closes the gap left by dropped deliveries.
Should the webhook handler do its work before responding?
No. Verify, persist, enqueue and respond, then process in a worker. A slow handler causes the provider to time out and retry, multiplying load exactly when the system is already under strain.
Can a payout change state after it settles?
Yes. A beneficiary bank can return a transfer days later. Your state model must allow settled to move to returned, or that event will crash the handler or be silently discarded.
Why reconcile if the API already reports status?
Because the API reports what it believes about requests it received. Reconciling against the statement catches the cases where your record and the provider's diverge — including payments you have no record of at all.
What is the most useful thing to monitor?
The age of the oldest pending payment. A count of pending items looks healthy right until you notice the same forty have been stuck since Tuesday, which a count will never tell you.
