Asynchronous payments raise an unavoidable question: how do you find out what happened? There are two mechanisms, they fail in different ways, and treating them as alternatives is the mistake. The reliable pattern uses webhooks for speed and polling as the safety net.
What each one is good at
| Webhooks | Polling | |
|---|---|---|
| Latency | Immediate | As often as you poll |
| Cost at rest | Nothing | Requests whether or not anything changed |
| Needs inbound connectivity | Yes | No |
| Survives your endpoint being down | No | Yes |
| Survives a provider outage | No | No |
| Ordering guarantees | None | You read current state, so ordering is moot |
| Good for | User-facing confirmation | Catching what was missed |
The asymmetry is the point. Webhooks fail when your side is unreachable — during a deployment, a certificate expiry, a firewall change. Polling runs from inside your own infrastructure, so most of what breaks webhooks does not affect it.
The failure modes in practice
- A deployment with a few seconds of downtime drops whatever arrived in that window. The provider retries, so this one usually self-heals.
- An expired TLS certificate drops everything until someone notices — and nothing in your application errors, because the requests never arrive.
- A firewall or WAF rule change silently starts rejecting the provider's requests, which look like bot traffic.
- A handler bug returns 500, the provider retries, the bug persists, and retries eventually stop.
- A provider incident delays delivery by hours, during which polling is the only way to know anything.
The pattern that works
- Webhooks as the primary pathThey give the latency that user-facing confirmation needs, and they cost nothing when nothing is happening.
- A sweep as the safety netOn a schedule, query anything still in a non-terminal state older than a threshold — fifteen minutes is a reasonable start — and update it from the provider's view.
- One code path for bothWhether an update arrives by webhook or by poll, run it through the same state-transition function. Two paths that both move money will drift, and the less-exercised one will hold the bug.
- Monitor the sweep's yieldIf it starts finding a lot, your webhook endpoint is in trouble. That signal is free and arrives before your customers complain.
Choosing the threshold and interval
Two numbers decide the sweep's behaviour: how old a record must be before it is swept, and how often the sweep runs. Both are derived from the rail rather than chosen arbitrarily.
- The threshold should sit comfortably beyond the normal time to settle, so the sweep is not re-querying things that are simply in progress.
- The interval should be short enough that a missed webhook costs minutes rather than hours.
- Back off per record: something swept ten times without changing does not need checking as often as something new.
- Cap the sweep's batch size so a backlog cannot turn one scheduled run into a thundering herd.
Getting webhook handling right
- Verify the signature before you act on the payload — always, including in staging.
- Compute it over the raw body, before parsing; re-serialising the JSON changes the bytes.
- Return quickly and process asynchronously; a slow handler causes the provider to time out and retry, multiplying your load exactly when it is already high.
- Make the handler idempotent, because duplicate deliveries are normal rather than exceptional.
- Guard transitions against current state, since events arrive out of order.
- Store the raw payload before processing, so an incident can be reconstructed from what you were actually sent.
What to return, and when
| Situation | Return | Why |
|---|---|---|
| Verified and queued | 200 | You have taken responsibility for it |
| Bad signature | 401 or 400 | Do not let a forged event be retried into success |
| Unknown event type | 200 | Acknowledge it; retries will not help either side |
| Event for an unknown record | 200, and alert | Retrying will not create the record; you have a different problem |
| Your datastore is down | 500 | This one genuinely should be retried |
The fourth row is counter-intuitive and worth getting right. Returning an error for an event you cannot place means the provider retries it for hours, none of which will succeed, while the real issue — that you are missing a record — goes uninvestigated.
Securing the endpoint
A webhook endpoint is a publicly reachable URL that changes payment state. Signature verification is the main control, but it is not the only one worth having, and the others cost almost nothing.
- Treat the URL as a secret even though it is not one. Do not log it, do not put it in client-side code, and do not reuse it across environments.
- Rate limit the endpoint. A signature check is cheap but not free, and an unauthenticated flood still costs you.
- Allow-list the provider's source addresses where they publish them, as a second layer rather than the primary one.
- Reject stale timestamps, which closes off replay of a genuine, correctly signed event captured earlier.
- Use a separate signing secret per environment, so a staging leak cannot be used against production.
Scaling the handler
Webhook volume is spiky by nature: a batch of four thousand payouts produces four thousand events over a short window. A handler sized for the average will fall over on the peak, and falling over triggers retries, which raises the peak.
- Decouple receipt from processingThe endpoint verifies, persists and enqueues. Nothing else. That keeps response time flat no matter how heavy the downstream work becomes.
- Size the queue, not the endpointBackpressure belongs in the worker pool. The endpoint should never be the thing that slows down, because that is what causes retries.
- Make the worker idempotent tooThe queue will deliver a message twice at some point. If only the endpoint is idempotent, you have moved the problem rather than solved it.
- Partition by record, not round-robinTwo workers processing two events for the same payment concurrently is how a state machine gets corrupted. Route events for one record to one worker.
- Shed load deliberately if you mustIf the queue is genuinely overwhelmed, returning a 500 so the provider retries later is better than accepting events you cannot process and losing them.
The batch runs that produce these spikes are covered from the sending side in the bulk payout guide.
Testing event handling without a provider
The parts of event handling most likely to be wrong are also the easiest to test, because they need no provider at all. A crafted payload and a signing secret is the entire fixture.
- A correctly signed event, asserting the resulting state rather than a 200 response.
- The same event delivered twice, asserting the state is identical and nothing was double-applied.
- A deliberately wrong signature, asserting rejection and no state change.
- A correctly signed event with an old timestamp, asserting rejection on the replay window.
- Two events out of order — settlement first, then the acknowledgement that logically preceded it — asserting the record ends settled.
- An event for a record that does not exist, asserting a 200 and an internal alert rather than an error response.
Replaying real traffic
Hand-written fixtures drift from reality: providers add fields, change formats and deprecate values, and your tests keep passing against a payload shape that no longer exists.
Because you are already persisting raw payloads before processing — the practice recommended above for incident reconstruction — you have a corpus of genuine events. Replaying a sample of those through the handler in a test environment catches contract drift that no fixture will.
Frequently asked questions
Can I rely on webhooks alone?
Not safely. Providers stop retrying after a while, and anything dropped after that is lost unless you have another way to find it. A periodic sweep over outstanding records closes that gap.
How often should the reconciliation sweep run?
Frequently enough that a missed event is noticed in minutes rather than hours, with an age threshold set beyond the rail's normal settlement time so it does not re-query things still legitimately in progress.
Why do duplicate webhooks arrive?
Because providers retry when they do not get a prompt success response — including when your handler was slow rather than broken. Idempotent handling is the only reliable defence.
What should I return for a webhook I cannot match to a record?
A 200, plus an internal alert. Retrying will not create the missing record, so an error response just produces hours of futile retries while the real problem goes uninvestigated.
Why verify the signature over the raw body?
Because parsing and re-serialising JSON changes the bytes — key order, whitespace, number formatting — and the signature was computed over the original. Capture the raw body before any parsing.
Should webhook and polling updates share code?
Yes. Route both through one state-transition function. Separate paths inevitably drift, and the one exercised less often is where the bug will be.
