Webhook handling best practices: signatures, idempotency, ordering and replay

Webhooks look trivial: a provider sends an HTTP POST, you update a row. In practice they are a distributed system you do not control. Deliveries arrive late, twice, out of order, or not at all while your deploy is restarting, and the sender only knows what your status code told it. Over a few years of integrating payment providers, a shipping carrier and a CRM, every webhook incident we had traced back to one of a small set of mistakes.

This guide collects the practices that removed those incidents. It is written for engineers adding a first or fifth webhook integration. By the end you will have a receiver design that is secure, fast, idempotent and observable, plus the tests and alerts to keep it that way.

The shape of a reliable receiver

Every integration we run now follows the same four steps, in this order:

1. Verify the request is authentic, using the raw body bytes.
2. Persist the event with a unique key on the provider's event ID.
3. Acknowledge with a 2xx immediately.
4. Process asynchronously from a queue, idempotently.

Nothing slow happens before the acknowledgement. Nothing irreversible happens without a uniqueness check enforced by the database.

provider ──POST──> [verify] ──> [insert event, unique id] ──> 200 OK
                                         │
                                         └──> queue ──> worker (idempotent effects)

Verify signatures on the raw body

Most providers sign each delivery with an HMAC over the exact request bytes, often combined with a timestamp. Stripe uses a Stripe-Signature header containing t= and v1= values, GitHub uses X-Hub-Signature-256, and the Standard Webhooks specification (adopted by several vendors) uses webhook-id, webhook-timestamp and webhook-signature. The details differ; the rules do not:

• Compute the HMAC over the raw bytes. Any JSON parsing and re-serialising changes whitespace and breaks the match.
• Compare with a constant-time function to avoid timing leaks.
• Reject timestamps outside a small window (five minutes is a common default) to limit replays of a captured request.
• Keep one secret per endpoint and per environment, and support two active secrets during rotation.

A generic verifier for a Standard Webhooks style signature, in Python:

import base64, hashlib, hmac, time

TOLERANCE_SECONDS = 300

def verify(raw_body: bytes, headers: dict[str, str], secrets: list[bytes]) -> None:
    msg_id = headers["webhook-id"]
    ts = int(headers["webhook-timestamp"])
    if abs(time.time() - ts) > TOLERANCE_SECONDS:
        raise ValueError("timestamp outside tolerance")

    signed = f"{msg_id}.{ts}.".encode() + raw_body
    provided = [s.split(",", 1)[1] for s in headers["webhook-signature"].split() if s.startswith("v1,")]

    for secret in secrets:  # current and previous secret during rotation
        expected = base64.b64encode(hmac.new(secret, signed, hashlib.sha256).digest()).decode()
        if any(hmac.compare_digest(expected, p) for p in provided):
            return
    raise ValueError("no matching signature")

Use the provider's official SDK verifier when one exists; it encodes details such as secret prefixes and header parsing that are easy to get wrong. Whatever you use, make sure your framework gives the handler the unparsed body. In Express that means mounting express.raw() on the webhook route before any global express.json().

Acknowledge fast, process later

Providers enforce response timeouts and treat slow responses as failures, then retry. If your handler calls three APIs and renders a PDF before returning, a slow dependency turns into duplicate deliveries. Stripe's documentation asks for a 2xx before any complex logic, and most providers say the same.

// Laravel receiver: verify, persist, enqueue, return
public function __invoke(Request $request)
{
    $event = $this->verifier->verify($request->getContent(), $request->headers->all());

    $isNew = DB::table('webhook_events')->insertOrIgnore([
        'provider' => 'acme', 'event_id' => $event['id'], 'type' => $event['type'],
        'payload' => $request->getContent(), 'received_at' => now(),
    ]) === 1;

    if ($isNew) {
        ProcessAcmeEvent::dispatch($event['id'])->onQueue('webhooks');
    }

    return response()->noContent(200);
}

Return 2xx for duplicates too. A non-2xx tells the provider to try again, which is the opposite of what you want for an event you already have.

Use status codes deliberately: 400 or 401 for signature failures (the provider will retry, which is fine while you fix a secret), 2xx for everything you have safely stored, and 5xx only when you could not store the event.

Make processing idempotent

Delivery is at least once. The same event will arrive twice: after timeouts, network errors, your own deploys, and someone clicking "resend" in a dashboard. Idempotency needs two layers:

• Event level: a unique index on (provider, event_id). Inserting first and checking the result is safe under concurrency; "select, then insert" is not.
• Business level: a unique key on the thing the event changes, such as a capture ID in a ledger, a shipment ID or an invoice number. Providers sometimes emit two different events about the same object, and manual replays may not reuse the event ID.

CREATE TABLE ledger_entries (
  id           BIGINT PRIMARY KEY AUTO_INCREMENT,
  source       VARCHAR(32)  NOT NULL,
  source_ref   VARCHAR(128) NOT NULL,
  amount_minor BIGINT       NOT NULL,
  currency     CHAR(3)      NOT NULL,
  UNIQUE KEY uq_source (source, source_ref)
);

Side effects that cannot be made unique in your database, such as sending an email or calling a third-party API, need their own "done" marker or the downstream API's idempotency key. Stripe's API, for example, accepts an Idempotency-Key header on POST requests.

Do not trust delivery order

Providers generally do not guarantee order. You can receive invoice.paid before invoice.created, or a "subscription updated" from five minutes ago after the latest one. Three techniques cover almost every case:

1. Fetch current state. Treat the webhook as a notification that something changed, then read the object from the provider's API. The API is the source of truth; the payload is a snapshot.
2. Compare versions. If the payload carries a version, sequence number or updated timestamp at sufficient resolution, ignore events older than what you have stored.
3. Model state transitions. Only allow moves your state machine permits. A late "pending" event must not move an order back from "paid".

ALLOWED = {
    "pending": {"paid", "failed", "cancelled"},
    "paid": {"refunded", "disputed"},
    "failed": {"paid"},
}

def apply_status(order, new_status):
    if new_status == order.status or new_status not in ALLOWED.get(order.status, set()):
        return False  # stale or duplicate transition, ignore
    order.status = new_status
    return True

Plan for replay and recovery

Webhooks will be missed: a broken deploy, an expired certificate, a firewall change. Design for catching up rather than hoping:

• Keep raw payloads for a defined period (we use 30 to 90 days depending on personal data) so you can reprocess after fixing a bug without asking the provider.
• Build a replay command that re-enqueues stored events by ID or time range. Because processing is idempotent, replaying is safe.
• Know the provider's retry window. Stripe retries for up to three days in live mode; PayPal documents up to 25 retries over three days. Outages longer than that need a backfill from the provider's list APIs.
• Reconcile on a schedule. A daily job that compares provider objects with your records catches anything both retries and replays missed.

# our replay command
php artisan webhooks:replay --provider=acme --from="2026-03-02 09:00" --to="2026-03-02 11:30"

Monitor the pipeline, not just the endpoint

A webhook endpoint that returns 200 can still be failing: the queue may be stuck, or the worker may be throwing on one event type. We alert on:

Log the provider event ID on every line related to an event, so one search shows receipt, processing and side effects together.

Security checklist

• Serve the endpoint over HTTPS only and reject requests without a valid signature.
• If the provider publishes source IP ranges, allowlisting them is a useful second layer, never a replacement for signatures.
• Limit body size and parse defensively; a webhook endpoint is a public, unauthenticated POST target until the signature check passes.
• Exempt only the webhook route from CSRF middleware, not a whole route group.
• Do not log full payloads or signature headers at info level; payloads often contain personal data.

Common mistakes

Key takeaways

• Verify, persist, acknowledge, then process: in that order, every time.
• Signatures are computed over raw bytes; protect the body from parsers.
• Idempotency belongs in database constraints at both the event and business level.
• Treat webhooks as notifications and the provider API as the source of truth.
• Store events, build replay, reconcile daily and alert on the whole pipeline.

References

• Stripe documentation: Receive Stripe events in your webhook endpoint — https://docs.stripe.com/webhooks
• Stripe documentation: Idempotent requests — https://docs.stripe.com/api/idempotent_requests
• PayPal developer documentation: Webhooks — https://developer.paypal.com/api/rest/webhooks/rest/
• GitHub Docs: Validating webhook deliveries — https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries
• GitHub Docs: Best practices for using webhooks — https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks
• Standard Webhooks specification — https://www.standardwebhooks.com/