Customers credited twice: PayPal webhook retries hit a non-idempotent handler

We run a tutoring marketplace where students top up a prepaid wallet through PayPal and spend the balance on lessons. On a Thursday evening the finance channel flagged that the PayPal settlement total for the day was about 4,100 USD lower than the sum of wallet credits we had recorded. PayPal had collected each payment once. We had credited some of them twice.

Querying the ledger by PayPal capture ID showed the shape immediately:

mysql> SELECT capture_id, COUNT(*) AS credits, SUM(amount_cents) AS total
    ->   FROM wallet_ledger WHERE created_at >= '2025-03-13 17:00:00'
    ->   GROUP BY capture_id HAVING credits > 1;
+-------------------+---------+-------+
| capture_id        | credits | total |
+-------------------+---------+-------+
| 5TY05013RG002845M |       2 | 10000 |
| 8GH92733HW118342P |       2 |  6000 |
...
41 rows in set (0.02 sec)

Forty-one students had received a second credit, all between 17:52 and 19:40. Nine of them had already booked lessons with the extra balance.

Reproducing it

The nginx access log for the webhook route (our format logs $request_time at the end) showed the pattern for every duplicated capture: one request ending in 504 after almost exactly 30 seconds, followed minutes later by a second delivery that returned 200.

203.0.113.24 [13/Mar/2025:17:52:08 +0000] "POST /webhooks/paypal HTTP/1.1" 504 rt=30.001
203.0.113.24 [13/Mar/2025:17:55:31 +0000] "POST /webhooks/paypal HTTP/1.1" 200 rt=0.912

In staging we reproduced it by adding a 35-second sleep in the accounting client and sending the same simulated PAYMENT.CAPTURE.COMPLETED event twice. Both requests credited the wallet.

What we ruled out

• PayPal sending two different events. Both deliveries carried the same event ID in the body ("id": "WH-..."). This was a retry of one event, not two business events.
• A double click at checkout. There was one order and one capture per student. The duplication was entirely on the webhook side.
• A race between two queue workers. At that time the handler ran synchronously inside the HTTP request; no queue was involved.

Root cause

The handler did all its work inline: verify the signature, credit the wallet, generate an invoice PDF and push the transaction to our accounting provider's API, then return 200. That evening the accounting API was answering in 20 to 40 seconds. nginx's fastcgi_read_timeout was 30 seconds, so nginx gave up and returned 504 to PayPal, but PHP-FPM kept executing the script and committed the wallet credit anyway. PayPal treats a non-2xx response as a failed delivery and retries. The retry arrived after the first credit had committed, and nothing in the handler checked whether that event or that capture had already been applied.

What is happening under the hood

Webhook delivery is at-least-once. Providers retry when they do not see a success status in time, and they cannot know whether your side effects happened before the failure. PayPal documents that it retries a delivery up to 25 times over three days when the listener does not return a 2xx. Stripe, GitHub and others have similar policies and explicitly warn that the same event can arrive more than once. A timeout is the worst case because the work often succeeded:

The only safe assumption is that every event can arrive twice, concurrently or hours apart, and the handler must make the second application a no-op.

Environment

Minimal reproduction

// app/Http/Controllers/PaypalWebhookController.php (before)
public function __invoke(Request $request)
{
    $this->paypal->verifySignature($request); // throws on failure

    $event   = $request->json()->all();
    $capture = $event['resource'];
    $topup   = Topup::where('paypal_order_id', $capture['supplementary_data']['related_ids']['order_id'])->firstOrFail();

    DB::transaction(function () use ($topup, $capture) {
        $topup->wallet->increment('balance_cents', $topup->amount_cents);
        WalletLedger::create([
            'wallet_id'    => $topup->wallet_id,
            'capture_id'   => $capture['id'],
            'amount_cents' => $topup->amount_cents,
        ]);
    });

    $this->invoices->generatePdf($topup);   // 2-5 s
    $this->accounting->push($topup);         // normally 1 s, 20-40 s that evening

    return response()->noContent(200);
}

Who is affected

Any webhook consumer that performs non-idempotent side effects (credits, emails, shipments, inventory decrements) and either does slow work before responding or has no deduplication. It does not require a timeout: providers also redeliver after network errors, deploy restarts, and manual "resend" clicks by someone on your own team.

The fix

We split the handler into a fast, dedup-first receiver and an idempotent worker, with uniqueness enforced by the database rather than by application checks.

1. Record every event once, keyed by provider and event ID.

// database/migrations/2025_03_14_000001_create_webhook_events_table.php
Schema::create('webhook_events', function (Blueprint $table) {
    $table->id();
    $table->string('provider', 20);
    $table->string('event_id', 100);
    $table->string('event_type', 100);
    $table->json('payload');
    $table->timestamp('processed_at')->nullable();
    $table->timestamps();
    $table->unique(['provider', 'event_id']);
});

2. Receive, verify, insert-or-ignore, enqueue, respond. No business logic in the request.

public function __invoke(Request $request)
{
    $this->paypal->verifySignature($request);

    $event = $request->json()->all();
    $inserted = DB::table('webhook_events')->insertOrIgnore([
        'provider'   => 'paypal',
        'event_id'   => $event['id'],
        'event_type' => $event['event_type'],
        'payload'    => json_encode($event),
        'created_at' => now(),
        'updated_at' => now(),
    ]);

    if ($inserted === 1) {
        ProcessPaypalEvent::dispatch($event['id']);
    }

    return response()->noContent(200); // duplicate or new, the provider gets success
}

3. Make the business effect idempotent too. Event-ID dedup protects against retries of the same event, but a capture can also be reported by a different event (for example after a manual resend from a dashboard or a replay script). The ledger therefore gets its own unique key on the business identifier:

Schema::table('wallet_ledger', function (Blueprint $table) {
    $table->unique(['source', 'source_ref']); // ('paypal_capture', '5TY05013RG002845M')
});

// app/Jobs/ProcessPaypalEvent.php (handle excerpt)
DB::transaction(function () use ($capture, $topup) {
    try {
        WalletLedger::create([
            'wallet_id'    => $topup->wallet_id,
            'source'       => 'paypal_capture',
            'source_ref'   => $capture['id'],
            'amount_cents' => $topup->amount_cents,
        ]);
    } catch (UniqueConstraintViolationException) {
        return; // already credited: do nothing
    }
    Wallet::whereKey($topup->wallet_id)->increment('balance_cents', $topup->amount_cents);
});

WebhookEvent::where('provider', 'paypal')->where('event_id', $eventId)
    ->update(['processed_at' => now()]);

The ledger insert happens first, inside the same transaction as the balance update, so a duplicate fails at the unique index before any money moves. The PDF and accounting push moved to their own queued jobs, each with its own "already done" check.

Alternatives and trade-offs

• Cache-based dedup (Redis SET NX with a TTL). Fast, but a cache flush or an expiry shorter than the provider's three-day retry window lets duplicates through. We use it only as an extra guard in front of the database.
• Raising the nginx timeout. It would have hidden this incident and left the design unsafe. Faster responses reduce retries; they do not remove them.
• Relying on "check, then insert" in code. Two concurrent deliveries can both pass the check. A unique index is the only check that holds under concurrency.

How to test the fix

A feature test posts the same signed fixture twice, then once more with a different event ID but the same capture:

public function test_same_capture_is_credited_once(): void
{
    // phpunit.xml sets QUEUE_CONNECTION=sync, so the job runs inline
    $payload = $this->fixture('paypal/capture_completed.json');

    $this->postJson('/webhooks/paypal', $payload)->assertOk();
    $this->postJson('/webhooks/paypal', $payload)->assertOk();
    $this->postJson('/webhooks/paypal', array_merge($payload, ['id' => 'WH-TEST-REPLAY-0002']))->assertOk();

    $this->assertSame(1, WalletLedger::where('source_ref', $payload['resource']['id'])->count());
    $this->assertSame(5000, $this->wallet->fresh()->balance_cents);
}

php artisan test --filter=PaypalWebhookTest

Confirming in production

• The duplicate query from the investigation now runs nightly. It has returned zero rows since the deploy on 14 March.
• p95 response time for /webhooks/paypal dropped from 6.8 seconds to 45 ms, and the PayPal Developer Dashboard's webhook events view shows no failed deliveries since.
• For the 41 affected wallets we posted reversing ledger entries with a reference to the incident, and support contacted the nine students who had spent the extra balance. We honoured those lessons; the cost was about 360 USD.

Regression guard

• The feature test above, plus a concurrency test that fires two identical deliveries in parallel against a real MySQL instance in CI.
• An alert on webhook_events rows older than 10 minutes with processed_at still null, which catches stuck queues before customers do.
• A daily reconciliation job comparing PayPal captures from the API with ledger entries, in both directions.

Security implications

Duplicate processing is also an abuse vector: if an attacker could replay a captured, validly signed payload, a non-idempotent handler would credit them repeatedly. Signature verification plus event-ID and business-key uniqueness close that path. We also kept verification before the insert, so unsigned requests cannot fill the webhook_events table, and we purge payloads older than 90 days because they contain payer names and email addresses.

Prevention checklist

• Assume every webhook arrives at least twice, possibly at the same time.
• Deduplicate on the provider's event ID with a unique index, not an application check.
• Put a second unique key on the business identifier (capture, invoice, shipment).
• Respond 2xx after persisting the event; do slow work in queued jobs.
• Reconcile against the provider's API daily so silent drift surfaces within a day.

References

• PayPal developer documentation: Webhooks — https://developer.paypal.com/api/rest/webhooks/rest/
• Stripe documentation: Handle duplicate events — https://docs.stripe.com/webhooks
• Laravel 11 documentation: Queues — https://laravel.com/docs/11.x/queues
• nginx documentation: ngx_http_fastcgi_module (fastcgi_read_timeout) — https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html