Nightly billing job ran twice on the DST fall-back night and double-charged customers

On Sunday, November 3, 2024, customer support for our membership billing platform opened the day with a queue of "charged twice" tickets. Card statements showed two identical charges a few minutes over an hour apart. Our payment provider dashboard confirmed it: 1,284 subscriptions renewing that night had two successful charges each, for a total of about $61,900 in duplicates.

The billing worker's log had two runs where there should have been one:

2024-11-03T01:30:00-05:00 INFO  billing.run start run_date=2024-11-03 due=1291
2024-11-03T01:41:12-05:00 INFO  billing.run done  charged=1284 failed=7
2024-11-03T01:30:00-06:00 INFO  billing.run start run_date=2024-11-03 due=1291
2024-11-03T01:42:03-06:00 INFO  billing.run done  charged=1284 failed=7

The offsets give it away: -05:00 is Central Daylight Time, -06:00 is Central Standard Time. US daylight saving time ended at 02:00 CDT that night, so the clock went back to 01:00 and every minute between 01:00 and 01:59 happened twice.

Reproducing it

We replayed the scheduler loop with a fake clock that walked through the transition in one-minute steps, using the America/Chicago zone:

from datetime import datetime, timedelta, timezone
from zoneinfo import ZoneInfo

tz = ZoneInfo("America/Chicago")
t = datetime(2024, 11, 3, 5, 0, tzinfo=timezone.utc)  # 00:00 CDT
for _ in range(240):
    local = t.astimezone(tz)
    if (local.hour, local.minute) == (1, 30):
        print("fire", local.isoformat())
    t += timedelta(minutes=1)
# fire 2024-11-03T01:30:00-05:00
# fire 2024-11-03T01:30:00-06:00

What we ruled out

• Two scheduler replicas. Only one worker pod held the leader lease all night; both runs logged the same pod name.
• Provider-side retries. Each duplicate charge had its own request ID from our side, so these were two separate requests, not a retried one.
• System cron. We do not use crontab for this job, and in any case the cronie man page documents that when time is adjusted backward by less than three hours, cron avoids running fixed-time jobs twice. Our problem was our own scheduler.

Root cause

Our in-house scheduler (a loop that wakes every 20 seconds) decided whether a job was due by comparing the current local wall-clock time with the configured 01:30. To avoid firing twice within the same minute, it kept a guard: "skip if this job started less than 10 minutes ago". At 01:30 CST that guard saw the last start 70 real minutes earlier and let the job run again.

The billing job itself had no protection either. It selected subscriptions with next_billing_date <= today and only advanced next_billing_date after the run finished. Because the second run started after the first had completed, that should have saved us, but the advance was done in a final batch update that a refactor had moved into a deferred task, which ran at 02:15. Both runs saw the same 1,291 subscriptions as due.

What is happening under the hood

Local time is not monotonic. On a fall-back night, local times in the repeated hour map to two different instants; on a spring-forward night, local times in the skipped hour map to none. Python models this with the fold attribute (PEP 495): fold=0 is the first occurrence, fold=1 the second. Any scheduler that asks "is it 01:30 now?" in local time gets two yeses in the fall and no yes at all for a 02:30 job in the spring.

Guards based on elapsed time ("not within the last N minutes") do not help, because an hour passes between the two occurrences. The guard needs to be about the logical run (billing for 2024-11-03), not about time since the last start.

And with money, scheduling bugs are just one of many ways a job can run twice: a deploy restart, a retry after a timeout, an operator rerunning it by hand. The billing step itself has to be idempotent.

Environment

Minimal reproduction

# scheduler.py (before) - simplified
JOBS = {"billing": (1, 30)}  # local 01:30
last_start: dict[str, datetime] = {}

def tick(now_local: datetime) -> None:
    for name, (h, m) in JOBS.items():
        due = (now_local.hour, now_local.minute) == (h, m)
        recent = name in last_start and now_local - last_start[name] < timedelta(minutes=10)
        if due and not recent:
            last_start[name] = now_local
            run(name)

Note that subtracting two aware datetimes in the same zone in Python compares wall-clock values, not instants, which makes this guard even less reliable around transitions.

Who is affected

Any job scheduled at a local time between 01:00 and 01:59 (for US zones; the hour differs elsewhere) by a scheduler that matches wall-clock time, and any job whose side effects are not idempotent. Jobs scheduled at 02:00 to 02:59 local silently skip the spring-forward night instead.

The fix

1. Schedule in UTC. Billing now runs at 07:30 UTC every day. The container TZ is UTC; local time is only for display.

- JOBS = {"billing": (1, 30)}          # America/Chicago wall clock
+ JOBS = {"billing": "30 7 * * *"}     # UTC, evaluated against time.time()

We kept the business meaning ("bill after midnight Central") by computing run_date explicitly from the scheduled instant: run_date = (scheduled_utc.astimezone(ZoneInfo("America/Chicago"))).date().

2. A run ledger keyed by logical date. Before doing any work, a run claims its slot in a table with a unique constraint. A second attempt for the same date fails immediately:

CREATE TABLE job_runs (
  job_name    text        NOT NULL,
  run_date    date        NOT NULL,
  started_at  timestamptz NOT NULL DEFAULT now(),
  finished_at timestamptz,
  status      text        NOT NULL DEFAULT 'running',
  PRIMARY KEY (job_name, run_date)
);

def claim_run(conn, job: str, run_date: date) -> bool:
    row = conn.execute(
        "INSERT INTO job_runs (job_name, run_date) VALUES (%s, %s) "
        "ON CONFLICT DO NOTHING RETURNING job_name",
        (job, run_date),
    ).fetchone()
    return row is not None

3. One charge per subscription per period, enforced by the database. Each charge row is written before calling the provider, with a unique key on subscription and billing period:

ALTER TABLE charges
  ADD CONSTRAINT charges_one_per_period UNIQUE (subscription_id, period_start);

4. Idempotency keys on the provider call. The key is derived from the same pair, so even a retry after a network timeout cannot create a second charge on the provider side:

idem_key = f"sub-{sub.id}-period-{period_start.isoformat()}"
processor.charges.create(
    amount=sub.amount_minor,
    currency=sub.currency,
    customer=sub.customer_ref,
    idempotency_key=idem_key,
)

5. Advance state per subscription, in the same transaction as the charge record. The deferred batch update is gone; next_billing_date moves forward as soon as the charge row is committed.

We refunded all 1,284 duplicate charges the same morning and emailed affected customers with the refund reference.

Alternatives and trade-offs

• Keep local schedules but use a DST-aware scheduler. Some schedulers handle the repeated hour correctly. It fixes the trigger, not the lack of idempotency, and every team must know the rule.
• Avoid 01:00 to 03:00 local entirely. A cheap mitigation many teams use. We do it too for non-billing jobs, but it is a convention, not a guarantee.
• Distributed lock only. A lock prevents concurrent runs, not sequential duplicates an hour apart. The ledger is what covers that.

How to test the fix

Unit tests drive the scheduler with a fake clock across both 2024 transitions and assert exactly one fire per day:

@pytest.mark.parametrize("start", [
    datetime(2024, 11, 2, 12, tzinfo=timezone.utc),  # fall back
    datetime(2024, 3, 9, 12, tzinfo=timezone.utc),   # spring forward
])
def test_billing_fires_once_per_day_across_dst(start):
    fired = simulate(scheduler, start=start, hours=48, step=timedelta(seconds=20))
    assert [f.run_date for f in fired] == sorted(set(f.run_date for f in fired))
    assert len(fired) == 2

An integration test calls the billing job twice for the same run_date against a test database and the processor's sandbox, and asserts one charge per subscription:

pytest tests/billing/test_idempotent_run.py -q

Confirming in production

• The next nights ran once each at 07:30 UTC, visible as one row per date in job_runs.
• A manual rerun on staging for an already billed date logged run already claimed and exited.
• Daily reconciliation now compares charges against subscriptions due:

SELECT subscription_id, period_start, count(*)
FROM charges
WHERE created_at >= now() - interval '1 day'
GROUP BY 1, 2
HAVING count(*) > 1;

It returns zero rows, and an alert fires if it ever does not.

Regression guard

• The DST simulation test runs in CI for every scheduler change, with the 2025 transition dates added.
• A CI check fails if any job definition uses a local-time schedule or if a Dockerfile sets TZ to anything other than UTC.
• The billing code path refuses to call the processor without an idempotency key.

Security implications

No data was exposed, but double charging is a direct financial harm to customers and a compliance issue with card network rules on duplicate processing. Idempotency keys and the unique constraint also protect against a different risk: an attacker or a buggy client triggering repeated renewal calls. Idempotency keys must not contain personal data; ours contain only internal IDs and dates.

Prevention checklist

• Schedule jobs in UTC and derive business dates explicitly.
• Guard runs by logical period in a ledger with a unique constraint, not by elapsed time.
• Make every money-moving step idempotent at the database and provider level.
• Test schedulers with a fake clock across both DST transitions.
• Reconcile daily for duplicates and alert on any.

References

• PEP 495: Local Time Disambiguation (fold) — https://peps.python.org/pep-0495/
• Python documentation: zoneinfo — https://docs.python.org/3/library/zoneinfo.html
• cron(8) manual page, daylight saving time handling — https://man7.org/linux/man-pages/man8/cron.8.html
• PostgreSQL documentation: INSERT ... ON CONFLICT — https://www.postgresql.org/docs/16/sql-insert.html