A catch-all exception handler reported a CRM sync as successful for five weeks

The sales team noticed that new customers from the past month were missing from the CRM. Leads that had signed up, paid and started using the product had no record there, so nobody had contacted them for onboarding.

The nightly sync job's history in our scheduler showed a green run every night:

job: crm-sync-customers
2026-10-01 02:00  succeeded  duration 4s   exit 0
2026-10-02 02:00  succeeded  duration 3s   exit 0
...
2026-11-05 02:00  succeeded  duration 4s   exit 0

Four seconds. Before October, the job took four to six minutes. Its log for a recent run:

02:00:01 INFO  crm-sync: 214 customers to sync
02:00:03 WARNING crm-sync: sync batch failed, will retry next run
02:00:04 INFO  crm-sync: done

"Will retry next run", every night, for five weeks. There was no retry logic in the job, and each run selected customers created "since the last successful sync", a timestamp that was updated at the end of every run, because every run ended with "done". So each night's failed customers were never selected again.

The code

def sync_customers():
    since = state.get("last_sync_at")
    customers = Customer.objects.filter(created_at__gte=since)
    log.info("crm-sync: %d customers to sync", len(customers))
    try:
        for batch in chunks(customers, 100):
            crm.upsert_contacts([to_contact(c) for c in batch])
    except Exception:
        log.warning("crm-sync: sync batch failed, will retry next run")
    state.set("last_sync_at", now())
    log.info("crm-sync: done")

Running the body of the try by hand in a shell showed the real error:

crm_client.errors.AuthenticationError: 401 Unauthorized: token expired

The CRM API token had a 12-month lifetime and expired on the 1st of October. The broad except Exception turned an authentication failure into a warning without the exception's message or traceback, the job then marked the sync as complete, and exited with code 0.

What the except clause actually did

The developer's intent was reasonable: one bad batch should not crash the whole job. But except Exception catches everything: a malformed record, a network blip, an expired credential, a bug in to_contact, an AttributeError from a typo. Those are very different failures:

Catching all of them in one place and treating them identically meant that the most serious kind, a failure that will happen on every run until a human acts, was handled like the most trivial kind.

The second bug: progress recorded regardless

Even with the swallowed exception, the damage would have been one night's customers if the job had not advanced its checkpoint. state.set("last_sync_at", now()) ran whether or not anything was synced. The checkpoint was claiming work that had not been done.

Why monitoring did not catch it

• The job exited with 0, so the scheduler showed success.
• The warning had no traceback and no error class, so log-based alerts on ERROR or on exception names did not match it.
• Nobody alerted on the duration dropping from five minutes to four seconds, or on the number of contacts actually created in the CRM.

Environment

1. Classify errors and handle each kind deliberately

We defined a small taxonomy for integrations, with exceptions raised by our CRM client wrapper:

class IntegrationError(Exception): ...
class TransientError(IntegrationError): ...      # timeouts, 429, 503: retry
class RecordRejected(IntegrationError): ...      # 400/422 for one record: skip, report
class ConfigurationError(IntegrationError): ...  # 401/403, missing settings: stop, page

The sync handles them differently:

def sync_customers() -> SyncResult:
    since = state.get("last_sync_at")
    customers = list(Customer.objects.filter(created_at__gte=since).order_by("created_at"))
    result = SyncResult(total=len(customers))

    for batch in chunks(customers, 100):
        try:
            with_retries(lambda: crm.upsert_contacts([to_contact(c) for c in batch]),
                         retry_on=TransientError, attempts=3)
        except RecordRejected as exc:
            result.rejected.extend(exc.records)
            log.warning("crm-sync: %d records rejected", len(exc.records), exc_info=exc)
            continue
        # ConfigurationError and anything unexpected propagate: the job fails
        result.synced += len(batch)
        state.set("last_sync_at", batch[-1].created_at)   # advance only past synced work

    return result

Key changes:

• Only expected, recoverable errors are caught, and each is caught where it can be handled.
• Configuration errors and unknown exceptions propagate. The job crashes, exits non-zero, and the scheduler marks it failed.
• The checkpoint advances per batch, only after that batch succeeded. A failure leaves the checkpoint at the last good batch, so the next run picks up exactly where this one stopped.
• exc_info is always logged with handled errors, so the traceback and the exception's message are kept.

2. Alert on handled errors too

"Handled" does not mean "fine". Rejected records are handled, but a sudden rise means something is wrong. The job now emits metrics at the end of each run:

metrics.gauge("crm_sync.total", result.total)
metrics.gauge("crm_sync.synced", result.synced)
metrics.gauge("crm_sync.rejected", len(result.rejected))

And we alert on:

• job failure (non-zero exit) from the scheduler, paging;
• synced / total < 0.95 for a run with more than 20 customers, ticket;
• no successful run in 26 hours (a dead man's switch on the job), paging.

3. Lint for broad excepts

Ruff's BLE001 (blind except) and S110 (try/except/pass) now run in CI for this repository. Each remaining broad except needs a # noqa: BLE001 with a comment explaining why, and the reviewer must agree. There are three left in the codebase, all at the top level of worker processes, where they log the full traceback and re-raise or exit non-zero.

4. Credential expiry

Separately, we added the CRM token's expiry date to our secrets inventory, with an alert 30 days before. Expired credentials are one of the most predictable failures there is.

1. Tests for each error class. With a fake CRM client:

def test_expired_token_fails_the_job(fake_crm):
    fake_crm.fail_with(ConfigurationError("401 token expired"))
    with pytest.raises(ConfigurationError):
        sync_customers()
    assert state.get("last_sync_at") == INITIAL_CHECKPOINT

def test_rejected_records_are_reported_not_fatal(fake_crm, customers_factory):
    customers = customers_factory(5)
    fake_crm.reject(ids=[customers[2].id])
    result = sync_customers()
    assert result.synced == 4 and len(result.rejected) == 1

On the old code, the first test failed: no exception, and the checkpoint moved.

2. Recovery. We rotated the CRM token, reset the checkpoint to 1 October, and ran the job. It synced 6,112 customers in 31 minutes and exited 0. Sales received a list of the affected customers for follow-up.

3. A live drill. We revoked the staging CRM token on purpose. The next staging run failed with exit code 1, the scheduler showed red, and the on-call channel received the page within a minute.

4. Audit of other jobs. A search for except Exception across our 23 scheduled jobs found the same pattern in four more. Two of them also advanced a checkpoint unconditionally. All fixed the same way.

5. Security implications. A swallowed AuthenticationError can also hide a revoked credential after a security incident, meaning integrations quietly stop while everyone believes they work. Failing loudly on authentication errors makes credential revocation visible to the teams who depend on it.

Takeaways

• except Exception treats every failure the same, including the ones a retry can never fix.
• Catch specific, recoverable errors where you can handle them; let configuration errors and unknown exceptions fail the job.
• Never record progress for work that did not succeed.
• Log handled errors with tracebacks, measure them, and alert on rates and on missing successful runs.

References: Python tutorial, errors and exceptions, Python logging, exc_info, Ruff rule BLE001 blind-except, Ruff rule S110 try-except-pass.