Kubernetes CronJob silently stopped for nine days: Forbid policy and a hung Job

Finance asked why the nightly invoice export file had not appeared in their bucket for over a week. The CronJob existed, was not suspended, and had a valid schedule:

$ kubectl -n billing get cronjob invoice-export
NAME             SCHEDULE    TIMEZONE         SUSPEND   ACTIVE   LAST SCHEDULE   AGE
invoice-export   0 2 * * *   Europe/Berlin    False     1        9d              311d

ACTIVE 1 and LAST SCHEDULE 9d. One Job was still running, nine days later.

Reproducing it

$ kubectl -n billing get jobs -l app=invoice-export
NAME                      STATUS    COMPLETIONS   DURATION   AGE
invoice-export-29021400   Running   0/1           9d         9d
$ kubectl -n billing logs job/invoice-export-29021400 | tail -3
2026-02-27T01:00:04Z export: fetched 18211 invoices
2026-02-27T01:00:05Z export: uploading to partner SFTP

The pod was alive, using no CPU, stuck after "uploading to partner SFTP". The CronJob spec had concurrencyPolicy: Forbid.

What we ruled out

• Controller or scheduling problems. Other CronJobs in the cluster ran normally.
• Timezone or schedule syntax. LAST SCHEDULE matched the expected time on the day it hung.
• Resource limits. The pod was not throttled or OOM-killed; it was simply waiting.

Root cause

The export uploads a file to a partner's SFTP server. That night the partner's server accepted the TCP connection and then stopped responding. Our SFTP client had no timeout on the transfer, so the process waited forever. The Job had no activeDeadlineSeconds, so Kubernetes never killed it. With concurrencyPolicy: Forbid, the CronJob controller skips a scheduled run while a previous Job is still active. Each night it skipped, and because the skipped runs are not failures, nothing alerted.

What is happening under the hood

At each scheduled time, the CronJob controller checks the concurrency policy:

Forbid is the right choice for exports that must not overlap, and it turns any hung Job into a permanent stop. startingDeadlineSeconds controls how late a missed run may still start; it does not help when runs are skipped because of Forbid.

A Job has its own controls: activeDeadlineSeconds limits total runtime, after which pods are terminated and the Job is marked failed with reason DeadlineExceeded. backoffLimit limits retries. Without a deadline, a Job can run indefinitely.

Environment

Minimal reproduction

apiVersion: batch/v1
kind: CronJob
spec:
  schedule: "*/2 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: hang
              image: busybox:1.37
              command: ["sh", "-c", "sleep infinity"]

After the first run, no further Jobs are created.

The fix

1. Timeouts in the code. The SFTP client now has connect, banner and per-operation timeouts, and the whole upload has an overall deadline:

import paramiko

transport = paramiko.Transport((host, 22))
transport.banner_timeout = 15
transport.connect(username=user, pkey=key)
transport.set_keepalive(15)
sftp = paramiko.SFTPClient.from_transport(transport)
sftp.get_channel().settimeout(60)           # per read/write operation

2. A deadline on the Job. The export normally takes 3 minutes. We allow 30:

spec:
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 3600
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      activeDeadlineSeconds: 1800
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never

If the code timeouts fail for some reason, Kubernetes kills the Job after 30 minutes, it is marked failed, and the next night's run can start. startingDeadlineSeconds lets a run that was delayed (for example by a control plane upgrade at 02:00) still start within the hour.

3. Alert on success, not just failure. kube-state-metrics exposes the last successful time for CronJobs. We alert when it is older than expected:

- alert: CronJobNotSucceeding
  expr: time() - kube_cronjob_status_last_successful_time{cronjob="invoice-export"} > 26 * 3600
  for: 15m
  labels: {severity: page}
  annotations:
    summary: "invoice-export has not succeeded in over 26 hours"

We also send a heartbeat to an external dead man's switch service at the end of a successful run, which catches the case where the whole cluster or monitoring is down.

Alternatives and trade-offs

• concurrencyPolicy: Replace. A hung Job is killed by the next run. Good for idempotent tasks; risky for exports where a half-finished upload and a new upload could both reach the partner.
• Allow with application-level locking. More control, more code.

How to test the fix

In staging, we pointed the export at a test SFTP server that accepts connections and never responds (a small socat listener). The upload failed after 60 seconds with a timeout, the Job failed, and the alert test fired after we shortened its window. With the code timeout disabled, activeDeadlineSeconds killed the Job at 30 minutes and the next scheduled run started normally.

Confirming in production

• We deleted the hung Job; the next run exported and uploaded nine days of invoices in one file, which finance accepted after confirming with the partner.
• Since then, two partner outages caused upload timeouts; both Jobs failed within two minutes, alerts fired, and retries the next night succeeded.

Regression guard

• A Kyverno policy requires activeDeadlineSeconds on every Job template in CronJobs.
• Every CronJob gets a last-success alert from a shared Helm template, with the expected interval as a value.
• Code review checklist item: every network client call has a timeout.

Security implications

None directly. The export contains invoice data, and the delayed export meant finance was working without current data for nine days, a business continuity problem rather than a security one. The SFTP connection uses key authentication and host key pinning; we checked that the timeout changes did not weaken host key verification.

Prevention checklist

• Put timeouts on every network call in batch code.
• Set activeDeadlineSeconds on Jobs, especially with Forbid.
• Alert on time since last success, not only on failures.
• Add an external heartbeat for critical scheduled jobs.
• Choose the concurrency policy deliberately per job.

References

• Kubernetes CronJob, concurrency policy and starting deadline — https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
• Job termination and cleanup, activeDeadlineSeconds — https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-termination-and-cleanup
• kube-state-metrics CronJob metrics — https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/cronjob-metrics.md
• Paramiko transport and channel timeouts — https://docs.paramiko.org/en/stable/api/transport.html