Tag See all →

#batch-jobs

2 stories

Banner: Kubernetes CronJob blocked because a previous Job hung and the policy forbids overlap

DevOps & CI/CD Bug Report

Kubernetes CronJob silently stopped for nine days: Forbid policy and a hung Job

Ryan Pham · Mar 11, 2026 · 6 min read

Our nightly invoice export stopped running and nobody noticed for nine days. One Job had hung on a network call with no timeout, and with concurrencyPolicy Forbid every later schedule was skipped. Deadlines on the Job, timeouts in code and an alert on last success fixed it.

Banner: Kubernetes batch jobs failing on Spot instances due to interruptions

Cloud & Infrastructure Bug Report

Moving to Spot saved 60 percent and broke our nightly jobs

Ryan Pham · Oct 22, 2025 · 5 min read

We moved our EKS batch node pool to Spot and cut its cost by 60 percent. Within a week, nightly jobs failed most nights: a single instance type, no interruption handling and jobs with no checkpoints. Diversified instances, graceful draining and checkpointing made Spot reliable.