Moving to Spot saved 60 percent and broke our nightly jobs

Our nightly data jobs run on a dedicated EKS node pool: about 40 large instances from 01:00 to 05:00. Moving that pool to Spot looked like the easiest FinOps win of the year. Cost for the pool dropped by 60 percent in the first week. So did reliability: the morning report showed 4 to 11 failed jobs every night, against roughly zero before.

Job pod events showed the pattern:

Warning  NodeNotReady   pod/aggregate-orders-28911-x7k2p   Node is not ready
Normal   Killing        pod/aggregate-orders-28911-x7k2p   Stopping container worker
Warning  BackoffLimitExceeded  job/aggregate-orders-28911   Job has reached the specified backoff limit

And in the node lifecycle logs, clusters of nodes disappearing within minutes of each other around 02:30.

Reproducing it

The node pool configuration:

requirements:
  - key: karpenter.sh/capacity-type
    operator: In
    values: ["spot"]
  - key: node.kubernetes.io/instance-type
    operator: In
    values: ["r6i.4xlarge"]

One instance type, in one region, at a popular time. When Spot capacity for r6i.4xlarge was reclaimed, many of our nodes were interrupted together.

What we ruled out

• Job bugs. The same jobs succeeded on retry when run during the day.
• Out of memory. No OOMKilled events; containers were terminated with the node.

Root cause

Three gaps. We used a single instance type, so interruptions were correlated: when that pool's capacity was reclaimed, many nodes went at once. Our Karpenter installation had no interruption queue configured, so nodes were not drained when the two-minute interruption notice arrived; pods died with the instance. And our jobs were long (40 to 90 minutes) without checkpoints, with backoffLimit: 1, so one interruption late in a run meant starting over, and two meant failure.

What is happening under the hood

EC2 can reclaim Spot instances when it needs the capacity back. It sends an interruption notice two minutes before, and may send an earlier "rebalance recommendation" when an instance is at elevated risk. Interruptions are more likely when many customers want the same instance type in the same zone.

Karpenter and the AWS Node Termination Handler can listen for these events (through an SQS queue fed by EventBridge) and cordon and drain the node: pods get SIGTERM, can finish or checkpoint, and are rescheduled elsewhere. Without that, the instance just stops and the kubelet's pods vanish.

Diversification is the main lever on interruption rates. With many instance types and sizes allowed, Karpenter's Spot allocation picks pools with more spare capacity and spreads risk.

Environment

Minimal reproduction

AWS Fault Injection Service can send Spot interruptions to chosen instances. Running one against a node in the batch pool killed its job pod without any graceful shutdown.

The fix

1. Diversify instance types. The NodePool now allows any memory-optimized instance of suitable size from several generations and families:

requirements:
  - key: karpenter.sh/capacity-type
    operator: In
    values: ["spot", "on-demand"]
  - key: karpenter.k8s.aws/instance-category
    operator: In
    values: ["r", "m"]
  - key: karpenter.k8s.aws/instance-generation
    operator: Gt
    values: ["5"]
  - key: karpenter.k8s.aws/instance-cpu
    operator: In
    values: ["16", "32"]

With both capacity types allowed, Karpenter prefers Spot and falls back to on-demand when Spot is unavailable.

2. Interruption handling. We created the SQS queue and EventBridge rules from the Karpenter documentation and set settings.interruptionQueue in the Helm values. Now an interruption notice cordons and drains the node immediately.

3. Checkpoints and retries in jobs. Jobs write progress to S3 every 5 minutes and resume from the last checkpoint on restart. On SIGTERM they write a final checkpoint. backoffLimit went up to 4, and a pod failure policy treats disruption as non-counting:

spec:
  backoffLimit: 4
  podFailurePolicy:
    rules:
      - action: Ignore
        onPodConditions:
          - type: DisruptionTarget

4. Critical jobs on on-demand. Two jobs that feed the finance close each month run with a node selector for on-demand capacity. Paying full price for them is cheaper than the human cost of a late close.

Alternatives and trade-offs

• On-demand base plus Spot burst. Guarantees a minimum capacity at full price; simpler to reason about.
• Spot for stateless services only. Many teams start there, because web replicas tolerate interruption naturally.
• Savings Plans instead of Spot. Smaller discount, no interruptions, commitment required.

How to test the fix

We ran a Fault Injection Service experiment interrupting 30 percent of batch nodes during a test run. Nodes were drained within seconds of the notice, jobs checkpointed and resumed on new nodes, and all jobs completed, about 8 minutes later than an undisturbed run.

Confirming in production

• Over the following month, failed jobs averaged 0.2 per night, all for unrelated bugs.
• Interruptions still happen, about 6 nodes per night on average, and are now invisible in job results.
• Pool cost stayed about 55 percent below on-demand; the small loss compared to week one is from on-demand fallback and critical jobs.

Regression guard

• Monthly Fault Injection Service experiment on the batch pool.
• An alert on Jobs failing with node-related reasons.
• The NodePool spec is checked in CI to allow at least 10 instance types.

Security implications

None directly. The interruption queue setup needs permissions for Karpenter to read SQS and describe instances; we scoped them to the specific queue. Checkpoints in S3 contain intermediate data, so they live in the same encrypted, access-controlled bucket as job outputs, with a lifecycle rule to delete them after a week.

Prevention checklist

• Diversify Spot across many instance types, sizes and generations.
• Configure interruption handling so nodes drain on notice.
• Checkpoint long-running jobs and handle SIGTERM.
• Do not count disruptions as job failures.
• Keep truly critical work on on-demand capacity.

References

• EC2 Spot instance interruptions — https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html
• Karpenter interruption handling — https://karpenter.sh/docs/concepts/disruption/#interruption
• Kubernetes Job pod failure policy — https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-failure-policy
• AWS Fault Injection Service Spot actions — https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html