Kubernetes CrashLoopBackOff with OOMKilled exit code 137: Node heap set above the container limit

Our storefront backend-for-frontend (BFF), a Node.js service that renders product pages and aggregates three internal APIs, started restarting a few times a day in late November. During the first big sale weekend it got worse: pods restarted every 20 to 40 minutes, several sat in CrashLoopBackOff, and the remaining pods took their traffic, ran out of memory faster, and followed them.

kubectl describe pod on one of them:

Containers:
  bff:
    Image:          registry.example.com/storefront-bff:2024.11.28-3
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Sat, 30 Nov 2024 10:12:41 +0000
      Finished:     Sat, 30 Nov 2024 10:47:03 +0000
    Ready:          False
    Restart Count:  9
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:     500m
      memory:  1Gi
Events:
  Type     Reason   Age                 From     Message
  ----     ------   ----                ----     -------
  Warning  BackOff  2m (x31 over 2h)    kubelet  Back-off restarting failed container bff in pod storefront-bff-7d9c6b8f5-q2vxl

Exit code 137 is 128 plus signal 9: the process was killed with SIGKILL. Reason: OOMKilled says who sent it: the kernel's out-of-memory killer, because the container exceeded its memory limit.

Reproducing it

In a staging pod with the same limit, a load test against the product page endpoint pushed memory up steadily:

kubectl -n storefront exec deploy/storefront-bff -- node -e \
  "const v8=require('v8');console.log(v8.getHeapStatistics().heap_size_limit/1048576,'MiB')"
# 2096 MiB

The V8 heap was allowed to grow to about 2 GiB inside a container limited to 1 GiB. Under load, the pod was killed at about 1 GiB of working set after 25 to 35 minutes, every time.

What we ruled out

• A sudden leak in the latest release. Rolling back two releases changed nothing; restarts had been happening for weeks at low traffic, just rarely.
• Node memory pressure evicting pods. Eviction shows as Reason: Evicted with a node-pressure message, not OOMKilled, and the nodes had free memory.
• The JavaScript heap limit. If V8 had hit its own limit, we would have seen FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory in the logs. There was no such message; the process just vanished.

Root cause

The Dockerfile carried a setting from when the service ran on 4 GB virtual machines:

ENV NODE_OPTIONS="--max-old-space-size=2048"

With that, V8 believed it could use 2 GiB of old-generation heap, so it collected garbage lazily and let the heap grow well past 1 GiB. The kernel enforces the cgroup limit regardless of what V8 thinks, and killed the process the moment the container's memory crossed 1 GiB. A second factor made the heap grow in the first place: an in-process cache of rendered product fragments used a plain Map with no size bound, so it grew with the number of distinct products viewed, which exploded during the sale.

What is happening under the hood

A container memory limit is enforced by the Linux kernel through cgroups (memory.max on cgroup v2). When the processes in the container try to use more than that, the kernel reclaims what it can and, failing that, invokes the OOM killer, which sends SIGKILL. There is no exception, no shutdown hook and no log line from the application. Kubernetes records the termination as OOMKilled with exit code 137 and restarts the container according to restartPolicy; repeated restarts back off exponentially, which is the CrashLoopBackOff state.

Managed runtimes have their own idea of available memory:

• Node.js sizes the V8 old-generation heap from a default, or from --max-old-space-size if set. An explicit value always wins. If it is larger than the container limit, the heap can outgrow the container before V8 feels any pressure to collect.
• The JVM has been container-aware since JDK 10 (and 8u191). By default it sizes the maximum heap at 25 percent of the container limit (-XX:MaxRAMPercentage=25). A hard-coded -Xmx larger than the limit causes exactly the same OOMKill.

And the heap is not the whole process. Node's resident memory also includes code, buffers outside the heap (Buffer, compression, TLS), native modules and thread stacks. A heap of 100 percent of the limit still gets killed.

Environment

Minimal reproduction

apiVersion: v1
kind: Pod
metadata:
  name: oom-demo
spec:
  restartPolicy: Always
  containers:
    - name: node
      image: node:22-alpine
      env:
        - name: NODE_OPTIONS
          value: "--max-old-space-size=1024"
      resources:
        limits:
          memory: 256Mi
      command: ["node", "-e", "const a=[];setInterval(()=>a.push(Buffer.alloc(1e6).toString('hex')),20)"]

kubectl apply -f oom-demo.yaml
kubectl get pod oom-demo -w
# oom-demo   0/1   OOMKilled          0   14s
# oom-demo   0/1   CrashLoopBackOff   1   28s

Who is affected

Any containerised Node.js service with --max-old-space-size copied from a VM or developer machine, JVM services with a fixed -Xmx, Python workers that load large datasets, and services whose limit was lowered later without touching runtime flags. Unbounded in-memory caches make it worse in any language.

The fix

1. Size the heap from the container limit. We pass the limit into the container with the downward API and compute the heap in the entrypoint, leaving 25 percent headroom for non-heap memory:

# deployment.yaml (excerpt)
resources:
  requests:
    memory: 1536Mi
    cpu: 500m
  limits:
    memory: 1536Mi
env:
  - name: MEMORY_LIMIT_BYTES
    valueFrom:
      resourceFieldRef:
        containerName: bff
        resource: limits.memory

#!/bin/sh
# docker-entrypoint.sh
set -e
if [ -n "$MEMORY_LIMIT_BYTES" ]; then
  heap_mb=$(( MEMORY_LIMIT_BYTES / 1048576 * 75 / 100 ))
  export NODE_OPTIONS="--max-old-space-size=${heap_mb} ${NODE_OPTIONS:-}"
fi
exec node dist/server.js

-ENV NODE_OPTIONS="--max-old-space-size=2048"
+COPY docker-entrypoint.sh /usr/local/bin/
+ENTRYPOINT ["docker-entrypoint.sh"]

With a 1,536 MiB limit, the heap cap is 1,152 MiB. When the heap approaches it, V8 collects aggressively; if the service truly needs more, it fails with a clear heap error and stack trace instead of a silent kill.

2. Bound the cache. The fragment cache now uses lru-cache with both a count and a size limit:

import { LRUCache } from 'lru-cache';

export const fragmentCache = new LRUCache<string, string>({
  max: 5_000,
  maxSize: 150 * 1024 * 1024,              // about 150 MiB of rendered HTML
  sizeCalculation: (value) => Buffer.byteLength(value),
  ttl: 5 * 60 * 1000,
});

3. Raise the limit to what the service really needs. Profiling under a replay of sale traffic showed a steady-state RSS of about 900 MiB per pod with the bounded cache, so 1 GiB left almost no room. We set request and limit to 1.5 GiB, keeping them equal so the pod's memory is guaranteed and the scheduler places it honestly.

For JVM services

The same fix applies with JVM flags. Remove fixed -Xmx and let the JVM follow the limit:

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=75 -XX:+ExitOnOutOfMemoryError"

Alternatives and trade-offs

• Remove --max-old-space-size entirely. Recent Node versions pick a default heap that is usually smaller than a generous container, but the default depends on the Node version and on detected memory, so we prefer an explicit value derived from the limit.
• No memory limit. Avoids OOMKills in the container but lets one leaking pod push the node into memory pressure and get unrelated pods evicted. We keep limits.
• More replicas. Spreading load reduces per-pod memory only if memory scales with traffic. Our cache grew with catalog breadth, so more pods would each have grown the same cache.

How to test the fix

kubectl -n storefront exec deploy/storefront-bff -- node -e \
  "console.log(require('v8').getHeapStatistics().heap_size_limit/1048576,'MiB')"
# about 1152 plus young generation, well under the 1536 MiB limit

k6 run --vus 400 --duration 45m load/product-pages.js
kubectl -n storefront top pod -l app=storefront-bff

During the 45-minute run, working set stayed between 780 and 1,010 MiB and no pod restarted.

Confirming in production

We tracked OOMKills with kube-state-metrics:

sum by (namespace, container) (
  increase(kube_pod_container_status_restarts_total{namespace="storefront"}[1h])
) * on (namespace, pod, container) group_left
  max by (namespace, pod, container) (kube_pod_container_status_last_terminated_reason{reason="OOMKilled"})

Regression guard

• A Prometheus alert fires when any container's working set stays above 85 percent of its limit for 10 minutes, and another on any OOMKilled termination.
• A CI check rejects Dockerfiles and manifests that set --max-old-space-size or -Xmx as a literal.
• The service's startup log prints the heap limit and the container limit side by side.

Security implications

Unbounded caches keyed on request input are a denial-of-service risk: anyone requesting many distinct URLs could grow memory until pods die. Bounding the cache by size and TTL closes that. The downward API only exposes the container's own resource values, so passing the limit in as an environment variable does not leak anything sensitive.

Prevention checklist

• Derive heap size from the container limit, leaving about 25 percent headroom.
• Never copy fixed heap flags from VM or laptop configs into images.
• Bound every in-process cache by size, not just count.
• Alert on working set near the limit and on any OOMKilled termination.
• Load test with production-like traffic variety, not just volume.

References

• Kubernetes documentation: Assign memory resources to containers and pods — https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/
• Kubernetes documentation: Resource management for pods and containers — https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
• Kubernetes documentation: Expose pod information through environment variables (resourceFieldRef) — https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/
• Node.js documentation: --max-old-space-size — https://nodejs.org/docs/latest-v22.x/api/cli.html#--max-old-space-sizesize-in-mib
• Oracle Java documentation: java command, MaxRAMPercentage — https://docs.oracle.com/en/java/javase/21/docs/specs/man/java.html
• kube-state-metrics: pod metrics — https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md