Every deploy waits 30 seconds and kills jobs: shell-form CMD swallows SIGTERM

Our background workers process video transcoding jobs from a queue. Each deploy produced a burst of failed jobs and duplicate retries. Kubernetes events and pod statuses showed the same thing every time:

$ kubectl -n media get pod worker-5c9d7b6d8-q2m4n -o jsonpath='{.status.containerStatuses[0].lastState}'
{"terminated":{"exitCode":137,"reason":"Error","startedAt":"...","finishedAt":"..."}}

Exit code 137 means the process was killed with SIGKILL (128 + 9). Pods took exactly 30 seconds to terminate, which is the default terminationGracePeriodSeconds.

Reproducing it

Locally, with Docker:

$ docker run -d --name w worker:1.42
$ time docker stop w
w
real    0m10.214s
$ docker inspect w --format '{{.State.ExitCode}}'
137

docker stop sends SIGTERM, waits 10 seconds, then sends SIGKILL. Our app had a SIGTERM handler that should finish the current job and exit within a few seconds. It never logged "shutdown requested".

Inside the container:

$ docker exec w ps -o pid,comm,args
PID   COMMAND  COMMAND
  1   sh       /bin/sh -c node dist/worker.js
  7   node     node dist/worker.js

What we ruled out

• A broken signal handler. Sending SIGTERM directly to PID 7 inside the container (kill -TERM 7) triggered the handler and a clean exit in 3 seconds.
• Long-running jobs exceeding the grace period. The handler never ran at all, so job length was irrelevant.

Root cause

The Dockerfile used the shell form of CMD:

CMD node dist/worker.js

Docker turns that into /bin/sh -c "node dist/worker.js". The shell becomes PID 1 and starts Node as a child. When the container runtime sends SIGTERM to PID 1, the shell does not forward it to its child (and PID 1 ignores signals it has no handler for). Node never learns it should stop, keeps working until SIGKILL arrives, and the in-flight job dies mid-way.

What is happening under the hood

Container runtimes deliver stop signals to PID 1 in the container. PID 1 is special in Linux: the kernel does not apply default signal actions to it, so a PID 1 process that does not install a handler for SIGTERM simply does not die from it. Shells like sh (dash or busybox ash) running a single command do not forward signals to their children.

There are two ways to write CMD and ENTRYPOINT:

Entrypoint scripts have the same problem if they run the app as a child instead of replacing themselves with exec.

A second, related issue: Node.js as PID 1 does not reap zombie child processes. For apps that spawn subprocesses (ours ran ffmpeg), a tiny init process like tini handles both signal forwarding and reaping.

Environment

Minimal reproduction

FROM node:22-bookworm-slim
COPY worker.js .
CMD node worker.js         # shell form

process.on("SIGTERM", () => { console.log("shutdown requested"); process.exit(0); });
setInterval(() => {}, 1000);

docker stop takes 10 seconds and never prints the message.

The fix

1. Exec form with a minimal init.

FROM node:22-bookworm-slim
RUN apt-get update && apt-get install -y --no-install-recommends tini ffmpeg \
    && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY --chown=node:node dist/ ./dist/
USER node
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["node", "dist/worker.js"]

tini runs as PID 1, forwards SIGTERM to Node, and reaps zombie ffmpeg processes. Alternatively, docker run --init adds the same behaviour without changing the image, but Kubernetes has no equivalent flag, so we put it in the image.

2. A real graceful shutdown in the worker.

let stopping = false;
let current = null;

async function main() {
  while (!stopping) {
    const job = await queue.reserve({ timeoutSeconds: 5 });
    if (!job) continue;
    current = processJob(job).finally(() => { current = null; });
    await current;
  }
}

process.on("SIGTERM", async () => {
  if (stopping) return;
  stopping = true;
  console.log("shutdown requested, finishing current job");
  const deadline = setTimeout(async () => {
    console.log("deadline reached, releasing job back to queue");
    await queue.releaseCurrent();
    process.exit(1);
  }, 240_000);
  await current;
  clearTimeout(deadline);
  process.exit(0);
});

main();

3. A grace period that matches the work. Jobs can take up to 4 minutes, so the Deployment sets terminationGracePeriodSeconds: 270, slightly longer than the worker's internal deadline, so the worker can release the job before SIGKILL.

Alternatives and trade-offs

• exec in an entrypoint script. If you need a shell script to set up the environment, end it with exec node dist/worker.js so Node replaces the shell as PID 1.
• Shorter jobs with checkpoints. For very long jobs, checkpointing progress lets you keep grace periods short.
• Kubernetes shareProcessNamespace changes PID 1 to the pause container; it solves reaping but complicates other things. We did not use it.

How to test the fix

$ docker run -d --name w worker:1.43
$ docker stop -t 60 w && docker logs w | tail -2
shutdown requested, finishing current job
job 8812 done
$ docker inspect w --format '{{.State.ExitCode}}'
0

In staging, a rollout during a load test with 200 queued jobs produced zero failed jobs.

Confirming in production

• Pods now stop in 2 to 90 seconds depending on the current job, with exit code 0.
• Failed jobs per deploy went from 15 to 40 down to zero. Duplicate retries after deploys disappeared.
• Zombie ffmpeg processes, which we had not even noticed, no longer accumulate on long-running workers.

Regression guard

• hadolint runs in CI; rule DL3025 flags shell-form CMD and ENTRYPOINT.
• A dashboard tracks container termination exit codes; any 137 during a rollout triggers a review.
• Shutdown behaviour is part of the service template's integration test.

Security implications

None directly. Running as a non-root USER and using a minimal init both reduce what a compromised process can do, so the new Dockerfile is a small hardening step as well.

Prevention checklist

• Use exec form for CMD and ENTRYPOINT.
• End entrypoint scripts with exec.
• Add tini (or equivalent) when the app spawns child processes.
• Handle SIGTERM: stop taking work, finish or release in-flight work, exit.
• Set terminationGracePeriodSeconds to fit the longest unit of work.

References

• Dockerfile reference, shell and exec form — https://docs.docker.com/reference/dockerfile/#shell-and-exec-form
• Kubernetes pod termination — https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination
• tini — https://github.com/krallin/tini
• hadolint DL3025 — https://github.com/hadolint/hadolint/wiki/DL3025