Docker "exec format error": image built on Apple Silicon fails on an amd64 server

At 18:40 on a Friday, one of our engineers pushed a hotfix for the notification worker straight from his laptop because CI was queued behind a long integration run. Ten minutes later the worker on both production hosts was in a restart loop and push notifications stopped going out.

docker compose logs on the server showed one line, over and over:

notifier-1  | exec /usr/local/bin/docker-entrypoint.sh: exec format error
notifier-1 exited with code 1

And pulling the image by hand printed a warning we had skimmed past in the deploy log:

$ docker run --rm registry.example.com/notifier:1.42.1 node --version
WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested
exec /usr/local/bin/docker-entrypoint.sh: exec format error

We rolled back to 1.42.0 within five minutes and started digging.

Reproducing it

On any x86_64 Linux host without QEMU binfmt handlers installed:

docker run --rm --platform linux/arm64 node:22-alpine node --version
# exec /usr/local/bin/docker-entrypoint.sh: exec format error

On the laptop, a MacBook Pro with an M3 chip, everything worked, because Docker Desktop builds and runs linux/arm64 natively.

What we ruled out

• A broken entrypoint script. The usual suspects for this message are a script without a shebang or with Windows line endings. docker-entrypoint.sh came unchanged from the official node base image and had a correct #!/bin/sh line.
• A corrupted push. The digest on the server matched the digest the laptop pushed.
• The code change. The same commit built by CI an hour later ran fine.

Root cause

We inspected the tag in the registry:

docker buildx imagetools inspect registry.example.com/notifier:1.42.1

Name:      registry.example.com/notifier:1.42.1
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:4be1...

Manifests:
  Name:      registry.example.com/notifier:1.42.1@sha256:91c0...
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  linux/arm64

  Name:      registry.example.com/notifier:1.42.1@sha256:0d7e...
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  unknown/unknown

Only linux/arm64 (the unknown/unknown entry is the build attestation). docker build on Apple Silicon defaults to the host platform, so the laptop produced an arm64 image. When the amd64 server pulled the tag, the only image available was arm64; Docker pulled it with a warning, and the kernel refused to execute the first arm64 binary it was asked to run, which happened to be the entrypoint shell.

What is happening under the hood

A container shares the host kernel. It does not emulate a CPU. When the container starts, the kernel execve()s the entrypoint, reads its ELF header, sees a machine type it cannot run (AArch64 on an x86_64 kernel), and returns ENOEXEC. The runtime reports that as "exec format error". The name of the file in the message is just whatever ran first.

Registries can store several images under one tag using an image index (a "manifest list"). Each entry is labelled with a platform such as linux/amd64 or linux/arm64. A client pulling the tag picks the entry matching its own platform. If no entry matches, older setups fail with "no matching manifest", while current Docker versions fall back to the only image and warn, which is what happened to us.

Docker Desktop on macOS can run amd64 images on Apple Silicon through emulation, so the mismatch is invisible in the other direction: an amd64 image works on a Mac, just slower.

Environment

Minimal reproduction

# on an Apple Silicon Mac
docker build -t registry.example.com/demo:arm-only .
docker push registry.example.com/demo:arm-only

# on an x86_64 Linux host
docker run --rm registry.example.com/demo:arm-only
# WARNING: The requested image's platform (linux/arm64/v8) does not match ...
# exec ...: exec format error

Who is affected

Anyone building on Apple Silicon (or Graviton, Ampere, Raspberry Pi) and deploying to x86_64, or the reverse. Also teams migrating part of a fleet to arm64 nodes, where an amd64-only image suddenly lands on an arm64 Kubernetes node. Interpreted languages are not exempt: the base image's shell, the runtime binary and any native modules are architecture-specific.

The fix

We made CI the only path to the production registry and had it build both architectures, so the same tag works on our x86 servers today and on arm64 if we move later:

# .github/workflows/image.yml (excerpt)
jobs:
  image:
    runs-on: ubuntu-24.04
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-qemu-action@v3
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: registry.example.com
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: true
          tags: registry.example.com/notifier:${{ github.ref_name }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

setup-qemu-action registers emulators so the arm64 half can be built on an amd64 runner.

For someone who truly needs to build locally for the servers, the explicit command is:

docker buildx build --platform linux/amd64 -t registry.example.com/notifier:dev-amd64 --push .

We also pinned the platform in the server's compose file, so a mismatched image fails at pull time instead of at start time:

services:
  notifier:
    image: registry.example.com/notifier:1.42.2
    platform: linux/amd64

Making emulated builds fast enough

Building the arm64 half under QEMU was slow: npm ci with sharp took about six minutes. Two things helped. First, the Dockerfile only runs architecture-specific steps where needed, and uses the build platform for steps that produce portable output:

# syntax=docker/dockerfile:1
FROM --platform=$BUILDPLATFORM node:22-alpine AS assets
WORKDIR /src
COPY package.json package-lock.json ./
RUN npm ci --ignore-scripts
COPY . .
RUN npm run build          # TypeScript to JS: same output on any CPU, runs natively

FROM node:22-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev      # runs per target platform, installs the right sharp binary
COPY --from=assets /src/dist ./dist
CMD ["node", "dist/worker.js"]

Second, we added an arm64 Linux larger runner (available on paid GitHub plans) and split the matrix so each architecture builds natively and a final job merges them with docker buildx imagetools create.

Alternatives and trade-offs

• Build amd64 only. Simplest and fastest if you will never run on arm64. Use --platform linux/amd64 everywhere and accept slower local runs on Macs.
• Install QEMU binfmt on the servers. Lets arm64 images run on amd64 hosts through emulation. It turns a crash into a slow service; we do not recommend it for production.
• Cross-compile. For Go or Rust, build with $BUILDPLATFORM and set GOARCH or the Rust target from $TARGETARCH, then copy the binary into a per-platform runtime stage. Fast and no emulation.

How to test the fix

docker buildx imagetools inspect registry.example.com/notifier:1.42.2 | grep Platform
#   Platform:  linux/amd64
#   Platform:  linux/arm64
#   Platform:  unknown/unknown
#   Platform:  unknown/unknown

docker run --rm --platform linux/amd64 registry.example.com/notifier:1.42.2 node -p process.arch
# x64
docker run --rm --platform linux/arm64 registry.example.com/notifier:1.42.2 node -p process.arch
# arm64

Confirming in production

• 1.42.2, built by CI, deployed to both hosts and stayed up; docker inspect on the running container reported "Architecture": "amd64".
• The notification backlog of about 3,800 messages drained in four minutes.
• Over the next month, 61 releases were built by the workflow, all with both platforms in the index. CI image build time went from 2 min 10 s (amd64 only) to 3 min 40 s with both, and back to 2 min 30 s after splitting into native runners.

Regression guard

• The registry only accepts pushes from the CI service account; developer credentials are pull-only.
• A deploy pre-check runs docker buildx imagetools inspect and refuses any tag without a linux/amd64 entry.
• Compose and Kubernetes manifests set the platform or a node selector on kubernetes.io/arch.

Security implications

The incident came from bypassing CI, which also bypassed our image scan and signed provenance. The fix restores both: every production image now comes from a reviewed commit with build attestations attached. Removing push rights from laptops also shrinks the blast radius of a stolen developer credential. QEMU on CI runners is fine; we do not install binfmt emulators on production hosts.

Prevention checklist

• Build production images in CI only, never on laptops.
• Set platforms explicitly in every build, even if it is a single platform.
• Inspect the image index of a tag before deploying it.
• Pin platform in compose files and use architecture node selectors in Kubernetes.
• Treat the "requested image's platform does not match" warning as an error.

References

• Docker documentation: Multi-platform builds — https://docs.docker.com/build/building/multi-platform/
• Docker documentation: Multi-platform image with GitHub Actions — https://docs.docker.com/build/ci/github-actions/multi-platform/
• Docker documentation: docker buildx imagetools inspect — https://docs.docker.com/reference/cli/docker/buildx/imagetools/inspect/
• Docker documentation: Build attestations — https://docs.docker.com/build/metadata/attestations/
• OCI Image Index specification — https://github.com/opencontainers/image-spec/blob/main/image-index.md