Two pipelines raced and the older commit was deployed last, silently reverting a fix

At 14:20 we merged a hotfix for a checkout bug that was rejecting some valid postcodes. At 14:31 the pipeline reported "Deployed to production". At 14:45 support said customers were still hitting the bug. The hotfix's code was not running in production.

The production deployment's image tag:

kubectl -n shop get deploy checkout -o jsonpath='{.spec.template.spec.containers[0].image}'

registry.example.com/checkout:main

A mutable tag, main. The pod's actual image digest and the commit label inside the image:

kubectl -n shop get pod -l app=checkout -o jsonpath='{.items[0].status.containerStatuses[0].imageID}'
docker buildx imagetools inspect registry.example.com/checkout@sha256:4be1... --format '{{json .Image.Config.Labels}}'

{"org.opencontainers.image.revision":"a91f3c2", ...}

a91f3c2 was the commit before the hotfix: a feature merged at 14:12. The hotfix was d07e8b4.

The pipeline history

run   commit    trigger          build   test    deploy   finished
#812  a91f3c2   merge 14:12      14:13   14:15   14:33    14:34
#813  d07e8b4   merge 14:20      14:21   14:23   14:30    14:31

Run 813 (hotfix) finished first. Run 812 (feature) finished three minutes later and deployed its image over the hotfix. Run 812 was slower because its changes touched more modules and ran a larger set of integration tests.

Both runs pushed to the same main tag and ran kubectl rollout restart, which pulls whatever main points to at that moment. Run 812's image push happened after run 813's, so main ended up pointing at the older commit's image.

Pipelines run concurrently by default

CI systems start a run for every push or merge, and those runs proceed in parallel. Nothing guarantees they finish in the order they started. If each run ends by deploying, the deploy order is decided by which run's tests happen to be faster, not by which commit is newer.

That is fine for build and test. It is not fine for deploy, where the last writer wins.

Mutable tags hide what is deployed

checkout:main means "whatever was pushed to main last". Two consequences:

1. The deploy step does not actually decide which code runs; the registry's tag state at pull time does.
2. Looking at the Deployment does not tell you which commit is running. We needed the image digest and its labels to find out.

A pod restarted later, for example after a node failure, would pull main again and could get yet another image, without any deploy.

Environment

1. Serialise deploys with a concurrency group

GitHub Actions supports concurrency at the workflow or job level. Only one job in a group runs at a time, and with cancel-in-progress, a newer run cancels an older pending or running one:

jobs:
  build-test:
    runs-on: ubuntu-latest
    steps:
      # build, test, push image tagged with the commit SHA

  deploy:
    needs: build-test
    runs-on: ubuntu-latest
    environment: production
    concurrency:
      group: deploy-production-checkout
      cancel-in-progress: false
    steps:
      # deploy by digest (below)

We put the concurrency group on the deploy job only, so builds and tests still run in parallel. We left cancel-in-progress: false for production: cancelling a deploy halfway can leave a rollout in an odd state. GitHub Actions keeps at most one pending job per concurrency group, and a newer pending job replaces an older pending one. So if three runs reach deploy at once, one deploys, the oldest waiting one is dropped, and the newest deploys next. That is the behaviour we want.

2. Deploy an immutable digest, not a tag

The build job outputs the digest it pushed, and deploy uses exactly that:

  build-test:
    outputs:
      digest: ${{ steps.push.outputs.digest }}
    steps:
      - id: push
        uses: docker/build-push-action@v6
        with:
          push: true
          tags: registry.example.com/checkout:${{ github.sha }}
          labels: org.opencontainers.image.revision=${{ github.sha }}

  deploy:
    steps:
      - run: |
          kubectl -n shop set image deploy/checkout \
            checkout=registry.example.com/checkout@${{ needs.build-test.outputs.digest }}
          kubectl -n shop rollout status deploy/checkout --timeout=5m

Now the Deployment spec states exactly which image runs, and changing it is a real change to the spec, which Kubernetes rolls out properly. A pod restarted later gets the same image. The main tag no longer exists.

3. Refuse to deploy an older commit

Serialisation alone still allows the wrong order in one case: if the newer run reaches the deploy job first and the older one reaches it afterwards (because the older one's tests were slower), the older one waits and then deploys. Concurrency prevents overlap, not reordering. We added a guard at the start of the deploy job:

current=$(kubectl -n shop get deploy checkout \
  -o jsonpath='{.metadata.annotations.deploy\.example\.com/revision}')
if [ -n "$current" ] && git merge-base --is-ancestor "$GITHUB_SHA" "$current"; then
  echo "Commit $GITHUB_SHA is an ancestor of deployed $current; skipping older deploy."
  exit 0
fi

git merge-base --is-ancestor A B succeeds if A is an ancestor of B, that is, if B already contains A. If the commit we are about to deploy is already included in what is running, deploying it would move backwards, so the job exits. After a successful rollout, the job writes the deployed SHA to the annotation:

kubectl -n shop annotate deploy/checkout --overwrite deploy.example.com/revision="$GITHUB_SHA"

The job needs the full git history for the ancestry check (fetch-depth: 0 on checkout).

4. Intentional rollbacks still work

Rollbacks are now a separate, manually triggered workflow that takes a specific digest and sets --force-older explicitly, so going back is a deliberate decision, never an accident of timing.

1. Reproduce the race. On a test repository we merged two commits a minute apart, with an artificial sleep 180 in the first commit's tests:

The middle row confirmed that serialisation alone was not enough.

2. What is running is always visible. A dashboard panel shows, per service, the deployed revision from the annotation and the digest from the pods. A check alerts if they ever disagree, or if the deployed revision is not the head of main for more than 30 minutes during working hours.

3. Hotfix replay. We re-ran the original sequence of merges on staging with the new workflow. The hotfix stayed deployed, and the feature, being an ancestor of the hotfix commit, was not redeployed after it.

4. Security implications. Deploying by digest also closes a supply chain gap: with a mutable tag, anyone who could push to the registry could change what runs in production on the next pod restart, with no deploy recorded. With digests, only the deploy job changes what runs, and the registry is now configured to make release tags immutable.

Takeaways

• Parallel pipeline runs can finish in any order. Never let "last to finish" decide what is deployed.
• Serialise deploy jobs with a concurrency group, but keep build and test parallel.
• Deploy immutable digests, never mutable tags like main or latest.
• Refuse to deploy a commit that is already contained in what is running, and make rollbacks explicit.

References: GitHub Actions concurrency, git merge-base, Kubernetes images and image pull policy, OCI image annotations.