Helm UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

On a busy release day, a developer cancelled a GitHub Actions run for the checkout service because they had pushed a newer commit. The next run failed at the deploy step, and so did every run after it:

Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Nothing was actually deploying. The cluster showed the old pods running normally.

Reproducing it

helm history told us what Helm believed:

$ helm history checkout -n shop --max 4
REVISION  UPDATED                   STATUS           CHART            APP VERSION  DESCRIPTION
41        Tue Jan 14 10:02:11 2025  superseded       checkout-1.18.0  2.31.0       Upgrade complete
42        Tue Jan 14 15:40:52 2025  deployed         checkout-1.18.0  2.32.0       Upgrade complete
43        Wed Jan 15 09:12:37 2025  pending-upgrade  checkout-1.18.1  2.33.0       Preparing upgrade

Revision 43 was stuck in pending-upgrade. We reproduced it in staging by starting helm upgrade --wait and killing the process after it had written the new release record but before it finished.

What we ruled out

• A concurrent deploy. No other pipeline or person was running Helm; we checked running jobs and asked the team.
• RBAC or cluster access. helm list and kubectl worked with the CI credentials.
• A broken chart. The same chart version installed fine into a fresh namespace.

Root cause

Helm stores each release revision as a Secret in the release namespace. At the start of an upgrade it writes the new revision with status pending-upgrade, applies the manifests, waits if --wait is set, and then marks it deployed. Cancelling the CI job killed the Helm process in between. The pending-upgrade record remained, and Helm refuses to start any new operation while the latest revision is pending, to avoid two operations racing on the same release.

What is happening under the hood

Helm 3 has no server component and no lock service. The "lock" is the status of the latest release Secret, labelled owner=helm, name=<release>, status=<status>, version=<revision>:

kubectl -n shop get secrets -l owner=helm,name=checkout --sort-by=.metadata.creationTimestamp

Statuses pending-install, pending-upgrade and pending-rollback mean "an operation is in progress". If the process dies, nothing changes them back. Common ways to get there: cancelled CI jobs, runner pods evicted mid-deploy, network loss to the API server, and timeouts at the CI level that are shorter than Helm's --timeout.

Environment

Minimal reproduction

helm upgrade --install demo ./chart --wait --timeout 5m &
sleep 5; kill -9 $!            # simulate a cancelled job
helm upgrade --install demo ./chart
# Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

The fix

Recovery first. Before touching anything, we confirmed that no Helm process was running for this release, then checked what was actually in the cluster. Some manifests from revision 43 had been applied (a ConfigMap) and some had not. Rolling back to the last good revision makes the cluster and Helm's record consistent:

helm rollback checkout 42 -n shop --wait --timeout 10m
helm history checkout -n shop --max 3
# 44  deployed  checkout-1.18.0  2.32.0  Rollback to 42

helm rollback works on a release whose latest revision is pending. After that, the next pipeline run upgraded normally to the new version.

Prevention in CI.

concurrency:
  group: deploy-${{ github.workflow }}-checkout-production
  cancel-in-progress: false          # queue, never cancel a running deploy

jobs:
  deploy:
    timeout-minutes: 20              # longer than Helm's own timeout
    steps:
      - name: Helm upgrade
        run: |
          helm upgrade --install checkout ./chart -n shop \
            --atomic --timeout 10m \
            --history-max 20 \
            -f values/production.yaml

• cancel-in-progress: false on the deploy concurrency group means a newer push waits for the current deploy instead of killing it. Build and test jobs can still be cancelled freely.
• --atomic rolls back automatically if the upgrade fails or times out, so a failed upgrade does not leave a half-applied state. It implies --wait.
• The job timeout is longer than Helm's timeout, so Helm always gets to finish or roll back itself.

A guard step before upgrading reports stuck releases clearly instead of failing with the generic error:

status=$(helm status checkout -n shop -o json | jq -r .info.status)
case "$status" in
  pending-*) echo "::error::release is $status; run the recovery runbook"; exit 1 ;;
esac

We chose not to auto-rollback in this guard: if a pending state ever appears again, a human should look at why.

Alternatives and trade-offs

• Delete the pending release Secret. Often suggested; it makes Helm forget revision 43 while its manifests may still be partially applied. Rollback is safer because it re-applies a known state.
• Argo CD or Flux instead of Helm in CI. A controller in the cluster performs the upgrade, so a cancelled CI job cannot interrupt it. A bigger change, and the direction we later took for most services.

How to test the fix

In staging we repeated the kill test against the new pipeline: pushing two commits in a row queued the second deploy until the first finished. Killing a runner pod mid-deploy still left a pending release, which the guard step reported with a link to the runbook, and the runbook's rollback command recovered it.

Confirming in production

• No another operation is in progress errors in the following quarter across 46 services using the shared workflow.
• Two upgrades failed health checks during that time; --atomic rolled both back automatically, and the release history shows Rollback to N entries with the failure reason.

Regression guard

• The shared deploy workflow enforces the concurrency group and timeouts; services cannot override cancel-in-progress for deploy jobs.
• A daily job lists Helm releases in any pending-* status across namespaces and posts them to the platform channel.

Security implications

None directly. One related point: the Helm release Secrets contain the full rendered manifests and values, which can include sensitive configuration. Access to Secrets in deploy namespaces should be limited to the deploy identity and administrators, and --history-max keeps old revisions from piling up.

Prevention checklist

• Never cancel a running deploy; queue newer ones.
• Use --atomic and a Helm timeout shorter than the CI job timeout.
• Check release status before upgrading and fail with a clear message.
• Recover stuck releases with helm rollback, not by deleting Secrets.
• Limit release history with --history-max.

References

• Helm upgrade command — https://helm.sh/docs/helm/helm_upgrade/
• Helm rollback command — https://helm.sh/docs/helm/helm_rollback/
• Helm storage drivers and release Secrets — https://helm.sh/docs/topics/advanced/#storage-backends
• GitHub Actions concurrency — https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/control-the-concurrency-of-workflows-and-jobs