Terraform "Error acquiring the state lock" after a cancelled CI pipeline

Our platform team manages the shared AWS networking and EKS clusters with Terraform, run from GitLab CI. On a Wednesday morning, every merge request touching the platform/prod stack failed at the plan job with the same error:

╷
│ Error: Error acquiring the state lock
│
│ Error message: operation error DynamoDB: PutItem, https response error
│ StatusCode: 400, RequestID: 8Q1M...,
│ ConditionalCheckFailedException: The conditional request failed
│ Lock Info:
│   ID:        2b6a91c4-0f1e-7d55-a3b9-6c0e21d4f8a7
│   Path:      example-tfstate-prod/platform/prod/terraform.tfstate
│   Operation: OperationTypeApply
│   Who:       root@runner-x7kq2-project-311-concurrent-0
│   Version:   1.11.1
│   Created:   2025-03-11 16:52:09.418273 +0000 UTC
│   Info:
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵

The lock was created the previous afternoon by an apply, by a CI runner, and was still there almost 17 hours later.

Reproducing it

Locally, with a throwaway stack using the same backend type:

terraform apply -auto-approve &      # start an apply that takes a while
sleep 20 && kill -9 $!               # kill it hard, the way a cancelled CI job does
terraform plan                       # Error acquiring the state lock

A normal Ctrl+C does not reproduce it: Terraform catches the interrupt, stops gracefully and releases the lock. Only a hard kill leaves it behind.

What we ruled out

• A run still in progress. This was the important one, and we checked it before anything else (see below).
• IAM permission problems on DynamoDB. A missing permission gives AccessDeniedException, not ConditionalCheckFailedException. The conditional failure means the lock item already exists.
• Clock skew or a second backend config. Every job used the same backend.hcl, and the Path matched the state we expected.

Root cause

We matched the lock's Created timestamp to the GitLab job history. At 16:51 the day before, a developer pushed a new commit to the same merge request; the project had auto-cancel redundant pipelines enabled and the apply job was marked interruptible: true in a shared template. GitLab cancelled the running pipeline, the runner removed the job container, and terraform apply died without any chance to clean up. The DynamoDB lock item stayed.

The job log ended mid-resource:

module.eks.aws_eks_node_group.general["b"]: Still modifying... [id=prod-general-b, 3m10s elapsed]
module.eks.aws_eks_node_group.general["b"]: Still modifying... [id=prod-general-b, 3m20s elapsed]
ERROR: Job failed: canceled

So we had two problems: a stale lock, and a state file that might not reflect what AWS actually did in that last minute.

What is happening under the hood

With the S3 backend, Terraform stores state as an object in S3 and, when locking is enabled, takes a lock before any operation that could write state. With the DynamoDB method, the lock is an item whose key is the state path; Terraform writes it with a condition "only if no item exists". With the S3-native method (use_lockfile = true, generally available since Terraform 1.11), the lock is a <key>.tflock object created with a conditional write. Either way, Terraform deletes the lock when the operation ends normally or after a graceful interrupt.

Nothing expires a lock automatically. If the process is killed (SIGKILL, a terminated container, a lost runner VM), the lock remains until someone removes it. That is deliberate: Terraform cannot tell a dead run from a slow one, and guessing wrong would let two applies write state concurrently.

When an apply is killed, Terraform also loses the chance to persist the latest state. Resources created or modified after the last state write can exist in AWS but be missing or stale in state.

Environment

Minimal reproduction

terraform {
  backend "s3" {
    bucket         = "example-tfstate-prod"
    key            = "platform/prod/terraform.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

# .gitlab-ci.yml (before)
apply:
  stage: deploy
  interruptible: true
  script:
    - terraform init -backend-config=backend.hcl
    - terraform apply -auto-approve plan.tfplan

Who is affected

Every Terraform and OpenTofu backend with locking: S3 (DynamoDB or lockfile), GCS, azurerm, Consul, HCP Terraform workspaces. It appears whenever the process is killed rather than interrupted: cancelled CI jobs, spot runner reclamation, OOM-killed containers, a closed laptop lid on a VPN that drops.

The fix

1. Prove no run is active. We checked GitLab for any running job on the project, confirmed the runner named in Who no longer existed, and looked at CloudTrail for API calls from the CI role in the last hour. Nothing. Only then:

terraform force-unlock 2b6a91c4-0f1e-7d55-a3b9-6c0e21d4f8a7

Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.
  This will allow local Terraform commands to modify this state, even though it
  may still be in use. Only 'yes' will be accepted to confirm.

  Enter a value: yes

Terraform state has been successfully unlocked!

force-unlock only removes the lock for the given ID, so it cannot remove a newer lock by accident.

2. Reconcile state with reality. We ran a refresh-only plan first:

terraform plan -refresh-only

It showed the general-b node group with a new launch template version already applied by AWS but not recorded in state. We accepted it with terraform apply -refresh-only, then ran a normal plan, which was clean apart from the change the merge request intended. If Terraform had failed to save state at the end of a crashed run, it would have written an errored.tfstate file in the working directory; with a destroyed CI container that file is gone, which is another reason not to let CI kill applies.

3. Stop CI from killing applies.

 apply:
   stage: deploy
-  interruptible: true
+  interruptible: false
+  resource_group: tf-platform-prod
   script:
     - terraform init -backend-config=backend.hcl
-    - terraform apply -auto-approve plan.tfplan
+    - terraform apply -lock-timeout=10m -auto-approve plan.tfplan

interruptible: false keeps the apply running when a newer pipeline starts. resource_group makes GitLab run only one job per group at a time, so two applies queue instead of colliding on the lock. -lock-timeout=10m makes a plan wait for a lock briefly instead of failing instantly.

4. Move to S3-native locking. With Terraform 1.11 we enabled the lockfile alongside DynamoDB for one release, then removed the table:

 backend "s3" {
   bucket         = "example-tfstate-prod"
   key            = "platform/prod/terraform.tfstate"
   region         = "eu-west-1"
-  dynamodb_table = "terraform-locks"
+  use_lockfile   = true
   encrypt        = true
 }

This does not prevent stale locks, but it removes a table and its IAM policy, and the stale lock becomes a visible terraform.tfstate.tflock object next to the state.

Alternatives and trade-offs

• -lock=false. Never for apply on shared state. It is acceptable only for read-only commands in an emergency, and even then plan -lock=false can be misleading while another apply runs.
• Deleting the DynamoDB item by hand. Same effect as force-unlock, but without the safety of matching the lock ID. Prefer the command.
• HCP Terraform or a Terraform-aware runner (Atlantis, Spacelift). They serialize runs per workspace and handle cancellation more carefully, at the cost of another system to run.

How to test the fix

We repeated the incident on a staging stack: started an apply, pushed a new commit to the merge request, and checked that GitLab let the running apply finish and queued the new pipeline's apply behind it:

glab ci view           # shows apply "running" and the new apply "waiting for resource"
aws s3 ls s3://example-tfstate-staging/platform/staging/
# terraform.tfstate.tflock appears during apply and disappears afterwards

Confirming in production

• The platform/prod plan job passed on all four queued merge requests after the unlock.
• terraform plan after the refresh-only apply showed only intended changes; we compared aws eks describe-nodegroup output for every node group against state and found no other drift.
• Over the next month, 214 pipelines ran on the stack with auto-cancel still enabled for other jobs; zero stale locks.

Regression guard

• A scheduled job checks for .tflock objects older than two hours and posts to the platform channel.
• The shared CI template now forbids interruptible: true on any job that runs apply or destroy.
• Runbook: before any force-unlock, the on-call engineer must confirm no running job and record the lock ID in the incident ticket.

Security implications

force-unlock is a privileged operation in practice: whoever can run it can let two writers corrupt production state. We restricted the IAM policy for deleting *.tflock objects (and the DynamoDB item, before the migration) to the CI role and the break-glass role. The state bucket keeps versioning and encryption on, so a bad write can be rolled back to a previous version.

Prevention checklist

• Never let CI cancel or time out a running apply or destroy.
• Serialize applies per state with resource_group in GitLab or concurrency in GitHub Actions.
• Use -lock-timeout so plans wait briefly instead of failing.
• Before force-unlock, prove the lock holder is dead, then run a refresh-only plan.
• Enable versioning on the state bucket.

References

• Terraform documentation: S3 backend (state locking, use_lockfile) — https://developer.hashicorp.com/terraform/language/backend/s3
• Terraform documentation: State locking — https://developer.hashicorp.com/terraform/language/state/locking
• Terraform documentation: terraform force-unlock — https://developer.hashicorp.com/terraform/cli/commands/force-unlock
• Terraform 1.11 changelog — https://github.com/hashicorp/terraform/blob/v1.11/CHANGELOG.md
• GitLab documentation: CI/CD YAML syntax, interruptible and resource_group — https://docs.gitlab.com/ee/ci/yaml/