DevOps & CI/CD Bug Report

Terraform init fails in CI: package doesn't match checksums in the dependency lock file

After a provider upgrade done on a MacBook, terraform init failed on Linux CI runners with a checksum mismatch against the lock file. The lock file only had hashes for darwin_arm64 packages. Locking all platforms we use fixed it without weakening verification.

Ryan Pham Ryan Pham @ryan_bugs · March 05, 2025 · 6 min read · 7 Views
Banner: Terraform provider checksum mismatch because the lock file lacks Linux hashes

A teammate upgraded the AWS provider on their laptop, ran terraform init -upgrade, opened a pull request with the updated .terraform.lock.hcl, and the plan job failed on our Linux runners:

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using hashicorp/aws v5.89.0 from the shared cache directory
╷
│ Error: Failed to install provider from shared cache
│
│ Error while importing hashicorp/aws v5.89.0 from the shared cache directory: the provider cache
│ at .terraform/providers has a copy of registry.terraform.io/hashicorp/aws 5.89.0 that doesn't
│ match any of the checksums recorded in the dependency lock file.
╵

Without the plugin cache the error text differed slightly but said the same thing: the downloaded package did not match the recorded hashes.

Reproducing it

The diff of the lock file in the pull request:

 provider "registry.terraform.io/hashicorp/aws" {
-  version     = "5.84.0"
+  version     = "5.89.0"
   constraints = "~> 5.80"
   hashes = [
-    "h1:...linux...",
-    "h1:...darwin...",
-    "zh:...",  # many lines
+    "h1:Q8m1...darwin_arm64...",
+    "zh:0c4e...",
+    "zh:...",
   ]
 }

Before the upgrade, the file had h1: hashes for several platforms. After the laptop upgrade, it had one h1: hash, for the laptop's platform.

What we ruled out

  • A corrupt download or registry issue. Re-running in CI gave the same error; a fresh cache directory too.
  • A malicious package. The package checksum matched the provider's published SHA256SUMS for linux_amd64. The lock file was what was incomplete.
  • Version constraints. ~> 5.80 allowed 5.89.0.

Root cause

When terraform init records a new provider version, it adds zh: hashes (from the registry's signed checksum list for zip files) and an h1: hash computed from the package it actually installed, which is only for the current platform. When the plugin cache is in use or the package must be verified against h1: hashes, Linux runners cannot find a matching h1: for linux_amd64 and refuse to proceed. The lock file needs hashes for every platform that will run Terraform.

What is happening under the hood

.terraform.lock.hcl pins provider versions and the checksums Terraform accepts for them. Two hash schemes appear:

  • zh: is the SHA-256 of the provider's release zip archive, taken from the signed checksums file published with the release. One per platform archive.
  • h1: is a hash of the unpacked package contents. Terraform computes it locally for what it installed, and it is what cache directories are checked against.

terraform init only records h1: for the platform it is running on. terraform providers lock can fetch packages for any list of platforms and record their hashes, which is how you make one lock file work across macOS laptops and Linux CI.

Environment

Component Version / value
Terraform 1.10.5
Provider hashicorp/aws 5.84.0 to 5.89.0
Developer machines macOS on Apple silicon (darwin_arm64), some Intel (darwin_amd64)
CI GitHub Actions, Ubuntu (linux_amd64) and ARM runners (linux_arm64)
Plugin cache TF_PLUGIN_CACHE_DIR set in CI

Minimal reproduction

# on macOS arm64
terraform init -upgrade                 # writes lock with h1 for darwin_arm64 only
git commit -am "bump aws provider"
# on linux amd64 CI with TF_PLUGIN_CACHE_DIR set
terraform init                          # checksum mismatch

The fix

Regenerate the lock entries for every platform we run on, and commit them:

terraform providers lock \
  -platform=linux_amd64 \
  -platform=linux_arm64 \
  -platform=darwin_amd64 \
  -platform=darwin_arm64

The command downloads each platform's package, verifies it against the provider's signed checksums, and records h1: hashes for all of them. The lock file now has four h1: lines per provider and the same zh: list.

To make this automatic, we added a Make target and a CI check:

PLATFORMS := linux_amd64 linux_arm64 darwin_amd64 darwin_arm64

lock:
	terraform providers lock $(foreach p,$(PLATFORMS),-platform=$(p))
- name: Lock file covers all platforms
  run: |
    make lock
    git diff --exit-code .terraform.lock.hcl || {
      echo "::error::Run 'make lock' and commit .terraform.lock.hcl"; exit 1; }

Dependabot and Renovate can also update Terraform providers; we configured Renovate to run the multi-platform lock after updates.

Alternatives and trade-offs

  • Delete the lock file in CI or use -upgrade in CI. Defeats the point of locking; CI could install a different version than the one reviewed.
  • Disable the plugin cache. Changes the error path but not the problem when hashes are missing; slower.
  • Only develop on Linux (dev containers). Valid, but laptops still run Terraform for quick checks.

How to test the fix

rm -rf .terraform "$TF_PLUGIN_CACHE_DIR"/*
terraform init -lockfile=readonly

-lockfile=readonly makes init fail if it would need to change the lock file, which is exactly what CI should enforce. It passed on all four runner types.

Confirming in production

  • Plan and apply jobs passed on Linux amd64 and arm64 runners for the upgrade PR after regenerating the lock.
  • Over the following two months, eight provider upgrades merged without checksum failures.

Regression guard

  • terraform init -lockfile=readonly in every CI job.
  • The "lock covers all platforms" check on pull requests that change .terraform.lock.hcl or provider constraints.
  • A short note in the repository README explaining make lock.

Security implications

The checksum mismatch is a security feature working as designed: Terraform refused a package it could not verify against the lock file. The wrong fix is to remove verification, for example by deleting the lock file in CI. The right fix records verified hashes for every platform, keeping the supply chain protection: a tampered provider package for any platform will be rejected. Reviewers should treat lock file changes as dependency changes and check that version bumps are intended.

Prevention checklist

  • Run terraform providers lock with all platforms after any provider change.
  • Use -lockfile=readonly in CI.
  • Never delete or regenerate the lock file inside CI.
  • Review lock file diffs like dependency upgrades.
  • Document the platforms your team uses.

References

Add a label

#ci #github-actions #terraform #lockfile #providers
Written by
Ryan Pham @ryan_bugs

I build and run the platforms other engineers ship on: CI/CD pipelines, Kubernetes clusters, GPU inference services, LLM features in production, and the cloud bills that come with all of it. I have also spent years writing and operating automated trading systems, where a rounding bug or a missed websocket message is not a log line but a position you did not intend to hold. Each post here starts from a real failure: the alert, the exact error, what we ruled out, the root cause, and how we proved the fix. Same path every time: Investigate, Explain, Fix, Verify. No hype, no price predictions, just engineering.

Advertisement

Comments

0

No comments yet. Be the first to comment!

Sign In to post a comment.