Terraform init fails in CI: package doesn't match checksums in the dependency lock file
After a provider upgrade done on a MacBook, terraform init failed on Linux CI runners with a checksum mismatch against the lock file. The lock file only had hashes for darwin_arm64 packages. Locking all platforms we use fixed it without weakening verification.
A teammate upgraded the AWS provider on their laptop, ran terraform init -upgrade, opened a pull request with the updated .terraform.lock.hcl, and the plan job failed on our Linux runners:
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using hashicorp/aws v5.89.0 from the shared cache directory
╷
│ Error: Failed to install provider from shared cache
│
│ Error while importing hashicorp/aws v5.89.0 from the shared cache directory: the provider cache
│ at .terraform/providers has a copy of registry.terraform.io/hashicorp/aws 5.89.0 that doesn't
│ match any of the checksums recorded in the dependency lock file.
╵
Without the plugin cache the error text differed slightly but said the same thing: the downloaded package did not match the recorded hashes.
Reproducing it
The diff of the lock file in the pull request:
provider "registry.terraform.io/hashicorp/aws" {
- version = "5.84.0"
+ version = "5.89.0"
constraints = "~> 5.80"
hashes = [
- "h1:...linux...",
- "h1:...darwin...",
- "zh:...", # many lines
+ "h1:Q8m1...darwin_arm64...",
+ "zh:0c4e...",
+ "zh:...",
]
}
Before the upgrade, the file had h1: hashes for several platforms. After the laptop upgrade, it had one h1: hash, for the laptop's platform.
What we ruled out
- A corrupt download or registry issue. Re-running in CI gave the same error; a fresh cache directory too.
- A malicious package. The package checksum matched the provider's published
SHA256SUMSfor linux_amd64. The lock file was what was incomplete. - Version constraints.
~> 5.80allowed 5.89.0.
Root cause
When terraform init records a new provider version, it adds zh: hashes (from the registry's signed checksum list for zip files) and an h1: hash computed from the package it actually installed, which is only for the current platform. When the plugin cache is in use or the package must be verified against h1: hashes, Linux runners cannot find a matching h1: for linux_amd64 and refuse to proceed. The lock file needs hashes for every platform that will run Terraform.
What is happening under the hood
.terraform.lock.hcl pins provider versions and the checksums Terraform accepts for them. Two hash schemes appear:
zh:is the SHA-256 of the provider's release zip archive, taken from the signed checksums file published with the release. One per platform archive.h1:is a hash of the unpacked package contents. Terraform computes it locally for what it installed, and it is what cache directories are checked against.
terraform init only records h1: for the platform it is running on. terraform providers lock can fetch packages for any list of platforms and record their hashes, which is how you make one lock file work across macOS laptops and Linux CI.
Environment
| Component | Version / value |
|---|---|
| Terraform | 1.10.5 |
| Provider | hashicorp/aws 5.84.0 to 5.89.0 |
| Developer machines | macOS on Apple silicon (darwin_arm64), some Intel (darwin_amd64) |
| CI | GitHub Actions, Ubuntu (linux_amd64) and ARM runners (linux_arm64) |
| Plugin cache | TF_PLUGIN_CACHE_DIR set in CI |
Minimal reproduction
# on macOS arm64
terraform init -upgrade # writes lock with h1 for darwin_arm64 only
git commit -am "bump aws provider"
# on linux amd64 CI with TF_PLUGIN_CACHE_DIR set
terraform init # checksum mismatch
The fix
Regenerate the lock entries for every platform we run on, and commit them:
terraform providers lock \
-platform=linux_amd64 \
-platform=linux_arm64 \
-platform=darwin_amd64 \
-platform=darwin_arm64
The command downloads each platform's package, verifies it against the provider's signed checksums, and records h1: hashes for all of them. The lock file now has four h1: lines per provider and the same zh: list.
To make this automatic, we added a Make target and a CI check:
PLATFORMS := linux_amd64 linux_arm64 darwin_amd64 darwin_arm64
lock:
terraform providers lock $(foreach p,$(PLATFORMS),-platform=$(p))
- name: Lock file covers all platforms
run: |
make lock
git diff --exit-code .terraform.lock.hcl || {
echo "::error::Run 'make lock' and commit .terraform.lock.hcl"; exit 1; }
Dependabot and Renovate can also update Terraform providers; we configured Renovate to run the multi-platform lock after updates.
Alternatives and trade-offs
- Delete the lock file in CI or use
-upgradein CI. Defeats the point of locking; CI could install a different version than the one reviewed. - Disable the plugin cache. Changes the error path but not the problem when hashes are missing; slower.
- Only develop on Linux (dev containers). Valid, but laptops still run Terraform for quick checks.
How to test the fix
rm -rf .terraform "$TF_PLUGIN_CACHE_DIR"/*
terraform init -lockfile=readonly
-lockfile=readonly makes init fail if it would need to change the lock file, which is exactly what CI should enforce. It passed on all four runner types.
Confirming in production
- Plan and apply jobs passed on Linux amd64 and arm64 runners for the upgrade PR after regenerating the lock.
- Over the following two months, eight provider upgrades merged without checksum failures.
Regression guard
terraform init -lockfile=readonlyin every CI job.- The "lock covers all platforms" check on pull requests that change
.terraform.lock.hclor provider constraints. - A short note in the repository README explaining
make lock.
Security implications
The checksum mismatch is a security feature working as designed: Terraform refused a package it could not verify against the lock file. The wrong fix is to remove verification, for example by deleting the lock file in CI. The right fix records verified hashes for every platform, keeping the supply chain protection: a tampered provider package for any platform will be rejected. Reviewers should treat lock file changes as dependency changes and check that version bumps are intended.
Prevention checklist
- Run
terraform providers lockwith all platforms after any provider change. - Use
-lockfile=readonlyin CI. - Never delete or regenerate the lock file inside CI.
- Review lock file diffs like dependency upgrades.
- Document the platforms your team uses.
References
- Terraform dependency lock file — https://developer.hashicorp.com/terraform/language/files/dependency-lock
- terraform providers lock command — https://developer.hashicorp.com/terraform/cli/commands/providers/lock
- Provider plugin cache — https://developer.hashicorp.com/terraform/cli/config/config-file#provider-plugin-cache
- terraform init -lockfile option — https://developer.hashicorp.com/terraform/cli/commands/init
Add a label
Embed this post
<iframe src="https://hibug.me/terraform-lock-file-checksum-mismatch-platforms" width="700" height="420" frameborder="0" loading="lazy"></iframe>
I build and run the platforms other engineers ship on: CI/CD pipelines, Kubernetes clusters, GPU inference services, LLM features in production, and the cloud bills that come with all of it. I have also spent years writing and operating automated trading systems, where a rounding bug or a missed websocket message is not a log line but a position you did not intend to hold. Each post here starts from a real failure: the alert, the exact error, what we ruled out, the root cause, and how we proved the fix. Same path every time: Investigate, Explain, Fix, Verify. No hype, no price predictions, just engineering.
Comments
0No comments yet. Be the first to comment!
Sign In to post a comment.