CI passed with stale node_modules: a cache key without the lockfile hash

On a Monday our booking widget's API started returning 500s for about 6 percent of requests right after a routine deploy. The stack trace pointed into request validation:

TypeError: Cannot read properties of undefined (reading '_zod')
    at Object.parse (/app/node_modules/@example/booking-schemas/dist/index.js:41:22)
    at validateBody (/app/dist/middleware/validate.js:18:31)

The deploy contained a single merged pull request from our dependency bot: a lockfile maintenance update that refreshed transitive versions inside package-lock.json without touching package.json. That pull request had a green CI run with 1,412 passing tests. Reverting it fixed production.

So the same commit passed CI and failed in production. Something in CI was not testing what we deployed.

Reproducing it

Locally, a clean install of the bot's commit failed the validation tests immediately:

git checkout 4f1c2ab
rm -rf node_modules && npm ci && npm test -- validate
# FAIL  test/validate.test.ts
#   TypeError: Cannot read properties of undefined (reading '_zod')

Re-running the CI job with the cache bypassed (we temporarily changed the cache key prefix) failed the same way.

What we ruled out

• Different Node versions. Both CI and the production image used Node 22.17.
• A flaky test. The failing tests were deterministic once the right dependencies were installed.
• The production image build. The Dockerfile runs npm ci from the lockfile in a clean stage, which is exactly why production got the new versions.

Root cause

The CI workflow cached node_modules directly and skipped installation on a cache hit:

- uses: actions/cache@v4
  id: deps
  with:
    path: node_modules
    key: deps-${{ runner.os }}-${{ hashFiles('**/package.json') }}
    restore-keys: |
      deps-${{ runner.os }}-

- if: steps.deps.outputs.cache-hit != 'true'
  run: npm ci

The key hashed package.json files only. A lockfile-only update leaves every package.json unchanged, so the key was identical to the previous run's, the cache reported an exact hit, and npm ci was skipped. CI ran the tests against the node_modules from weeks earlier. The lockfile update had moved our internal @example/booking-schemas package onto a newer zod major through a transitive range, and only the production build saw it.

What is happening under the hood

actions/cache looks up a cache entry by key. On an exact match it restores the files and sets the cache-hit output to true. If there is no exact match it tries each restore-keys prefix in order and restores the most recent entry that starts with it; cache-hit is then false. At the end of the job, if there was no exact hit, it saves the path under key.

Two consequences matter here:

1. The key defines what "same dependencies" means. If the lockfile is not part of the key, two different dependency trees share a key, and the first one saved wins.
2. Cache entries are immutable. Once saved, a key is never overwritten. A stale entry stays until it is evicted (unused for 7 days, or when the repository exceeds its cache size limit) or deleted.

restore-keys adds a third trap. A prefix match restores some older node_modules. If the workflow then runs npm install instead of npm ci, npm may keep packages that are already present and satisfy the ranges, so the result depends on whatever cache happened to be restored. npm ci deletes node_modules first, which makes restoring it pointless.

Environment

Minimal reproduction

on: push
jobs:
  test:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - uses: actions/cache@v4
        id: deps
        with:
          path: node_modules
          key: deps-${{ runner.os }}-${{ hashFiles('**/package.json') }}
      - if: steps.deps.outputs.cache-hit != 'true'
        run: npm ci
      - run: npm ls zod   # prints the cached version even after a lockfile-only change

Push once, then run npm update zod (which edits only the lockfile when the range already allows it), push again, and npm ls in CI still prints the old version.

Who is affected

Any CI system with key-based caches (GitHub Actions, GitLab CI cache:key:files, CircleCI save_cache, Bitbucket) when the key omits the lockfile, or when an install step is skipped based on a fuzzy match. The same bug happens with vendor/ keyed on composer.json instead of composer.lock, Python virtualenvs keyed on requirements.in or pyproject.toml instead of the lock, and Go module caches keyed on go.mod without go.sum.

The fix

We stopped caching node_modules altogether and cached npm's download cache instead, which is what actions/setup-node does for you:

       - uses: actions/setup-node@v4
         with:
           node-version-file: .nvmrc
+          cache: npm
+          cache-dependency-path: package-lock.json
-      - uses: actions/cache@v4
-        id: deps
-        with:
-          path: node_modules
-          key: deps-${{ runner.os }}-${{ hashFiles('**/package.json') }}
-          restore-keys: |
-            deps-${{ runner.os }}-
-      - if: steps.deps.outputs.cache-hit != 'true'
-        run: npm ci
+      - run: npm ci

setup-node caches ~/.npm with a key that includes the lockfile hash. npm ci always runs, always installs exactly what the lockfile says, and reuses downloaded tarballs from the cache, so it stays fast. The cache can only make installation faster; it can no longer change what gets installed.

Install time went from about 4 seconds on a stale hit (and 95 seconds on a miss) to a consistent 22 seconds.

If you must cache node_modules

Some large repositories do need to skip installation. Then the key must include everything that changes the installed tree, and there must be no restore-keys:

# the setup-node step needs id: node so its node-version output is available
- uses: actions/cache@v4
  id: deps
  with:
    path: |
      node_modules
      packages/*/node_modules
    key: nm-${{ runner.os }}-${{ runner.arch }}-node${{ steps.node.outputs.node-version }}-${{ hashFiles('package-lock.json', '.npmrc') }}
- if: steps.deps.outputs.cache-hit != 'true'
  run: npm ci

Node version and architecture belong in the key because native modules are compiled for them.

Alternatives and trade-offs

• restore-keys with npm ci. Harmless but useless for node_modules, since npm ci wipes it. It is useful for ~/.npm, where a partial cache still saves downloads.
• Cache per branch. GitHub already scopes caches by branch with fallback to the default branch. Adding the branch to the key does not fix a key that ignores the lockfile.
• Turborepo or Nx remote caches. These cache task outputs, keyed on inputs including the lockfile. Good for build outputs, but they do not replace a correct install step.

How to test the fix

We repeated the failure on a branch: lockfile-only update, push, and check the installed version in CI matches the lockfile:

# in the workflow, after npm ci
node -e "const l=require('./package-lock.json');console.log(l.packages['node_modules/zod'].version)"
npm ls zod --depth=0 --all

Both print the same version on every run now. The bot's original pull request, re-run with the new workflow, failed the validation tests as it should have.

Confirming in production

• The bot's update was fixed properly (we pinned the schema package to a compatible zod and updated our validation code), passed CI with a real install, and deployed without errors.
• Over the next 40 dependency pull requests, CI failed 3 of them on real incompatibilities. Before the change, the bot's pull requests had a 100 percent pass rate, which in hindsight was the warning sign.
• Total CI time per pull request went up by about 18 seconds on average. We accepted that.

Regression guard

• A CI step compares npm ls --all --json against the lockfile and fails on any mismatch.
• The workflow lint job rejects actions/cache steps whose path includes node_modules or vendor unless the key contains a hashFiles call on the matching lockfile.
• Weekly lockfile maintenance pull requests must show a full install in the job log.

Security implications

A stale cache means CI tests code that will not ship, so security patches that arrive through lockfile updates were never actually tested, and a malicious version could pass review because CI ran against the old one. Caches are also writable by any workflow run on the branch, so treat them as untrusted: never cache credentials, and do not restore caches from pull request runs into release jobs.

Prevention checklist

• Hash the lockfile, not the manifest, in every dependency cache key.
• Cache the package manager's download cache, and always run the clean install command.
• Avoid restore-keys for installed dependency directories.
• Include runtime version and architecture in keys for anything with native modules.
• Be suspicious of a dependency bot whose pull requests never fail.

References

• actions/cache README: inputs, outputs and cache-hit — https://github.com/actions/cache
• actions/setup-node: Caching global packages data — https://github.com/actions/setup-node#caching-global-packages-data
• GitHub Docs: Caching dependencies to speed up workflows — https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows
• npm documentation: npm ci — https://docs.npmjs.com/cli/v10/commands/npm-ci
• npm documentation: package-lock.json — https://docs.npmjs.com/cli/v10/configuring-npm/package-lock-json