Kubernetes ConfigMap updated but pods still use old values

During an abuse spike, we lowered the per-client rate limit for our public API from 600 to 120 requests per minute. The change was a one-line edit to a ConfigMap in the GitOps repository. Argo CD synced it within a minute and showed green. Two days later, during the next spike, the API was still allowing 600 requests per minute per client.

Reproducing it

The ConfigMap in the cluster had the new value:

$ kubectl -n api get configmap api-config -o jsonpath='{.data.RATE_LIMIT_PER_MINUTE}'
120

The running container did not:

$ kubectl -n api exec deploy/api -- printenv RATE_LIMIT_PER_MINUTE
600

A second setting, the CORS allowlist, was mounted as a file, and it was also stale:

$ kubectl -n api exec deploy/api -- cat /etc/api/cors.yaml | head -2
allowed_origins:
  - https://old.example.com

What we ruled out

• Argo CD sync failure. The ConfigMap object was correct in the cluster.
• A cached image with baked-in config. The image had no config file; the values came from the ConfigMap.
• Wrong namespace or ConfigMap name. kubectl describe pod showed the right ConfigMap references.

Root cause

Changing a ConfigMap does not restart pods. Two mechanisms were involved. Environment variables populated from a ConfigMap (envFrom or valueFrom) are resolved once when the container starts and never change. The CORS file was mounted with subPath, and files mounted with subPath are not updated when the ConfigMap changes; only full volume mounts are refreshed by the kubelet. Since the Deployment spec did not change, there was no rollout, and the old values lived on until the pods restarted for unrelated reasons.

What is happening under the hood

Even with a refreshing volume mount, the application must notice the file change and reload it. Many applications read config once at startup.

A Deployment only rolls out new pods when its pod template changes. A ConfigMap is a separate object, so editing it does not touch the template.

Environment

Minimal reproduction

containers:
  - name: api
    envFrom:
      - configMapRef: {name: api-config}         # read once
    volumeMounts:
      - name: config
        mountPath: /etc/api/cors.yaml
        subPath: cors.yaml                        # never refreshed

Change any value in api-config, wait, and check the container: nothing changes until the pod restarts.

The fix

1. Roll pods when config changes, using a checksum annotation in the pod template. This is the standard Helm pattern:

# templates/deployment.yaml
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

When any value in the ConfigMap template changes, the hash changes, the pod template changes, and the Deployment performs a normal rolling update with readiness checks and the usual rollback behaviour.

2. Remove subPath where hot reload is wanted. For the CORS file, we mount the whole ConfigMap into a directory:

volumeMounts:
  - name: config
    mountPath: /etc/api/config        # directory; files update in place
volumes:
  - name: config
    configMap: {name: api-config}

The app now watches the directory with fsnotify and reloads CORS rules when the file changes. Kubernetes updates projected files by swapping a symlink, so the watcher listens for events on the directory, not on the file itself.

3. Immutable, versioned ConfigMaps for critical settings. For values where a gradual reload would be confusing, like the rate limit, we let the rollout handle it instead of hot reload: one version of config per ReplicaSet, visible in kubectl rollout history.

Alternatives and trade-offs

• A reloader controller (for example Stakater Reloader) that watches ConfigMaps and Secrets and restarts annotated Deployments. Works for config not managed by Helm templates, such as ConfigMaps created by other tools.
• kubectl rollout restart by hand or in CI after config changes. Easy to forget, which is how we got here.
• Kustomize configMapGenerator creates ConfigMaps with a content hash in the name and updates references, which triggers rollouts naturally. Good for Kustomize users.

How to test the fix

# change a value in values.yaml, render, and compare annotations
helm template api ./chart -f values/prod.yaml | yq 'select(.kind=="Deployment").spec.template.metadata.annotations'

In staging, after merging a config change, Argo CD synced and the Deployment rolled:

$ kubectl -n api rollout history deploy/api | tail -2
14   <none>
15   <none>
$ kubectl -n api exec deploy/api -- printenv RATE_LIMIT_PER_MINUTE
120

For the CORS hot reload, we edited only cors.yaml (which is excluded from the checksum), and the app logged cors config reloaded within about 70 seconds.

Confirming in production

• The rate limit change rolled out within three minutes of merge, with the usual surge and readiness gates.
• The application exposes a /debug/config-version endpoint (internal only) that returns the hash of loaded config; all pods report the same hash as the annotation after each rollout.

Regression guard

• A chart lint rule requires a checksum/config annotation on every Deployment that references a ConfigMap from the same chart.
• subPath mounts for ConfigMaps are flagged in review unless the file is intentionally static.
• A dashboard panel compares config hash per pod; mismatches for more than 10 minutes alert.

Security implications

This bug turned a security control into a no-op: we believed the lower rate limit was active during an abuse incident when it was not. Configuration that enforces security (rate limits, allowlists, feature kill switches) needs a way to verify that running processes use the intended values, not just that the object in the cluster is correct. The config hash endpoint gives us that.

Prevention checklist

• Remember that env vars from ConfigMaps never update in running containers.
• Avoid subPath for config that should change at runtime.
• Add a config checksum annotation so changes trigger rollouts.
• Expose the loaded config version and compare it across pods.
• Treat security-relevant config changes as deploys, with verification.

References

• Kubernetes ConfigMaps, mounted ConfigMaps are updated automatically — https://kubernetes.io/docs/concepts/configuration/configmap/#mounted-configmaps-are-updated-automatically
• Kubernetes volumes, using subPath — https://kubernetes.io/docs/concepts/storage/volumes/#using-subpath
• Helm tips and tricks, automatically roll deployments — https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
• Stakater Reloader — https://github.com/stakater/Reloader