ImagePullBackOff from ECR after 12 hours: no basic auth credentials

We run a small self-managed k3s cluster on EC2 for internal tools, pulling images from Amazon ECR. After setting it up, deploys worked all afternoon. The next morning, any pod that needed a new image was stuck:

$ kubectl -n tools describe pod report-api-6f9c7d8b5-ztq2x
...
  Warning  Failed     12s   kubelet  Failed to pull image "111122223333.dkr.ecr.eu-west-1.amazonaws.com/report-api:1.4.2":
           failed to pull and unpack image "...": failed to resolve reference "...":
           pull access denied, repository does not exist or may require authorization:
           authorization failed: no basic auth credentials
  Warning  Failed     12s   kubelet  Error: ErrImagePull
  Normal   BackOff    1s    kubelet  Back-off pulling image "...": ImagePullBackOff

Pods with images already cached on the node kept running, which made the failure look random.

Reproducing it

The setup guide we had followed created the pull secret like this:

kubectl -n tools create secret docker-registry ecr \
  --docker-server=111122223333.dkr.ecr.eu-west-1.amazonaws.com \
  --docker-username=AWS \
  --docker-password="$(aws ecr get-login-password --region eu-west-1)"

Decoding the secret and checking the token's expiry made it obvious: it was created at 14:05 the previous day and had expired at 02:05.

What we ruled out

• IAM permissions. The node role had AmazonEC2ContainerRegistryReadOnly; aws ecr get-login-password worked from the node.
• Repository policy. No cross-account access involved; same account.
• Image tag missing. aws ecr describe-images showed the tag.

Root cause

ECR authorization tokens are valid for 12 hours. The docker-registry secret is a static copy of one token. Once it expired, every pull that needed authentication failed. Managed EKS nodes avoid this because the kubelet uses a credential provider that fetches fresh tokens with the node's IAM role. Our self-managed nodes had no such provider configured.

What is happening under the hood

When a pod references an image in a private registry, the kubelet needs credentials. It looks in two places: imagePullSecrets on the pod or service account, and kubelet credential providers configured on the node. Credential providers are plugins the kubelet executes to get short-lived credentials for matching image patterns, and the kubelet caches the result for the duration the plugin returns.

AWS publishes ecr-credential-provider, which uses the node's IAM credentials (instance profile) to call ecr:GetAuthorizationToken and returns a fresh token. On EKS-optimised AMIs it is already installed and configured. On k3s, kubeadm, or other self-managed clusters, you install the binary and point the kubelet at a config file.

Environment

Minimal reproduction

Create the pull secret from get-login-password, wait 12 hours, then deploy a pod with an image not cached on the node. The pull fails with no basic auth credentials.

The fix

We configured the kubelet credential provider on every node.

1. Install the binary (built from the cloud-provider-aws repository or taken from a release artifact), into a directory for credential provider plugins:

sudo install -m 0755 ecr-credential-provider /var/lib/rancher/credentialprovider/bin/ecr-credential-provider

2. Write the provider config:

# /var/lib/rancher/credentialprovider/config.yaml
apiVersion: kubelet.config.k8s.io/v1
kind: CredentialProviderConfig
providers:
  - name: ecr-credential-provider
    matchImages:
      - "*.dkr.ecr.*.amazonaws.com"
    defaultCacheDuration: "6h"
    apiVersion: credentialprovider.kubelet.k8s.io/v1

3. Point the kubelet at it. On k3s, the default locations above are picked up automatically when the files exist; on other distributions, set the kubelet flags:

--image-credential-provider-config=/etc/kubernetes/credential-provider/config.yaml
--image-credential-provider-bin-dir=/usr/local/bin/credential-provider

4. Remove the static secrets from namespaces and service accounts, so there is one source of credentials and nothing silently expires.

The node role keeps read-only ECR access. For nodes that should only pull specific repositories, we scoped the policy to those repository ARNs.

Alternatives and trade-offs

• A CronJob that refreshes the secret every few hours. Works, but it is another moving part with its own credentials, and if it fails you are back to expiring tokens.
• External Secrets Operator with an ECR token generator. Neat if you already run ESO; still secret-based.
• Pull-through cache or a registry inside the cluster network. Different trade-offs; not needed here.

How to test the fix

We deleted the static secret, cleared the image from one node's containerd cache, and deployed:

sudo k3s crictl rmi 111122223333.dkr.ecr.eu-west-1.amazonaws.com/report-api:1.4.2
kubectl -n tools rollout restart deploy/report-api
kubectl -n tools get pods -w

Pods pulled successfully. We repeated the test the next day, more than 12 hours later, with the same result. The kubelet log shows the provider being invoked for ECR images.

Confirming in production

• No no basic auth credentials events in two months, across node reboots and new nodes joining.
• New nodes are provisioned with the binary and config through our Ansible role, and a smoke test pulls an ECR image during provisioning.

Regression guard

• Alert on any ErrImagePull or ImagePullBackOff events lasting more than five minutes.
• The provisioning smoke test fails the node join if an ECR pull does not work.
• No docker-registry secrets for ECR are allowed; a Kyverno policy rejects secrets with ECR server addresses.

Security implications

Better than before. The static secret contained a registry token readable by anyone with secret read access in the namespace, valid for 12 hours, for every repository the creating user could access. The credential provider uses the node's instance role, so no token is stored in the cluster API at all. The trade-off is that every pod on a node can pull any image the node role can pull, so scope the node role to the repositories that cluster needs.

Prevention checklist

• Never put ECR tokens in static pull secrets.
• On self-managed clusters, configure the kubelet credential provider for ECR.
• Scope node IAM roles to the repositories the cluster needs.
• Test pulls after more than 12 hours when validating a new cluster.
• Alert on image pull failures.

References

• Kubernetes kubelet credential providers — https://kubernetes.io/docs/tasks/administer-cluster/kubelet-credential-provider/
• AWS cloud provider, ecr-credential-provider — https://github.com/kubernetes/cloud-provider-aws
• Amazon ECR registry authentication — https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html
• k3s private registry and credential provider configuration — https://docs.k3s.io/installation/private-registry