Our non-root pod could not write to its own volume: runAsUser, fsGroup and slow mounts

As part of a hardening project, we changed our document-processing service to run as a non-root user. The image already created a user with UID 10001, and we added a pod security context:

securityContext:
  runAsUser: 10001
  runAsGroup: 10001
  runAsNonRoot: true

The pod went into CrashLoopBackOff. Its log:

2026-10-20T09:12:03Z FATAL cannot create work directory: mkdir /data/work: permission denied

/data was a PersistentVolumeClaim, backed by a block storage volume formatted as ext4. Inspecting it from a debug pod that ran as root:

kubectl debug -it doc-processor-0 --image=busybox:1.36 --target=processor -- sh

/ # ls -ld /proc/1/root/data /proc/1/root/data/*
drwxr-xr-x    4 root     root          4096 Oct  2 11:40 /proc/1/root/data
drwxr-xr-x    2 root     root         16384 Oct  2 11:40 /proc/1/root/data/lost+found
drwxr-xr-x  812 root     root         45056 Oct 19 23:58 /proc/1/root/data/cache

Everything on the volume belonged to root:root with mode 755. Before the change, the service ran as root, so it could write anywhere. As UID 10001, it could read but not write.

Who owns a volume's files

A container's runAsUser decides which UID the process runs as. It does not change who owns files on a volume. A freshly formatted ext4 filesystem has a root directory owned by root:root, and files the old root process created are owned by root too. Kubernetes does not chown volumes by default.

For many volume types, Kubernetes can change ownership for you with fsGroup. When a pod specifies fsGroup: N, the kubelet, while mounting a volume that supports ownership management (most block-backed CSI volumes, emptyDir, and others; not NFS in most configurations, and not hostPath):

1. Changes the group of every file and directory on the volume to N.
2. Sets the setgid bit on directories, so new files inherit group N.
3. Adds group write permission where needed.
4. Adds N to the supplementary groups of every container in the pod.

So the process, running as UID 10001 with supplementary group N, can write through group permissions.

The first fix, and the next problem

securityContext:
  runAsUser: 10001
  runAsGroup: 10001
  runAsNonRoot: true
  fsGroup: 10001

The pod started and could write. Then the next deploy took nine minutes for the pod to become ready. The pod was stuck in ContainerCreating, and the kubelet log on the node showed:

kubelet: Setting volume ownership for /var/lib/kubelet/pods/.../volumes/kubernetes.io~csi/pvc-.../mount
         and fsGroup set. If the volume has a lot of files then setting volume ownership could be slow,
         see https://github.com/kubernetes/kubernetes/issues/69699

By default, the kubelet applies the ownership change recursively on every mount. Our volume held about 3.8 million cached files. Walking and chowning all of them took over eight minutes, every time a pod started, even when nothing had changed since the last mount.

Environment

1. fsGroupChangePolicy: OnRootMismatch

Kubernetes added fsGroupChangePolicy (stable since 1.23) for exactly this case:

securityContext:
  runAsUser: 10001
  runAsGroup: 10001
  runAsNonRoot: true
  fsGroup: 10001
  fsGroupChangePolicy: OnRootMismatch

With OnRootMismatch, the kubelet checks only the root directory of the volume. If its ownership and permissions already match what fsGroup requires, the kubelet skips the recursive walk. The first mount after the change still did the full chown (eight minutes, once). Every later mount took under a second.

The default, Always, re-applies ownership on every mount. That is safer if something might change ownership of files on the volume behind Kubernetes' back, but with a single writer it is wasted work.

The caveat with OnRootMismatch: if something writes files with different ownership into the volume (a restore job running as root, for example), the root directory still matches, so those files are not fixed on the next mount. We made the restore job run as UID 10001 too.

2. Some volumes ignore fsGroup

fsGroup only works when the volume plugin supports ownership management. For CSI drivers, the driver's CSIDriver object declares it with fsGroupPolicy:

kubectl get csidriver -o custom-columns=NAME:.metadata.name,FSGROUP:.spec.fsGroupPolicy

NAME                     FSGROUP
block.csi.example.com    ReadWriteOnceWithFSType
files.csi.example.com    None

ReadWriteOnceWithFSType means the kubelet applies fsGroup only for read-write-once volumes with a filesystem type set. None means it never does. Our shared file storage (NFS-based) is None, so fsGroup does nothing there. For a second service using that storage, we set ownership on the NFS export itself (the storage team created the export directory owned by UID 10001), which is the correct place for it.

Some CSI drivers report File, in which case the driver, not the kubelet, applies the ownership, sometimes by passing a mount option. That can be instant even for huge volumes.

3. When an init container is better

An alternative we considered is an init container running as root that fixes ownership once:

initContainers:
  - name: fix-perms
    image: busybox:1.36
    command: ["sh", "-c", "[ \"$(stat -c %u /data)\" = 10001 ] || chown -R 10001:10001 /data"]
    securityContext:
      runAsUser: 0
      runAsNonRoot: false
      capabilities: { drop: ["ALL"], add: ["CHOWN"] }
    volumeMounts:
      - { name: data, mountPath: /data }

It works with volumes that fsGroup does not support (except where root squashing on NFS prevents chown), and sets the user owner as well as the group. The downsides: the pod now contains a container running as root, which our Pod Security Admission policy at the restricted level does not allow, and it needs the CHOWN capability. For block volumes, fsGroup with OnRootMismatch gave the same result without either. We use the init container approach only for one legacy volume type that needed it, in a namespace with a documented exception.

4. Make the image and the volume agree from the start

For new volumes, the cleanest result is that files are created by the right user from the beginning. With fsGroup set from the first deploy, the volume's root gets the right group on first mount, and every file the app creates afterwards is group-writable and owned by UID 10001.

1. Writes work as non-root.

kubectl exec doc-processor-0 -- sh -c 'id; touch /data/work/probe && ls -ln /data/work/probe'

uid=10001 gid=10001 groups=10001
-rw-r--r--    1 10001    10001            0 Oct 21 08:02 /data/work/probe

2. Mount time. We measured pod start time from scheduling to ready over ten restarts:

3. Admission policy. The namespace now enforces Pod Security Admission at the restricted level, which requires runAsNonRoot, drops all capabilities and disallows privilege escalation. The deployment passes:

kubectl label --dry-run=server --overwrite ns docs pod-security.kubernetes.io/enforce=restricted

namespace/docs labeled (server dry run)

No warnings about existing pods violating the policy.

4. Security implications. Running as non-root limits what an attacker who compromises the process can do inside the container and on the node, especially combined with a read-only root filesystem and dropped capabilities. The volume's files are now group-owned by 10001 and not world-writable, so other pods that might mount the same volume (none should) would not gain write access by default.

Takeaways

• runAsUser changes the process's UID, not the ownership of files on volumes.
• fsGroup makes the kubelet chown supported volumes to a group the pod's containers join.
• The default recursive chown on every mount is slow for large volumes; use fsGroupChangePolicy: OnRootMismatch.
• Check the CSI driver's fsGroupPolicy; for volumes that ignore fsGroup, fix ownership at the storage or with a tightly scoped init container.

References: Configure a security context for a pod, Configure volume permission and ownership change policy, CSI driver fsGroup support, Pod Security Standards.