Orphaned EBS volumes and snapshots after Kubernetes churn: 4,800 USD a month of nothing

EBS costs in our development account had grown from 1,900 to 6,700 USD per month over a year, while the number of nodes and the size of our clusters were flat. The breakdown in Cost Explorer:

Usage type                         Cost (USD)
EUW1-EBS:VolumeUsage.gp3              3,920
EUW1-EBS:SnapshotUsage                2,410
EUW1-EBS:VolumeP-IOPS.gp3               370

Reproducing it

Listing volumes not attached to any instance:

aws ec2 describe-volumes --filters Name=status,Values=available \
  --query 'Volumes[].{id:VolumeId,gb:Size,created:CreateTime,pv:Tags[?Key==`kubernetes.io/created-for/pv/name`]|[0].Value,ns:Tags[?Key==`kubernetes.io/created-for/pvc/namespace`]|[0].Value}' \
  --output table | head

|  created                  | gb  |  id                    |  ns        |  pv                                        |
|  2024-06-11T09:14:22Z     | 50  |  vol-0a1b2c3d4e5f60718 |  pr-3912   |  pvc-6f1e2a7c-...                          |
|  2024-06-11T09:14:25Z     | 20  |  vol-0f1e2d3c4b5a69788 |  pr-3912   |  pvc-91d0c4b2-...                          |

612 available volumes, 41 TB in total, almost all tagged with namespaces like pr-NNNN that no longer existed. Snapshots told a similar story: 9,400 of them, the oldest from 2022.

What we ruled out

• Volumes still needed by something outside Kubernetes. Tags showed they were created by the EBS CSI driver for PVCs; none had been attached in months according to CloudTrail.
• Snapshots required for compliance. The development account has no retention requirement; production snapshots are managed separately.

Root cause

Our preview environments used a StorageClass with reclaimPolicy: Retain, copied from production where we want volumes to survive accidental PVC deletion. When the preview namespace was deleted, the PVC went away, the PersistentVolume moved to Released, and the EBS volume stayed, billed, forever. A nightly snapshot job created snapshots of every volume tagged backup=true (which the template also set) and had no expiry.

What is happening under the hood

A PersistentVolume's reclaimPolicy decides what happens to the underlying disk when its claim is deleted:

• Delete: the CSI driver deletes the cloud volume.
• Retain: the PV becomes Released and the volume stays until an administrator deals with it.

Retain is a safety feature for important data. For ephemeral environments it is a leak. Deleting the namespace deletes the PVC; with Retain, nothing deletes the volume, and the Released PV object may even be deleted later by cleanup scripts, leaving a volume with no Kubernetes object pointing at it.

EBS snapshots are incremental, but each one keeps the blocks it references, and without expiry, old snapshots of deleted volumes keep their data billed.

Environment

Minimal reproduction

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: {name: gp3-retain}
provisioner: ebs.csi.aws.com
reclaimPolicy: Retain
parameters: {type: gp3}

Create a PVC with this class in a namespace, delete the namespace, and the EBS volume remains available.

The fix

1. Verify before deleting. For each available volume we checked: no attachment in the last 30 days (CloudTrail AttachVolume events), namespace no longer exists, and owner tag is a preview environment. We snapshotted nothing extra; the data was test data by definition. The deletion script runs in dry-run first and writes a CSV for review:

aws ec2 describe-volumes --filters Name=status,Values=available Name=tag:env,Values=preview \
  --query 'Volumes[].VolumeId' --output text | tr '\t' '\n' > candidates.txt
# review candidates.txt, then:
while read v; do aws ec2 delete-volume --volume-id "$v"; done < candidates.txt

2. Use the right reclaim policy per environment. Preview and development namespaces now use a gp3-delete StorageClass, enforced by an admission policy that rejects Retain classes outside production namespaces.

3. Replace the snapshot Lambda with Amazon Data Lifecycle Manager policies that have retention built in:

resource "aws_dlm_lifecycle_policy" "dev_daily" {
  description        = "dev daily snapshots, keep 7"
  execution_role_arn = aws_iam_role.dlm.arn
  state              = "ENABLED"
  policy_details {
    resource_types = ["VOLUME"]
    target_tags    = { backup = "daily" }
    schedule {
      name = "daily"
      create_rule { interval = 24, interval_unit = "HOURS", times = ["03:00"] }
      retain_rule { count = 7 }
      copy_tags = true
    }
  }
}

4. Clean old snapshots whose source volume no longer exists and that are older than 30 days, after the same review step.

Alternatives and trade-offs

• Keep Retain and run a reaper that deletes Released PVs and their volumes after N days. Useful when you want a grace period to recover data.
• Recycle Bin for snapshots. Lets you recover deleted snapshots for a set period, a good safety net when cleaning up at scale.
• No PVCs in previews (ephemeral databases with emptyDir). Cheapest, but slower environment start when data must be seeded.

How to test the fix

In a sandbox namespace we created PVCs with the new class, deleted the namespace, and confirmed the volumes were deleted within a minute. For DLM, we checked after eight days that exactly seven snapshots existed per tagged volume.

Confirming in production

• 612 volumes (41 TB) and 9,400 snapshots deleted over two days after review.
• Monthly EBS cost in the development account dropped from 6,700 to 1,900 USD.
• The weekly count of available volumes older than 7 days has stayed below 5.

Regression guard

• A weekly report of unattached volumes older than 7 days and snapshots without a DLM policy tag, per account.
• AWS Trusted Advisor and Compute Optimizer checks for idle volumes reviewed in the monthly cost meeting.
• Admission policy for reclaim policies by namespace type.

Security implications

Orphaned volumes and snapshots are forgotten copies of data. Ours were test data, but the same pattern in production would leave customer data in places nobody tracks, outside retention policies and deletion requests. We also found 14 snapshots shared publicly by mistake years ago (test images); we removed the public permission and enabled the account-level setting that blocks public sharing of EBS snapshots.

Prevention checklist

• Match reclaim policy to environment: Delete for ephemeral, Retain only where needed.
• Use DLM or AWS Backup with retention instead of custom snapshot scripts.
• Report unattached volumes and orphaned snapshots weekly.
• Tag volumes with owner and environment through StorageClass parameters.
• Block public snapshot sharing at the account level.

References

• Kubernetes persistent volumes, reclaim policy — https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming
• Amazon EBS CSI driver — https://github.com/kubernetes-sigs/aws-ebs-csi-driver
• Amazon Data Lifecycle Manager — https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-lifecycle.html
• Block public access for EBS snapshots — https://docs.aws.amazon.com/ebs/latest/userguide/block-public-access-snapshots.html