S3 bill keeps growing while the bucket looks small: noncurrent versions and lost uploads

Our photo-sharing feature stores user uploads in an S3 bucket. The admin dashboard, which sums object sizes from our database, said 4.1 TB. The bill said 31 TB of S3 Standard storage for that bucket, and it was growing by about 2 TB a month while the dashboard grew by 150 GB.

Reproducing it

The CloudWatch storage metrics for the bucket matched the bill, not our database. S3 Storage Lens, with the bucket selected, gave the breakdown:

Metric                                   Value
Total storage                            31.2 TB
Current version bytes                     4.3 TB
Noncurrent version bytes                 24.6 TB
Delete marker object count                 3.9 M
Incomplete multipart upload bytes         2.3 TB

79 percent of storage was noncurrent versions, and 7 percent was parts from multipart uploads that never completed.

What we ruled out

• Replication or another bucket. The metrics were for this bucket alone.
• A database bug undercounting files. Current version bytes in Storage Lens matched the dashboard within 5 percent.
• Data in other storage classes. Everything was in Standard.

Root cause

Versioning was enabled on the bucket two years earlier to protect against accidental deletes. Our app regenerates thumbnails and overwrites them when users edit photos, and deletes originals when users remove photos. With versioning, an overwrite keeps the old object as a noncurrent version and a delete adds a delete marker; neither frees storage. No lifecycle rule ever expired noncurrent versions. Separately, the mobile app uploads large videos with multipart upload; when users closed the app mid-upload, the parts stayed in the bucket, billed but invisible in normal listings.

What is happening under the hood

In a versioned bucket:

• PUT to an existing key creates a new current version; the previous one becomes noncurrent and is still stored and billed.
• DELETE without a version id adds a delete marker; all versions remain and are billed.
• Only deleting specific version ids, or lifecycle expiration, removes data.

Incomplete multipart uploads are parts uploaded through UploadPart without a final CompleteMultipartUpload or AbortMultipartUpload. They do not appear in ListObjects, they are billed as storage, and they stay forever unless aborted or cleaned up by a lifecycle rule.

Environment

Minimal reproduction

aws s3api put-object --bucket demo-versioned --key a.jpg --body v1.jpg
aws s3api put-object --bucket demo-versioned --key a.jpg --body v2.jpg   # v1 becomes noncurrent
aws s3api delete-object --bucket demo-versioned --key a.jpg              # delete marker, v1+v2 still billed
aws s3api list-object-versions --bucket demo-versioned --prefix a.jpg

The fix

We added lifecycle rules matched to how long we actually need recovery:

resource "aws_s3_bucket_lifecycle_configuration" "media" {
  bucket = aws_s3_bucket.media.id

  rule {
    id     = "noncurrent-versions"
    status = "Enabled"
    filter {}
    noncurrent_version_expiration {
      noncurrent_days           = 30
      newer_noncurrent_versions = 2
    }
  }

  rule {
    id     = "expired-delete-markers"
    status = "Enabled"
    filter {}
    expiration {
      expired_object_delete_marker = true
    }
  }

  rule {
    id     = "abort-incomplete-mpu"
    status = "Enabled"
    filter {}
    abort_incomplete_multipart_upload {
      days_after_initiation = 3
    }
  }
}

• Noncurrent versions are deleted 30 days after they become noncurrent, but the two most recent noncurrent versions of each object are kept regardless, which covers "undo my last edit" style recovery.
• Delete markers with no remaining versions are removed, which keeps listings fast.
• Incomplete multipart uploads are aborted after 3 days.

Thumbnails do not need versioning at all, since they can be regenerated. We moved them to a separate unversioned bucket so overwrites no longer create noncurrent versions.

Alternatives and trade-offs

• Turn versioning off. Versioning cannot be removed once enabled, only suspended, and existing versions remain. It also removes protection against accidental or malicious deletes.
• Move noncurrent versions to a cheaper storage class first (Glacier tiers), then expire later. Good when you need long recovery windows.
• Delete versions with a script. Lifecycle rules are free and run continuously; scripts cost request fees and must be maintained.

How to test the fix

On a test bucket with the same rules, we created objects, overwrote and deleted them, and checked versions after the lifecycle window, using shortened days on the test bucket. Lifecycle actions run asynchronously, typically within a day or two of eligibility, so we checked Storage Lens over the following week in production.

Confirming in production

• Within six days, noncurrent bytes dropped from 24.6 TB to 1.1 TB and incomplete multipart upload bytes to near zero.
• Billed storage for the bucket stabilised at about 5.2 TB.
• Monthly S3 storage cost for the feature fell from roughly 720 USD to 120 USD.

Regression guard

• A Storage Lens dashboard is reviewed in the monthly cost meeting, with noncurrent and incomplete upload bytes as explicit rows.
• A Terraform policy check requires a lifecycle configuration with an incomplete multipart upload rule on every bucket, and a noncurrent version rule on every versioned bucket.

Security implications

Versioning is also a defence against ransomware and accidental mass deletes. Expiring noncurrent versions after 30 days shortens that safety net, so we kept versioning on for originals, combined it with MFA delete on the production account's critical buckets, and replicate originals to a separate account with its own retention. Users who delete photos also expect the data to be gone eventually; lifecycle expiry makes that true, which helps with data protection obligations.

Prevention checklist

• Pair versioning with a noncurrent version lifecycle rule from day one.
• Abort incomplete multipart uploads after a few days on every bucket.
• Separate regenerable data from data that needs versioning.
• Use Storage Lens to compare current and noncurrent bytes.
• Decide recovery windows explicitly instead of keeping everything forever.

References

• S3 versioning — https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html
• Lifecycle configuration elements — https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html
• Aborting incomplete multipart uploads with lifecycle — https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html
• S3 Storage Lens metrics glossary — https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage_lens_metrics_glossary.html