Lambda writes to the bucket that triggers it: a recursive loop and a cost spike

At 14:20 on a Wednesday, the Lambda invocation alarm for our image service went off. The thumbnailer function, which creates resized copies of uploaded product photos, normally runs about 40,000 times a day. CloudWatch showed it climbing steeply:

14:00  Invocations: 1,710
14:30  Invocations: 46,220
15:00  Invocations: 1,302,800
15:30  Invocations: 2,988,410  (Throttles: 410,733)

S3 PUT requests on the media bucket rose in step, and the bucket's object count grew by millions of tiny files with names like thumbs/thumbs/thumbs/12345_200.jpg.

Reproducing it

The function code after that morning's deploy:

def handler(event, context):
    for record in event["Records"]:
        key = record["s3"]["object"]["key"]          # e.g. uploads/12345.jpg
        img = load(bucket, key)
        for size in (200, 600):
            out_key = f"thumbs/{basename(key)}_{size}.jpg"
            s3.put_object(Bucket=bucket, Key=out_key, Body=resize(img, size))

And the bucket notification:

{"LambdaFunctionConfigurations": [{
  "LambdaFunctionArn": "arn:aws:lambda:eu-west-1:111122223333:function:thumbnailer",
  "Events": ["s3:ObjectCreated:*"],
  "Filter": {"Key": {"FilterRules": [{"Name": "suffix", "Value": ".jpg"}]}}
}]}

The notification filtered by suffix only. Every thumbnail written under thumbs/ was itself a new .jpg, which triggered the function again.

What we ruled out

• A traffic spike from users. Uploads to uploads/ were normal.
• Retries of failing invocations. Error rate was low; invocations were new events.

Root cause

Previously, the notification had a prefix filter uploads/ and thumbnails went to a separate bucket. A refactor moved thumbnails into the same bucket under thumbs/ to simplify CDN configuration, and the Terraform change replaced the notification filter with a suffix-only one, because "we only want images". Each upload now produced two thumbnails, each thumbnail produced two more, and so on, growing exponentially until concurrency limits throttled it.

What is happening under the hood

S3 event notifications fire for every object created that matches the filter, including objects created by the function the notification invokes. Nothing in S3 knows that an object was produced by a Lambda reacting to another object.

Lambda has a recursive loop detection feature that stops some loops automatically by tracking how many times the same event chain has invoked the function; it covers specific services listed in the documentation. It is a useful safety net, but whether it covers your trigger type depends on the service and integration, so design so that loops cannot happen in the first place.

Cost of a loop comes from several places at once: Lambda requests and duration, S3 PUT and GET requests, storage for the generated objects, and anything downstream such as CDN invalidations or replication.

Environment

Minimal reproduction

Any function that writes objects matching its own notification filter into the bucket that triggers it.

The fix

Stop the bleeding. Setting reserved concurrency to zero throttles every invocation immediately, which is faster than redeploying code:

aws lambda put-function-concurrency --function-name thumbnailer --reserved-concurrent-executions 0

S3 retries failed asynchronous invocations for a while and then drops or sends them to a failure destination, so the event backlog drained without running the function. We then deleted the recursive objects:

aws s3 rm s3://example-media/thumbs/thumbs/ --recursive

Fix the structure.

1. Separate input and output: thumbnails go back to a dedicated bucket, example-media-thumbs, which has no notifications. Using separate buckets makes loops impossible by construction, even if filters change later.
2. Restore a prefix filter on the input bucket notification (uploads/ plus suffix .jpg) as a second layer.
3. A guard in the code that refuses to process keys outside the input prefix:

INPUT_PREFIX = "uploads/"

def handler(event, context):
    for record in event["Records"]:
        key = record["s3"]["object"]["key"]
        if not key.startswith(INPUT_PREFIX):
            logger.warning("ignoring unexpected key %s", key)
            continue
        ...

1. Reserved concurrency of 50 for the function in normal operation, which caps the blast radius of any future loop.

Alternatives and trade-offs

• Same bucket with strict prefixes. Works, and depends on filters never being edited carelessly.
• Write metadata on outputs and skip objects with that metadata. Requires a HEAD request per event, which adds cost.
• EventBridge rules for S3 events, with more expressive filters. Useful when several consumers need different subsets.

How to test the fix

A Terraform test asserts that no bucket notification target writes to its own source bucket, by checking a tag that declares each function's output buckets. In staging, we uploaded 100 images and confirmed exactly 100 invocations and 200 thumbnails.

Confirming in production

• After the fix, invocations returned to about 40,000 a day.
• The six-hour incident cost about 2,900 USD across Lambda, S3 requests and storage, most of it in S3 PUT requests.
• Thumbnail generation for normal uploads was delayed for about 90 minutes during the incident; affected products showed the original image meanwhile.

Regression guard

• CloudWatch anomaly detection alarm on invocations for every event-driven function.
• Reserved concurrency set on all event-driven functions through a Terraform module default.
• Code review checklist for event-driven code: "can the output trigger the input?"

Security implications

Recursive loops are a denial-of-wallet risk, and an attacker who can upload files can sometimes trigger expensive processing deliberately. Reserved concurrency and per-user upload limits cap that. Separating input and output buckets also lets us give the function write access only to the output bucket, so a compromised function cannot overwrite uploads.

Prevention checklist

• Never write outputs where they can trigger the same function; use separate buckets.
• Use prefix and suffix filters on notifications as a second layer.
• Set reserved concurrency on event-driven functions.
• Know that setting reserved concurrency to zero is the emergency stop.
• Alarm on invocation anomalies, not only errors.

References

• Using an Amazon S3 trigger with Lambda — https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html
• Lambda recursive loop detection — https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html
• S3 event notification filtering — https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-filtering.html
• Lambda reserved concurrency — https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html