Origin egress doubled after a release: Set-Cookie made the CDN bypass its cache

The week after a major web framework upgrade, our cost dashboard showed origin data transfer from the load balancer to the CDN at twice the usual level, and the origin servers' CPU had risen by a third. The CDN's cache statistics for the /assets/* path:

                    Before release   After release
Cache hit ratio          97.2%           40.8%
Origin requests/s          120           2,950
Bytes from origin/day     190 GB           3.9 TB

Static assets, which are fingerprinted and never change, were being fetched from the origin most of the time.

Reproducing it

A request to an asset straight to the origin showed the headers:

$ curl -sI https://origin.example.com/assets/app.3f9c21e7.js
HTTP/2 200
content-type: application/javascript
cache-control: private, no-cache
set-cookie: sid=eyJhbGciOi...; Path=/; HttpOnly; Secure; SameSite=Lax
etag: "3f9c21e7"

Before the release, the same request returned cache-control: public, max-age=31536000, immutable and no cookie.

What we ruled out

• CDN configuration changes. The cache policy and behaviours were unchanged in Terraform state.
• Cache key fragmentation from query strings. Asset URLs had no query strings.
• Many new asset versions. One deploy, one set of fingerprints.

Root cause

The framework upgrade changed the default middleware order. Session middleware, which used to run only for routes under the application router, now ran globally, including for the static file handler. It created a session for every request without a cookie, attached Set-Cookie, and set Cache-Control: private, no-cache to protect session-bearing responses. Our CDN behaviour for assets was configured to respect origin cache headers, so it treated every asset response as uncacheable.

What is happening under the hood

Shared caches like CDNs decide whether to store a response from its headers and their own policy:

• Cache-Control: private means only the browser may store it; shared caches must not.
• no-cache requires revalidation before every reuse.
• Many CDNs do not cache responses that contain Set-Cookie by default, or forward cookies and include them in behaviour decisions, because caching a response with a session cookie could hand one user's session to another.

Each miss is a full request to the origin, billed as data transfer from origin to CDN (when the origin is outside the CDN's free path) and adding load. For large JavaScript bundles and images, misses add up fast.

Environment

Minimal reproduction

app.use(session({ secret: SECRET, saveUninitialized: true }));   // runs for every request
app.use("/assets", express.static("dist/assets", { immutable: true, maxAge: "1y" }));

With session middleware first and saveUninitialized: true, every asset response gets a new cookie.

The fix

1. Serve static assets before session middleware, without cookies.

app.use("/assets", express.static("dist/assets", {
  immutable: true,
  maxAge: "1y",
  setHeaders: (res) => res.removeHeader("Set-Cookie"),
}));
app.use(session({ secret: SECRET, saveUninitialized: false }));

saveUninitialized: false also stopped creating sessions for anonymous visitors who never log in, which reduced session store writes by about 60 percent.

2. Make the CDN independent of origin mistakes for immutable paths. The /assets/* behaviour now uses a cache policy with a minimum TTL that overrides origin headers for fingerprinted assets, and does not forward cookies to the origin for that path:

resource "aws_cloudfront_cache_policy" "immutable_assets" {
  name        = "immutable-assets"
  min_ttl     = 86400
  default_ttl = 31536000
  max_ttl     = 31536000
  parameters_in_cache_key_and_forwarded_to_origin {
    cookies_config       { cookie_behavior = "none" }
    headers_config       { header_behavior = "none" }
    query_strings_config { query_string_behavior = "none" }
    enable_accept_encoding_gzip   = true
    enable_accept_encoding_brotli = true
  }
}

We only do this for paths with content-hashed file names, where a stale response is impossible by construction.

3. Move assets to object storage behind the CDN, with the app no longer serving them at all. This took longer and happened the following sprint.

Alternatives and trade-offs

• Only fix the app. Correct, but leaves the CDN vulnerable to the next middleware change.
• Override TTLs for all paths. Dangerous for dynamic pages with personal data; we limited overrides to immutable assets.

How to test the fix

curl -sI https://origin.example.com/assets/app.3f9c21e7.js | grep -iE 'cache-control|set-cookie'
# cache-control: public, max-age=31536000, immutable

curl -sI https://www.example.com/assets/app.3f9c21e7.js | grep -i x-cache
# x-cache: Hit from cloudfront   (on second request)

An integration test now asserts that responses under /assets/ never contain Set-Cookie and always contain immutable.

Confirming in production

• Cache hit ratio for /assets/* returned to 97 percent within an hour, as the CDN refilled.
• Origin bytes per day dropped from 3.9 TB to about 200 GB; origin CPU returned to normal.
• Page load times for first-time visitors improved slightly, since assets were again served from edge locations.

Regression guard

• CloudWatch alarm on the CDN cache hit ratio for the assets behaviour below 90 percent for 30 minutes.
• The header integration test in CI.
• Framework upgrades include a checklist item: compare response headers for a sample of routes before and after.

Security implications

The CDN did the right thing by not caching responses with Set-Cookie: caching them could have served one visitor's session cookie to others. The fix keeps that protection for dynamic routes and only changes behaviour for static assets that never carry user data. We also confirmed no HTML or API path was given the long-TTL policy.

Prevention checklist

• Serve static assets before session or auth middleware.
• Never attach cookies to static asset responses.
• Use immutable, long cache headers for fingerprinted files.
• Monitor cache hit ratio per path, with alerts.
• Compare response headers before and after framework upgrades.

References

• MDN, Cache-Control — https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
• RFC 9111, HTTP caching — https://www.rfc-editor.org/rfc/rfc9111
• CloudFront cache policies — https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/controlling-the-cache-key.html
• CloudFront, caching content based on cookies — https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Cookies.html