AWS NAT Gateway bill spike from S3 and ECR traffic in private subnets

Nobody got paged for this one. Our finance contact forwarded an AWS Budgets alert in early August: the forecast for the month was 41 percent over budget, and the July invoice was 9,400 USD higher than June. No incident, no outage, no new service launch that would explain it.

The first stop was Cost Explorer, grouped by Usage type for July, and one line stood out:

Usage type                 Cost (USD)   Usage quantity
USE1-NatGateway-Bytes      10,212.45    226,943 GB
USE1-NatGateway-Hours         131.40      2,920 Hrs
USE1-DataTransfer-Regional     88.10      8,810 GB

Two hundred and twenty-seven terabytes processed by NAT gateways in one month. In June the same line was about 18 TB.

Reproducing it

There is nothing to reproduce in the usual sense, but we needed to find who was sending that traffic. We enabled VPC Flow Logs on the NAT gateway's network interfaces with a custom format that includes the AWS service field, delivered to S3 in Parquet so Athena could query it:

aws ec2 create-flow-logs \
  --resource-type NetworkInterface \
  --resource-ids eni-0a1b2c3d4e5f60718 eni-0f1e2d3c4b5a69788 eni-0123456789abcdef0 \
  --traffic-type ALL \
  --log-destination-type s3 \
  --log-destination arn:aws:s3:::example-flowlogs-bucket/nat/ \
  --destination-options FileFormat=parquet,HiveCompatiblePartitions=true,PerHourPartition=true \
  --log-format '${start} ${pkt-srcaddr} ${pkt-dstaddr} ${bytes} ${pkt-src-aws-service} ${pkt-dst-aws-service} ${flow-direction}'

After 24 hours we ran this in Athena:

SELECT pkt_src_aws_service AS service,
       pkt_dstaddr        AS internal_host,
       sum(bytes) / 1e9   AS gb
FROM nat_flow_logs
WHERE flow_direction = 'egress'
  AND pkt_dstaddr LIKE '10.40.%'
GROUP BY 1, 2
ORDER BY gb DESC
LIMIT 20;

The top rows were all S3, flowing to IP addresses in the subnets of our ECS cluster that runs nightly data jobs. A smaller but steady chunk was AMAZON traffic to the same hosts at task start times, which matched ECR image pulls.

What we ruled out

• Internet egress or a compromised host. The destination services were AWS-owned prefixes, and DataTransfer-Out-Bytes to the internet was flat month over month.
• Cross-AZ traffic. DataTransfer-Regional was small. The NAT gateways were already one per AZ.
• A bug in the S3 bucket replication setup. Replication charges show up under S3 usage types, not NAT, and the replication metrics were unchanged.

Root cause

In July the data team moved the nightly enrichment jobs from a public-subnet EC2 instance to ECS Fargate tasks in private subnets, as part of a security cleanup. The jobs read about 7 TB a night of Parquet files from S3 in the same region. In the old setup, traffic went to S3 over the instance's public IP, which is free for same-region S3. In the new setup, private subnets had a default route to a NAT gateway and no S3 endpoint, so every byte went through NAT and was billed at 0.045 USD per GB of data processing. On top of that, each of 140 tasks a night pulled a 2.8 GB image from ECR, and ECR image layers are served from S3, so those went through NAT too.

What is happening under the hood

A NAT gateway charges per hour and per gigabyte it processes, in both directions, regardless of the destination. It does not know or care that the destination is an AWS service in the same region. In us-east-1 that is 0.045 USD per GB, which is more than S3 standard storage costs per GB-month.

The way to avoid it is to give the VPC a private route to the service:

• Gateway endpoints (S3 and DynamoDB only) add a route for the service's prefix list to the route tables you choose. They have no hourly or data charge.
• Interface endpoints (PrivateLink) put an ENI for the service in your subnets. They cost a small hourly fee per AZ plus a per-GB fee that is far lower than NAT processing.

When both a gateway endpoint route and a NAT default route exist, the more specific prefix list route wins, so S3 traffic stops going through NAT without any application change.

ECR is a special case: the registry API and Docker registry endpoints are separate interface endpoints (ecr.api and ecr.dkr), and the actual layer blobs are downloaded from S3. Fargate tasks on platform version 1.4.0 and later need both ECR interface endpoints plus the S3 gateway endpoint to pull privately.

Environment

Minimal reproduction

The cost math alone shows the problem:

S3 reads:    7,000 GB/night x 31 nights        = 217,000 GB
ECR pulls:   2.8 GB x 140 tasks x 31 nights    =  12,152 GB
NAT charge:  ~229,000 GB x 0.045 USD/GB        = ~10,300 USD per month

Any task in a private subnet whose route table sends 0.0.0.0/0 to a NAT gateway and has no S3 prefix list route will produce this pattern.

Who is affected

Workloads in private subnets that talk to S3, ECR, CloudWatch Logs, STS, Secrets Manager or other AWS APIs at volume. EKS nodes pulling images, Glue or EMR in a VPC, and Lambda functions attached to a VPC that write to S3 are common sources. Public-subnet instances and workloads that already use endpoints are not affected.

The fix

1. S3 gateway endpoint on every private route table. This alone removed about 95 percent of the bytes. We manage the VPC with Terraform:

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.us-east-1.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = aws_route_table.private[*].id

  tags = { Name = "s3-gateway" }
}

2. Interface endpoints for ECR and logs. These are shared by every service in the VPC, so the hourly cost is spread across workloads:

locals {
  interface_services = ["ecr.api", "ecr.dkr", "logs", "sts"]
}

resource "aws_vpc_endpoint" "interface" {
  for_each            = toset(local.interface_services)
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.us-east-1.${each.key}"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private[*].id
  security_group_ids  = [aws_security_group.endpoints.id]
  private_dns_enabled = true
}

The endpoint security group allows TCP 443 from the VPC CIDR only. With private_dns_enabled, the normal service hostnames resolve to the endpoint ENIs, so no SDK or task definition change was needed.

3. Smaller images. We rebuilt the job image on a slim base with a multi-stage build, bringing it from 2.8 GB to 640 MB. That is not a NAT fix, but it cut task start time by almost a minute.

Alternatives and trade-offs

• Put the jobs back in a public subnet. Free S3 access, but it undoes the security cleanup that started all this.
• S3 interface endpoint instead of gateway. Needed only when traffic comes from on-premises over Direct Connect or VPN, or from another VPC. Inside the VPC the gateway endpoint is free and simpler.
• Interface endpoints for everything. Each endpoint costs per hour per AZ. For low-traffic services, NAT can be cheaper. We only added endpoints where flow logs showed real volume.

Gateway endpoints only cover S3 in the same region. Reads from a bucket in another region still go through NAT and are also billed as inter-region transfer.

How to test the fix

Before rolling to production we ran one job in staging and checked the route and the path:

aws ec2 describe-route-tables --route-table-ids rtb-0abc1234def567890 \
  --query 'RouteTables[].Routes[?DestinationPrefixListId!=`null`]'

# From inside a running task (ECS Exec), the S3 hostname should still resolve to public S3 IPs,
# but traffic follows the prefix-list route instead of NAT.
aws ecs execute-command --cluster data-jobs --task 3f2e... --interactive --command "aws s3 cp s3://example-data/sample.parquet /tmp/"

Then we watched the NAT gateway's BytesOutToDestination metric during the staging run. It stayed flat while the job read 300 GB.

Confirming in production

• BytesOutToDestination across the three NAT gateways dropped from about 7.3 TB per night to about 220 GB per night the first night after rollout.
• The NAT flow log Athena query no longer showed S3 in the top 20 sources.
• The August USE1-NatGateway-Bytes line came in at 612 USD, down from 10,212 USD. The new interface endpoints added about 120 USD per month.
• Task startup p50 went from 94 seconds to 31 seconds thanks to the private ECR path and smaller image.

Regression guard

• A Cost Anomaly Detection monitor scoped to the EC2-Other service, which is where NAT charges land, with an alert threshold of 200 USD per day.
• A CloudWatch alarm on the sum of BytesOutToDestination for all NAT gateways above 1 TB per day.
• A Terraform module check: every new private route table must be listed in the S3 endpoint's route_table_ids, enforced by a tflint custom rule in CI.

Security implications

The fix keeps traffic on the AWS network and lets us tighten access. We added an endpoint policy on the S3 gateway endpoint that only allows our own buckets, which blocks data exfiltration to an attacker-controlled bucket from inside the VPC, and a bucket policy condition on aws:SourceVpce for the data bucket so it can only be read through that endpoint plus the CI role. Be careful with that second step: a too-strict bucket policy can lock out the console and other legitimate paths. Test with a non-production bucket first.

Prevention checklist

• Add the S3 gateway endpoint when you create a VPC, not when the bill arrives.
• Before moving workloads into private subnets, list every AWS service they call and decide endpoint versus NAT for each one.
• Turn on Cost Anomaly Detection and budgets with forecast alerts.
• Keep VPC Flow Logs with the AWS service fields available so attribution takes hours, not days.
• Keep container images small; image pulls multiply by task count.

References

• NAT gateway pricing and data processing — https://aws.amazon.com/vpc/pricing/
• Gateway endpoints for Amazon S3 — https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html
• Amazon ECR interface VPC endpoints — https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html
• VPC Flow Logs record fields — https://docs.aws.amazon.com/vpc/latest/userguide/flow-log-records.html
• NAT gateway CloudWatch metrics — https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway-cloudwatch.html