Networking & DNS Bug Report

New AWS line item for public IPv4 addresses: where 2,400 USD a month came from

A VPC line item we had never paid attention to reached 2,400 USD a month. Since February 2024, AWS charges for every public IPv4 address, in use or idle. Our EKS nodes, idle Elastic IPs and load balancers in many subnets added up. Inventory and cleanup cut it by 70 percent.

Ryan Pham Ryan Pham @ryan_bugs · December 18, 2024 · 6 min read · 13 Views
Banner: AWS charges for public IPv4 addresses adding up across nodes and idle Elastic IPs

During a monthly review of our AWS bill we found a VPC-related line item that had been growing quietly all year:

Usage type                        Cost (USD)   Quantity
USE1-PublicIPv4:InUseAddress         1,891.20   378,240 hours
USE1-PublicIPv4:IdleAddress            518.40   103,680 hours

At 0.005 USD per address per hour, that was about 518 public IPv4 addresses in use at any time, plus 144 idle ones. We did not think we had more than 30 public endpoints.

Reproducing it

VPC IP Address Manager (IPAM) has a free Public IP Insights view that lists every public IPv4 address in the account by type. The breakdown:

Address type                      Count
EC2 public IPs (auto-assigned)      402
Elastic IPs (associated)             38
Elastic IPs (unassociated)          144
Load balancer IPs                    66
NAT gateway EIPs                     12

402 auto-assigned public IPs on EC2 instances, almost all of them EKS worker nodes.

What we ruled out

  • An attack or a compromised account creating resources. All addresses belonged to known resources.
  • A pricing error. The charge is documented and applies to all public IPv4 addresses, including those on AWS-managed resources in your account.

Root cause

Since February 1, 2024, AWS charges for every public IPv4 address, whether attached to a running resource or idle. Before that, in-use addresses on running instances were free. Three habits made us pay for far more than we needed:

  1. Our EKS node groups ran in subnets with "auto-assign public IPv4" enabled, a leftover from an early quick-start template. Nodes never needed public IPs; they reached the internet through NAT.
  2. 144 Elastic IPs were allocated but not associated, mostly from decommissioned bastions, old test environments and "reserved for the future" requests.
  3. Internet-facing load balancers in six subnets each, for services that only needed internal access.

What is happening under the hood

Every public IPv4 address is charged by the hour:

  • Auto-assigned public IPs on EC2 instances and ENIs.
  • Elastic IPs, associated or not.
  • Addresses used by internet-facing load balancers (one per subnet the load balancer is in, for Application and Network Load Balancers), NAT gateways and other managed services.

Private IPv4 addresses and IPv6 addresses are not charged. IPAM's Public IP Insights shows all public addresses by service and type, which is more complete than listing Elastic IPs.

Environment

Component Version / value
Compute EKS 1.30, ~400 nodes across 3 AZs
Subnets Node subnets with MapPublicIpOnLaunch=true
Load balancers 22 internet-facing ALBs/NLBs, 3 subnets each
Elastic IPs 182 allocated

Minimal reproduction

aws ec2 describe-subnets --query 'Subnets[?MapPublicIpOnLaunch].[SubnetId,Tags[?Key==`Name`]|[0].Value]' --output table
aws ec2 describe-addresses --query 'Addresses[?AssociationId==null].[PublicIp,AllocationId]' --output table

The fix

1. Move EKS nodes to private subnets. We created private node subnets with routes to NAT gateways (which we already had), migrated node groups with a rolling replacement, and set MapPublicIpOnLaunch=false on the old subnets so nothing new gets a public IP by accident:

aws ec2 modify-subnet-attribute --subnet-id subnet-0abc123def4567890 --no-map-public-ip-on-launch

Nodes that pull images and talk to AWS APIs did so through NAT and VPC endpoints from then on.

2. Release idle Elastic IPs after checking each one: no association history in the last 90 days in CloudTrail, no DNS records pointing at it (we searched our Route 53 zones and the external DNS provider), and no allowlist at a partner that references it. Twelve addresses were whitelisted by partners and were kept and documented. The rest were released.

3. Internal load balancers where possible. Services that were only called from inside the VPC or over VPN moved to internal load balancers, which use private IPs. For public services, we checked whether every load balancer needed to be in all three AZ subnets.

4. Consider IPv6 for new public endpoints. Dual-stack load balancers still need IPv4 for clients that lack IPv6, so this did not reduce our count yet, but new internal-to-AWS traffic paths default to IPv6 where supported.

Alternatives and trade-offs

  • Accept the charge for resources that genuinely need public IPv4, such as a few public load balancers. It is a real cost of public IPv4 scarcity.
  • Bring your own IP addresses (BYOIP). Organisations with their own IPv4 ranges can use them; Public IPv4 charges are different for BYOIP, but the operational overhead is significant.
  • Shared ingress. Consolidating many small public services behind one load balancer with host-based routing reduces the number of load balancers and their addresses.

How to test the fix

After each node group migration, we checked that new nodes had no public IP and could still pull images and join the cluster:

kubectl get nodes -o wide | awk '{print $1, $7}'      # EXTERNAL-IP column should be <none>

Confirming in production

  • Public IPv4 addresses dropped from about 660 to about 190.
  • The monthly public IPv4 charge fell from roughly 2,400 USD to about 700 USD.
  • Idle address hours dropped to near zero after releasing unused Elastic IPs.

Regression guard

  • AWS Config rule flags subnets with auto-assign public IPv4 enabled in workload accounts.
  • A weekly report of unassociated Elastic IPs older than 7 days.
  • Terraform module for EKS node groups only accepts private subnets.

Security implications

This was as much a security fix as a cost one. Nodes with public IPs are directly reachable from the internet if a security group is too permissive, and we found two security groups that allowed SSH from anywhere on old node groups. Moving nodes to private subnets removed that exposure entirely. Releasing idle Elastic IPs also removes the risk of "dangling DNS": a DNS record pointing at an address you released can later point at someone else's resource, so we checked DNS before releasing each one.

Prevention checklist

  • Disable auto-assign public IPv4 on subnets that do not need it.
  • Release Elastic IPs that are not used, after checking DNS and allowlists.
  • Prefer internal load balancers for internal services.
  • Use IPAM Public IP Insights to see every public address.
  • Keep workloads in private subnets and reach the internet through NAT or endpoints.

References

Add a label

#aws #cost-optimization #ipv4 #vpc #elastic-ip
Written by
Ryan Pham @ryan_bugs

I build and run the platforms other engineers ship on: CI/CD pipelines, Kubernetes clusters, GPU inference services, LLM features in production, and the cloud bills that come with all of it. I have also spent years writing and operating automated trading systems, where a rounding bug or a missed websocket message is not a log line but a position you did not intend to hold. Each post here starts from a real failure: the alert, the exact error, what we ruled out, the root cause, and how we proved the fix. Same path every time: Investigate, Explain, Fix, Verify. No hype, no price predictions, just engineering.

Advertisement

Comments

0

No comments yet. Be the first to comment!

Sign In to post a comment.