Cross-AZ data transfer quietly became our third biggest AWS cost

In our quarterly cost review, a usage type nobody had looked at before was suddenly third on the list:

Usage type                        Cost (USD, monthly)
EUW1-BoxUsage:m6i.2xlarge            31,400
EUW1-RDS:Multi-AZUsage               14,900
EUW1-DataTransfer-Regional-Bytes     11,060

DataTransfer-Regional-Bytes is data moving between availability zones in the same region, billed in both directions. 11,060 USD at 0.01 USD per GB each way meant about 553 TB of cross-zone traffic a month.

Reproducing it

We enabled VPC Flow Logs with the az-id related fields and queried them in Athena, grouping by source and destination zone and by the Kubernetes workload behind each IP (joined from a periodic export of pod IPs):

SELECT src_workload, dst_workload, sum(bytes) / 1e12 AS tb
FROM flows_with_workloads
WHERE src_az <> dst_az
GROUP BY 1, 2
ORDER BY tb DESC
LIMIT 5;

src_workload        dst_workload          tb
orders-consumer     kafka-broker          188.2
kafka-broker        kafka-broker          121.7
catalog-api         catalog-cache(redis)   74.9
checkout-api        catalog-api            51.3
search-indexer      kafka-broker           40.6

What we ruled out

• Internet egress. DataTransfer-Out-Bytes was stable.
• NAT gateway processing. Separate usage type, and small.
• A misconfigured replication job. Kafka replication between brokers is expected cross-zone traffic for durability; it was large but explainable.

Root cause

Our services ran in all three zones for resilience, which is right. But traffic between them ignored zones entirely. Kubernetes Services load-balanced across all endpoints, so roughly two thirds of calls crossed a zone boundary. Kafka consumers always fetch from the partition leader, which is in a random zone relative to the consumer, so most consumer reads crossed zones too. As traffic grew, this hidden multiplier grew with it.

What is happening under the hood

AWS charges for data transfer between availability zones in the same region, per GB in each direction. Traffic within a zone is free. For a service with replicas in three zones and uniform load balancing, about two thirds of requests cross zones, and both the request and the response are billed.

Kubernetes offers ways to prefer endpoints in the same zone:

• Topology Aware Routing, enabled per Service with the annotation service.kubernetes.io/topology-mode: Auto. The EndpointSlice controller adds zone hints when endpoints are spread evenly enough to be safe.
• Traffic distribution via the spec.trafficDistribution: PreferClose field on Services (beta in 1.31, GA later), a simpler way to express "prefer endpoints in my zone".

For Kafka, KIP-392 added follower fetching: consumers declare their rack (client.rack), brokers use a rack-aware replica selector, and consumers can read from an in-sync replica in their own zone instead of the leader.

Environment

Minimal reproduction

Deploy a client and a server Deployment with three replicas each across three zones, send traffic through a ClusterIP Service, and count per-zone flows. About 67 percent cross zones.

The fix

1. Prefer same-zone endpoints for internal services.

apiVersion: v1
kind: Service
metadata:
  name: catalog-api
spec:
  selector: {app: catalog-api}
  ports: [{port: 80, targetPort: 8080}]
  trafficDistribution: PreferClose

This only works well if each zone has enough capacity for its local traffic. We set topology spread constraints so replicas are balanced across zones, and HPAs scale per Deployment as before:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector: {matchLabels: {app: catalog-api}}

2. Kafka follower fetching. On brokers:

broker.rack=euw1-az1
replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector

On consumers, the rack is set from the node's zone, injected through the Downward API and an init script:

client.rack=euw1-az2

3. Read from the local Redis replica for read-only cache lookups, where slightly stale data is acceptable. Writes still go to the primary.

Alternatives and trade-offs

• Run everything in one zone. Cheapest network bill, worst availability. Not acceptable for us.
• Zone-local deployments (separate Deployments per zone with zone-specific Services). Maximum control, more objects to manage.
• Service mesh locality load balancing. Meshes can prefer local endpoints with failover; useful if you already run one.

Same-zone routing makes zones less independent in one way: if one zone's replicas are overloaded, traffic does not automatically spread unless the mechanism falls back. Monitor per-zone load after enabling it.

How to test the fix

We re-ran the flow log query for a week after each change. We also load-tested catalog-api with one zone's replicas scaled down to confirm traffic fell back to other zones without errors.

Confirming in production

• Cross-zone traffic dropped from about 553 TB to 177 TB per month; the line item fell from 11,060 to 3,540 USD.
• Kafka consumer fetch latency improved slightly, since reads are now local.
• p99 latency for checkout-api to catalog-api calls improved by about 1 ms, not a goal but welcome.

Regression guard

• A weekly Athena report of top cross-zone workload pairs, posted to the platform channel.
• A cost anomaly monitor on the regional data transfer usage type.
• Service templates use trafficDistribution: PreferClose by default for internal services, with an opt-out.

Security implications

None directly. Reading from replicas means a consumer in one zone reads data from a broker in that zone, which does not change who can access the data; ACLs apply the same way. The flow logs used for analysis contain internal IPs and traffic patterns, so they live in a restricted bucket with 30-day retention.

Prevention checklist

• Track DataTransfer-Regional-Bytes as its own line.
• Prefer same-zone routing for chatty internal services, with balanced replicas.
• Enable rack-aware follower fetching for Kafka consumers.
• Read from local replicas where staleness is acceptable.
• Use flow logs with zone fields to find the biggest pairs.

References

• AWS EC2 data transfer pricing — https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer
• Kubernetes Topology Aware Routing — https://kubernetes.io/docs/concepts/services-networking/topology-aware-routing/
• Kubernetes Service traffic distribution — https://kubernetes.io/docs/concepts/services-networking/service/#traffic-distribution
• KIP-392, allow consumers to fetch from closest replica — https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica