Our main cluster costs about 48,000 USD a month. On the AWS bill, that money appears as EC2 instances, EBS volumes, load balancers and data transfer, with tags like cluster=prod-eu. Fourteen teams run services on it. When finance asked "how much does the search team cost us?", nobody could answer with a number they would defend.

This guide is the allocation model we built, the trade-offs behind each decision, and how we present it so teams actually change something. It applies to EKS, GKE, AKS or self-managed clusters; the tooling we used is OpenCost, but the decisions are the same with any tool.

Step 1: get the cluster's total cost right

Allocation starts with a correct total. Include:

  • Node compute, at the price you actually pay (on-demand, Spot, after Savings Plans or committed use discounts).
  • Persistent volumes and snapshots.
  • Load balancers and public IPs created by Services and Ingresses.
  • Control plane fees for managed Kubernetes.
  • Data transfer, if you can attribute it (often the hardest part).

OpenCost can read on-demand prices from cloud APIs, and can be configured to use your actual billing data so discounts are reflected. Without that, allocated costs will not add up to the bill, and finance will notice.

Step 2: requests or usage?

Each pod reserves CPU and memory through requests and uses some amount of it. Which one do you charge for?

Method Charge for Pros Cons
Requests What the pod reserved Matches what blocks capacity; predictable Over-requesting teams pay for waste they created, which is fair but can feel harsh
Usage What the pod consumed Feels fair to teams Ignores that reserved capacity cannot be used by others
Max of requests and usage The larger of the two Covers both over-requesting and bursting above requests Slightly harder to explain

We use the maximum of requests and usage per pod per hour. Requests reserve capacity on nodes; a pod requesting 4 CPUs and using 0.5 still prevents others from being scheduled there. Charging by usage alone rewards over-requesting and hides the biggest source of waste in most clusters.

Memory and CPU need prices per unit. OpenCost derives them from node prices; we weight CPU and memory by their share of the node price for each instance family.

Step 3: idle capacity

After allocating to pods, part of the cluster cost remains: capacity on nodes that no pod requested. In our cluster this was 27 percent at first. Options:

  • Spread idle across teams in proportion to their allocated cost. Everyone pays a share, the total adds up.
  • Keep idle as a platform cost. The platform team owns cluster efficiency, so it owns idle.
  • Show idle separately to everyone, without assigning it.

We show idle as a separate line owned by the platform team, and the platform team has a target for it. That puts the incentive where the lever is: autoscaling, bin packing and node sizing are platform decisions. Teams are responsible for right-sizing their requests, which shows up in their own numbers.

Step 4: shared services

Ingress controllers, monitoring agents, logging DaemonSets, service mesh control planes and CoreDNS serve everyone. Allocating them by namespace would charge the platform namespace for everything. We distribute shared namespaces in proportion to each team's allocated cost. DaemonSets are a special case: their cost scales with node count, not with any one team, so they go into the shared pool too.

Step 5: map workloads to owners

Cost per namespace is only useful if namespaces map to owners. We require two labels on every namespace, enforced by an admission policy:

metadata:
  labels:
    team: search
    cost-center: cc-4120

Workloads that span teams use the same labels at the Deployment level, and OpenCost aggregates by label. Anything unlabelled goes into an "unallocated" bucket that is reviewed weekly until it is close to zero.

Step 6: present it so people act

Our monthly report per team has five numbers:

  1. Allocated cost this month and trend over three months.
  2. Efficiency: usage divided by requests for CPU and memory.
  3. Top five workloads by cost.
  4. Estimated savings if requests matched p95 usage.
  5. Share of shared and idle cost, for context.

The fourth number drives behaviour. "You could save 2,300 USD a month by lowering requests on these three Deployments" is actionable; a pie chart is not.

kubectl cost namespace --window 30d --show-efficiency

The kubectl cost plugin (or the OpenCost API) gives engineers the same numbers on demand, without waiting for the monthly report.

Step 7: close the loop with right-sizing

Allocation is only half the job; the point is to change behaviour. Each month we pick the ten workloads with the largest gap between requests and p95 usage and open a pull request against their Helm values with suggested requests, generated from a week of Prometheus data with a 20 percent buffer above p95 for CPU and 30 percent for memory. Teams review and merge or explain why not. Memory is handled more carefully than CPU, because a memory request that is too low leads to evictions and OOM kills, while a CPU request that is too low only means throttling under contention. The first round of these pull requests reduced requested CPU across the cluster by 31 percent and let the autoscaler remove eleven nodes, which also shrank the idle line the platform team owns.

Common mistakes

Mistake Why it hurts Do this instead
Allocating by usage only Rewards over-requesting Max of requests and usage
Using list prices Totals do not match the bill Reconcile with actual billing data
Hiding idle in team numbers Nobody owns cluster efficiency Show idle separately with an owner
No ownership labels Large unallocated bucket Enforce team labels on namespaces
Reports without actions Numbers ignored Include right-sizing savings per workload

Key takeaways

  • Start from a total that matches the bill, including discounts.
  • Allocate by the maximum of requests and usage.
  • Make idle capacity visible and give it an owner.
  • Distribute shared services proportionally.
  • Report savings opportunities, not just costs.

References