AWS Budget alert arrived two days after the spend spike started

On a Sunday afternoon, the finance lead forwarded an AWS Budgets email: the account had exceeded 100 percent of its monthly budget, on the 10th of the month. The on-call engineer found 60 g5.12xlarge instances running in a machine learning account, launched on Friday at 19:40 by a training pipeline whose retry logic had gone wrong. Each attempt launched new instances without terminating the previous ones.

By the time the instances were stopped, about 14,000 USD had been spent. The budget alert fired roughly 44 hours after the first instance started.

Reproducing it

The budget configuration explained most of the delay:

Budget:      ml-account-monthly
Amount:      40,000 USD per month
Alerts:      80% actual, 100% actual, 100% forecasted

Spend before Friday was about 12,000 USD. The account needed to accumulate another 20,000 USD before the 80 percent alert, and the forecast alert did not trigger earlier because forecasts need enough data and the spike was recent.

What we ruled out

• A broken alert subscription. The emails were delivered; they were simply late relative to the incident.
• Delayed instance tagging or wrong account scope. The budget covered the right account.

Root cause

AWS Budgets works on billing data, which is not real time: cost and usage data refreshes several times a day and can lag hours behind actual usage. On top of that, our only budget was monthly with thresholds at 80 and 100 percent. A spike that is huge per hour but starts early in the month needs a long time to cross a monthly percentage. Budgets are good for tracking spend against plan; they are not a fast incident detector.

What is happening under the hood

Three tools, three speeds:

For fast detection of runaway resources, cost data is the wrong signal. Resource counts and usage metrics are available in minutes: running instance counts per type, Lambda invocations, data transfer bytes, Bedrock or SageMaker usage, API calls in CloudTrail.

Environment

Minimal reproduction

Not something to reproduce. The arithmetic:

60 instances x ~5.67 USD/hour x 40 hours  ~= 13,600 USD
Monthly budget 40,000; spend before spike 12,000; 80% threshold at 32,000
-> threshold crossed only after ~20,000 USD of extra spend, plus billing data lag

The fix

1. Daily budgets per account with low thresholds, alongside the monthly ones. A daily budget on the ML account at 2,000 USD (normal day: about 1,100) alerts on the day of a spike instead of days later.

resource "aws_budgets_budget" "ml_daily" {
  name         = "ml-account-daily"
  budget_type  = "COST"
  limit_amount = "2000"
  limit_unit   = "USD"
  time_unit    = "DAILY"

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 100
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_sns_topic_arns  = [aws_sns_topic.cost_alerts.arn]
  }
}

2. Cost Anomaly Detection monitors per linked account and per service, with alerts to the on-call channel through SNS and a chat integration, and an absolute impact threshold so small anomalies do not page anyone.

3. Usage guardrails in minutes. A CloudWatch alarm on the number of running GPU instances in the account, based on EC2 usage metrics:

resource "aws_cloudwatch_metric_alarm" "gpu_instances" {
  alarm_name          = "ml-running-g-instances"
  namespace           = "AWS/Usage"
  metric_name         = "ResourceCount"
  dimensions = {
    Service  = "EC2"
    Type     = "Resource"
    Resource = "vCPU"
    Class    = "G/OnDemand"     # on-demand G and VT instance vCPUs
  }
  statistic           = "Maximum"
  period              = 300
  evaluation_periods  = 2
  threshold           = 192          # 4 x g5.12xlarge worth of vCPUs
  comparison_operator = "GreaterThanThreshold"
  alarm_actions       = [aws_sns_topic.oncall.arn]
}

4. Hard limits. Service Quotas for G instance vCPUs in the ML account were lowered to what the team actually needs, so a runaway launcher hits a quota error instead of a bill. The launcher itself now tags instances with a run id and terminates previous attempts before retrying.

Alternatives and trade-offs

• Budget actions can apply an IAM policy or stop instances when a threshold is reached. Powerful, but stopping resources automatically is risky in production accounts; we use it only in sandboxes.
• Third-party cost tools with near real-time estimates. They still mostly depend on the same billing data, plus their own usage polling.

How to test the fix

In a sandbox account we launched GPU instances above the alarm threshold. The usage alarm fired after 10 minutes, the daily budget alert the same evening, and the anomaly detection alert the next morning, which shows why we need all three.

Confirming in production

• A month later, a different misconfiguration launched 12 extra instances in the ML account; the usage alarm paged on-call within 12 minutes, and the cost impact was under 150 USD.
• Daily budget alerts fired twice for legitimate experiments, which the team acknowledged with a note; noise has been acceptable.

Regression guard

• Every account gets daily and monthly budgets from a Terraform module at creation.
• Usage alarms for expensive resource classes (GPU instances, large databases, NAT data) in all accounts.
• Service quotas reviewed quarterly against actual needs.

Security implications

Runaway spend is also what compromised credentials often look like: attackers launching GPU instances for cryptocurrency mining is a common pattern. The same fast usage alarms help detect that. Low service quotas in accounts that do not need GPUs, and SCPs that deny unused regions, limit the damage of both mistakes and compromises.

Prevention checklist

• Do not rely on monthly budgets for incident detection.
• Add daily budgets and Cost Anomaly Detection per account.
• Alarm on resource usage metrics for expensive services, in minutes.
• Set service quotas close to real needs.
• Make retry logic clean up previous attempts.

References

• AWS Budgets user guide — https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html
• AWS Cost Anomaly Detection — https://docs.aws.amazon.com/cost-management/latest/userguide/manage-ad.html
• CloudWatch usage metrics — https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Usage-Metrics.html
• Service Quotas — https://docs.aws.amazon.com/servicequotas/latest/userguide/intro.html