CloudWatch Logs bill tripled: debug logging left on and health checks logged

The monthly cost review flagged CloudWatch: 6,900 USD in October against 2,100 USD in September. No new services had launched. Cost Explorer, filtered to the CloudWatch service and grouped by usage type, put almost all of it in one line:

Usage type                     Cost (USD)    Usage
USE1-DataProcessing-Bytes        6,214.30    12,428 GB
USE1-TimedStorage-ByteHrs          402.11
USE1-CW:MetricMonitorUsage         188.40

DataProcessing-Bytes for CloudWatch Logs is log ingestion. We had ingested 12.4 TB of logs in a month, up from about 3.8 TB.

Reproducing it

To find which log groups were responsible, we used the IncomingBytes metric per log group. A short script ranks them:

for lg in $(aws logs describe-log-groups --query 'logGroups[].logGroupName' --output text); do
  bytes=$(aws cloudwatch get-metric-statistics --namespace AWS/Logs --metric-name IncomingBytes \
    --dimensions Name=LogGroupName,Value="$lg" --start-time 2024-10-01T00:00:00Z \
    --end-time 2024-11-01T00:00:00Z --period 2678400 --statistics Sum \
    --query 'Datapoints[0].Sum' --output text)
  echo "$bytes $lg"
done | sort -rn | head -5

6.91e+12 /ecs/orders-api
2.02e+12 /ecs/catalog-api
8.80e+11 /aws/lambda/image-resizer
...

orders-api alone ingested 6.9 TB. Sampling its logs showed two patterns:

DEBUG orders.repository: SELECT ... params={...}  (thousands per second)
INFO  http: GET /healthz 200 1ms

What we ruled out

• A traffic increase. Requests to orders-api grew 4 percent month over month.
• A logging agent duplicating lines. Each request produced each line once.
• Storage costs. Retention was set, and storage was a small part of the bill.

Root cause

During an incident on October 3, an engineer set LOG_LEVEL=DEBUG on orders-api through the ECS task definition to capture SQL statements. The incident was resolved; the setting was not reverted. Debug logs included every SQL query with parameters, about 1.2 KB per line. Separately, the load balancer health check hit /healthz every 5 seconds on each of 24 tasks from multiple load balancer nodes, and each check was logged at INFO. That second issue had existed for a year and accounted for about 10 percent of ingestion.

What is happening under the hood

CloudWatch Logs charges per GB ingested, per GB-month stored, and for queries. Ingestion is usually the largest part. In us-east-1, the Standard log class costs 0.50 USD per GB ingested; the Infrequent Access class costs less per GB but supports fewer features. Every byte written by awslogs from ECS, by Lambda, or by the CloudWatch agent is billed on ingestion, before you ever look at it.

Debug logging is often 10 to 100 times the volume of info logging. Health checks look tiny but run constantly: 24 tasks, checked every 5 seconds from several load balancer nodes, is hundreds of thousands of lines per day.

Environment

Minimal reproduction

Debug SQL logging:  ~2,600 lines/s x 1.2 KB x 86,400 s  = ~270 GB/day
Health checks:      ~250,000 lines/day x 120 B           = ~30 MB/day per task group, plus framework lines
Monthly debug cost: 270 GB x 30 x 0.50 USD              = ~4,000 USD

The fix

1. Revert the log level, and make it impossible to forget. LOG_LEVEL is now read from a parameter with an expiry. Our service template reads LOG_LEVEL_OVERRIDE and LOG_LEVEL_OVERRIDE_UNTIL; after the timestamp passes, the app logs a warning and reverts to INFO on its own:

@Scheduled(fixedDelay = 60_000)
void enforceLogLevelExpiry() {
    if (overrideUntil != null && Instant.now().isAfter(overrideUntil)) {
        loggingSystem.setLogLevel("ROOT", LogLevel.INFO);
        log.warn("log level override expired at {}, reverted to INFO", overrideUntil);
        overrideUntil = null;
    }
}

2. Stop logging health checks. We excluded /healthz from the access log filter. Health check failures are visible in load balancer metrics; successful checks carry no information.

3. Sample high-volume debug data instead of logging all of it. When we need SQL visibility now, we enable tracing with a sample rate, not debug logs for every request.

4. Retention on every log group. Several groups had "never expire". We set 30 days by default, 365 for audit logs, through Terraform:

resource "aws_cloudwatch_log_group" "service" {
  name              = "/ecs/${var.service}"
  retention_in_days = var.retention_days   # default 30
  log_group_class   = var.infrequent_access ? "INFREQUENT_ACCESS" : "STANDARD"
}

Alternatives and trade-offs

• Infrequent Access log class for groups you rarely query in real time. Cheaper ingestion, fewer features (no metric filters or live tail, for example). We use it for some batch job logs.
• Ship logs elsewhere (object storage with a query engine). Much cheaper per GB for high volumes, more work to operate.
• Subscription filters to drop logs. Filtering happens after ingestion, so you still pay for it. Drop noise at the source.

How to test the fix

After deploying the changes, we compared IncomingBytes for the log group over the same 6-hour weekday window:

before: 41.3 GB / 6h
after:   2.9 GB / 6h

We also set an override with a 10-minute expiry in staging and confirmed the level reverted automatically.

Confirming in production

• November CloudWatch spend: 1,310 USD, 81 percent lower than October and 38 percent lower than September thanks to the health check and retention changes.
• No loss of useful information: the on-call team reviewed a week of incidents and did not miss anything that the removed logs had provided.

Regression guard

• A CloudWatch alarm on IncomingBytes per log group: more than twice the trailing 7-day average for 3 hours triggers a message to the owning team.
• AWS Cost Anomaly Detection with a monitor on CloudWatch.
• Terraform policy: every log group must have retention_in_days.

Security implications

Debug logs contained SQL parameters, which included customer emails and addresses. Logging them for a month put personal data in a place with broader access than the database, which is a privacy issue in itself. We purged the affected log streams after confirming nothing was needed for the incident review, and the logging library now masks known personal data fields even at debug level.

Prevention checklist

• Give temporary log level changes an expiry.
• Do not log successful health checks.
• Set retention on every log group.
• Alert on ingestion per log group, not just on the total bill.
• Never log raw query parameters or request bodies by default.

References

• Amazon CloudWatch pricing — https://aws.amazon.com/cloudwatch/pricing/
• CloudWatch Logs metrics (IncomingBytes) — https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch-Logs-Monitoring-CloudWatch-Metrics.html
• CloudWatch Logs log classes — https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch_Logs_Log_Classes.html
• Analyzing CloudWatch usage in Cost Explorer — https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_billing.html