Monitoring bill jumped after one label change: high-cardinality custom metrics

Our hosted Prometheus-compatible metrics service bills by active time series. The invoice for February was four times January's. At the same time, dashboards that used to load in two seconds took twenty, and some alerts evaluated late.

The usage page showed active series:

Jan 31: 1.21 M active series
Feb 06: 1.24 M
Feb 07: 9.80 M
Feb 10: 19.4 M

Reproducing it

Finding the metric responsible with a cardinality query:

topk(10, count by (__name__) ({__name__=~".+"}))

http_server_request_duration_seconds_bucket   17,912,440
http_server_request_duration_seconds_count        942,760
container_cpu_usage_seconds_total                  84,120

And which label exploded:

count(count by (user_id) (http_server_request_duration_seconds_count{service="accounts-api"}))
# 78,412

A commit on February 6 had added user_id to the request duration histogram in accounts-api, to investigate complaints from a few slow accounts.

What we ruled out

• More pods or services. Pod counts were stable.
• Scrape interval changes. Billing was by active series, and the interval had not changed.

Root cause

Every unique combination of label values is its own time series. The histogram already had labels for route (40 values), method, status and 12 buckets. Adding user_id with about 78,000 active users per week multiplied that by tens of thousands. Each user who made a request created new series for every bucket, route and status combination they touched. The metrics backend, priced and sized per active series, grew accordingly.

What is happening under the hood

Cardinality is the number of distinct series for a metric: the product of the number of values of each label, limited by which combinations actually occur. Histograms multiply it further, one series per bucket plus _sum and _count.

Labels must have small, bounded value sets: status code, method, route template, region. Unbounded values (user ids, request ids, email addresses, full URLs with ids, session ids) belong in logs or traces, which are designed for high-cardinality data and are sampled or retained differently.

Most hosted metrics products price by active series or by samples ingested, and query performance degrades with series count, so cardinality is both a cost and a reliability problem.

Environment

Minimal reproduction

REQUEST_DURATION = Histogram(
    "http_server_request_duration_seconds", "Request duration",
    ["route", "method", "status", "user_id"],      # user_id is unbounded
)

The fix

1. Drop the label at the collector immediately, so the fix did not wait for a deploy:

# Prometheus agent scrape config for accounts-api
metric_relabel_configs:
  - action: labeldrop
    regex: user_id

Dropping a label merges the series back into the original aggregates. Active series dropped to 1.3 million within the hour.

2. Remove it from the code, and give the developer what they needed in a better place: the user id is recorded as a span attribute on traces, and request logs include it. Finding slow requests for a specific account is now a trace search filtered by attribute and duration.

3. Use exemplars where helpful: a histogram observation can carry a trace id as an exemplar, linking a slow bucket to a specific trace without adding a label.

4. Set limits. Per-scrape sample limits in Prometheus, and cardinality limits in the hosted backend per tenant, so one bad change hits a limit and alerts instead of quadrupling the bill:

scrape_configs:
  - job_name: accounts-api
    sample_limit: 50000
    label_limit: 12
    label_value_length_limit: 200

When a target exceeds sample_limit, the scrape fails and the up metric drops to 0 for that target, which fires our "target down" alert. That is noisy by design.

Alternatives and trade-offs

• Bucket users into tiers (plan="enterprise") instead of ids. Bounded and often answers the real question.
• Recording rules that pre-aggregate and drop high-cardinality raw series. Useful when you cannot change instrumentation quickly.
• Native histograms reduce bucket series counts, but do not fix unbounded labels.

How to test the fix

A CI step runs the service's test suite with a metrics registry and fails if any metric has a label not in an allowlist per metric, or more than 1,000 distinct values for any label during the test run.

Confirming in production

• Active series back to about 1.3 million; March invoice at January level.
• Dashboard load times back to two seconds.
• The slow-account investigation was completed with traces in a day.

Regression guard

• Alert on active series growth above 20 percent day over day, per service.

• Weekly report of top metrics and labels by cardinality.

• Instrumentation guidelines with a list of forbidden label types, linked in the PR template.

• A pre-merge review rule: any new metric label in instrumentation code needs a comment stating its expected number of values. Reviewers question anything that could grow with users, requests, or time.

• A quarterly cleanup removes metrics that no dashboard or alert has queried in 90 days, using the backend's query logs. The first cleanup removed 140 unused metrics and about 15 percent of active series.

Security implications

User ids in metric labels also spread identifiers to a system with broad read access and long retention, where they are hard to delete on request. Keeping user-level data in traces and logs, with their own access controls and retention, is better for privacy as well as cost.

Prevention checklist

• Only bounded values as metric labels.
• Put per-user and per-request detail in traces and logs.
• Drop bad labels at the collector as the fastest fix.
• Set sample and cardinality limits so mistakes fail loudly.
• Check label cardinality in CI.

References

• Prometheus naming and labels best practices — https://prometheus.io/docs/practices/naming/#labels
• Prometheus relabelling (metric_relabel_configs) — https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs
• Prometheus scrape limits — https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
• OpenTelemetry exemplars — https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exemplars