A canary that a human watches on a dashboard is only as good as that human's attention. This guide covers automated canary analysis: which metrics to gate on, how to size steps and durations, how to handle low traffic, and the traps that make canaries pass when they should fail.
Our vLLM deployment had an HPA targeting 70 percent CPU. Under a load spike, requests queued for 40 seconds while CPU sat at 15 percent, so nothing scaled. GPU-bound servers need to scale on queue depth. We switched to KEDA on vLLM's waiting-requests metric and fixed warm-up.
A developer added user_id as a label on a request latency histogram to debug slow accounts. Active time series went from 1.2 million to 19 million, our hosted metrics bill quadrupled and the query layer slowed down. We found the culprit, dropped the label and set cardinality limits.