During a traffic spike, new GPU pods for our LLM service took nine minutes to become ready, and users waited or got errors. The image contained 15 GB of model weights, so every new node pulled the whole thing. Separating weights from the image and caching them per node brought scale-up under two minutes.
Enabling dynamic batching in Triton Inference Server doubled throughput for our ranking model and pushed p99 latency from 38 to 85 ms, breaking our SLO. A large queue delay and preferred batch sizes made requests wait for batches that rarely filled at low traffic. Measured tuning fixed both.
A new model that wins offline can still lose in production, through skew, latency, or effects on users that offline data cannot show. This guide covers a staged rollout for models: offline gates, shadow traffic, canaries on business metrics, A/B tests when needed, and fast rollback.
Our fraud model scored 0.91 AUC on holdout data and close to random in production. The serving service re-implemented preprocessing in another language: the scaler was refit on each batch, one-hot categories came in a different order and text was not lowercased. Shipping the pipeline with the model fixed it.