DevOps & CI/CD
Bug Report
An 18 GB inference image took 9 minutes to scale up: model weights baked into Docker
During a traffic spike, new GPU pods for our LLM service took nine minutes to become ready, and users waited or got errors. The image contained 15 GB of model weights, so every new node pulled the whole thing. Separating weights from the image and caching them per node brought scale-up under two minutes.