#vllm

DevOps & CI/CD Bug Report

GPU inference never scaled under load: HPA on CPU for a GPU-bound LLM server

Ryan Pham · Jul 29, 2026 · 5 min read

Our vLLM deployment had an HPA targeting 70 percent CPU. Under a load spike, requests queued for 40 seconds while CPU sat at 15 percent, so nothing scaled. GPU-bound servers need to scale on queue depth. We switched to KEDA on vLLM's waiting-requests metric and fixed warm-up.

AI & Machine Learning Bug Report

vLLM fails to start: max seq len is larger than the tokens the KV cache can store

Ryan Pham · Mar 26, 2025 · 5 min read

Serving Llama 3.1 8B with vLLM on a 24 GB GPU crashed at startup with a KV cache error. By default vLLM sizes for the model's full 128k context, and after loading weights there was room for about 45k tokens of KV cache. Setting max_model_len to what we need fixed it.