vLLM fails to start: max seq len is larger than the tokens the KV cache can store

We moved an internal summarisation service from a hosted API to self-hosted Llama 3.1 8B Instruct with vLLM on NVIDIA L4 GPUs (24 GB). The container started, loaded weights, profiled memory, and exited:

INFO  Loading model weights took 14.99 GB
INFO  Memory profiling takes 4.21 seconds
INFO  the current vLLM instance can use total_gpu_memory (22.05GiB) x gpu_memory_utilization (0.90) = 19.84GiB
INFO  model weights take 14.99GiB; non_torch_memory takes 0.12GiB; PyTorch activation peak memory takes 1.26GiB; the rest of the memory reserved for KV Cache is 3.47GiB.
...
ValueError: The model's max seq len (131072) is larger than the maximum number of tokens that can be
stored in KV cache (28432). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when
initializing the engine.

Reproducing it

vllm serve meta-llama/Llama-3.1-8B-Instruct

on any 24 GB GPU reproduces it. The same command worked on an 80 GB A100 in a colleague's test.

What we ruled out

• Another process using GPU memory. nvidia-smi showed nothing else on the GPU.
• A broken download or wrong dtype. Weights loaded in bfloat16 as expected, about 15 GB.

Root cause

Unless told otherwise, vLLM sets max_model_len to the model's maximum context, 131,072 tokens for Llama 3.1. It checks at startup that at least one sequence of that length fits in the KV cache. On a 24 GB GPU, after weights and activation memory, only about 3.5 GiB remained for KV cache, enough for roughly 28,000 tokens. Our summaries never needed more than 16,000 tokens of context.

What is happening under the hood

During generation, the model keeps keys and values for every token of every active sequence in every layer. For Llama 3.1 8B:

KV bytes per token = 2 (K and V) x 32 layers x 8 KV heads x 128 head dim x 2 bytes (bf16)
                   = 131,072 bytes = 128 KiB per token

128k tokens for one sequence would need 16 GiB of KV cache, more than the whole GPU has left after weights. vLLM manages KV cache in blocks (PagedAttention) and serves many sequences concurrently from the same pool, so the KV cache size also determines how many requests can run at once.

The knobs:

• --max-model-len: maximum context per request.
• --gpu-memory-utilization: fraction of GPU memory vLLM may use (default 0.9).
• --kv-cache-dtype fp8: halves KV cache memory on supported hardware, with a small quality trade-off.
• --tensor-parallel-size: splits the model across GPUs, freeing memory per GPU.
• Quantised weights (AWQ, GPTQ, FP8) free memory for KV cache.

Environment

Minimal reproduction

docker run --gpus all -p 8000:8000 vllm/vllm-openai:v0.8.1 --model meta-llama/Llama-3.1-8B-Instruct

The fix

Set the context to what the workload needs, with headroom, and check the resulting concurrency:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 32

With 3.6 GiB of KV cache and 128 KiB per token, the pool holds about 29,000 tokens, enough for one 16k request plus several shorter ones at the same time. vLLM logs the maximum concurrency for the configured length at startup, which we now check after every config change.

That concurrency was too low for our peak, so we also tested the FP8 KV cache:

  --kv-cache-dtype fp8

which roughly doubled KV capacity to about 58,000 tokens. Our summarisation eval set (200 documents, rubric-graded) showed no measurable quality difference, so we kept it.

The service also rejects inputs longer than 15,000 tokens before calling vLLM, with a clear error, so a very long document cannot occupy most of the cache.

Alternatives and trade-offs

• Bigger GPU or two GPUs with tensor parallelism. More KV cache and throughput, at a higher cost per hour.
• Quantised weights (AWQ or FP8). An 8B model in 4-bit uses about 5 GB, freeing a lot of memory for KV cache; quality impact must be evaluated per task.
• Raising gpu-memory-utilization close to 1.0. Gains a little, risks out-of-memory errors from CUDA graphs or other allocations.

How to test the fix

Startup succeeds, and a load test with our real document length distribution measured throughput and latency:

config                          max_concurrency  p95 latency  throughput (req/min)
max_len 16k, bf16 KV                 ~1.8x           21 s           38
max_len 16k, fp8 KV                  ~3.6x           14 s           71

Confirming in production

• The service has run for a month on two L4 nodes, replacing the hosted API for this workload.
• vLLM's metrics show KV cache usage peaking at about 85 percent at the busiest hour, with no preemptions.

Regression guard

• The deployment fails health checks if vLLM logs a maximum concurrency below 2.5 at startup.
• An alert on vllm:num_requests_waiting above 10 for 5 minutes and on KV cache usage above 95 percent.
• Input length limits enforced at the API layer.

Security implications

Self-hosting moved document contents off a third-party API, which was one of the reasons for the change. The vLLM server has no authentication by default, so it runs on a private network behind our API gateway, and the --api-key option is set as a second layer. The input length limit also protects against a single client exhausting the KV cache.

Prevention checklist

• Set --max-model-len to what the workload needs, not the model's maximum.
• Do the KV cache math: bytes per token times tokens needed times concurrency.
• Check the concurrency vLLM reports at startup.
• Consider FP8 KV cache or quantised weights, and evaluate quality.
• Enforce input length limits before the inference server.

References

• vLLM engine arguments — https://docs.vllm.ai/en/latest/serving/engine_args.html
• vLLM optimization and tuning — https://docs.vllm.ai/en/latest/performance/optimization.html
• vLLM quantized KV cache — https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache.html
• Efficient Memory Management for LLM Serving with PagedAttention — https://arxiv.org/abs/2309.06180