Tag See all →

#transformers

2 stories

Banner: batched text generation broken by right padding in a decoder-only language model

AI & Machine Learning Bug Report

Batched LLM generation garbles short prompts: right-padding with a decoder-only model

Ryan Pham · Jul 23, 2025 · 5 min read

Our batch classification job with a small open model produced good output for the longest prompt in each batch and nonsense for the rest. Transformers warned that right-padding was detected for a decoder-only model. Left padding, a proper pad token and the attention mask fixed it.

Banner: a GPU memory bar filling up with fragmented blocks until it overflows during model inference

Performance & Scaling Bug Report

PyTorch CUDA out of memory in an inference service: autograd, batch size and fragmentation

Daniel Tran · Dec 26, 2024 · 9 min read

Our text classification service crashed with CUDA out of memory every few hours, always on a big batch. Logits kept for drift monitoring were holding autograd graphs alive, batches were sized by count not tokens, and the allocator was fragmented. How we measured each one and fixed it.