Tag See all →

#batching

2 stories

Performance & Scaling Bug Report

Triton dynamic batching doubled throughput and also doubled p99 latency

Ryan Pham · Nov 12, 2025 · 5 min read

Enabling dynamic batching in Triton Inference Server doubled throughput for our ranking model and pushed p99 latency from 38 to 85 ms, breaking our SLO. A large queue delay and preferred batch sizes made requests wait for batches that rarely filled at low traffic. Measured tuning fixed both.

Banner: batched text generation broken by right padding in a decoder-only language model

AI & Machine Learning Bug Report

Batched LLM generation garbles short prompts: right-padding with a decoder-only model

Ryan Pham · Jul 23, 2025 · 5 min read

Our batch classification job with a small open model produced good output for the longest prompt in each batch and nonsense for the rest. Transformers warned that right-padding was detected for a decoder-only model. Left padding, a proper pad token and the attention mask fixed it.