#triton

Triton dynamic batching doubled throughput and also doubled p99 latency

Enabling dynamic batching in Triton Inference Server doubled throughput for our ranking model and pushed p99 latency from 38 to 85 ms, breaking our SLO. A large queue delay and preferred batch sizes made requests wait for batches that rarely filled at low traffic. Measured tuning fixed both.