Triton dynamic batching doubled throughput and also doubled p99 latency

Our product ranking model runs on NVIDIA Triton Inference Server with a 50 ms p99 SLO for the ranking call. To reduce GPU count, we enabled dynamic batching. Load tests at peak traffic looked great: throughput per GPU doubled. In production, p99 latency went from 38 ms to 85 ms, and the SLO alert fired every night between 01:00 and 07:00.

Triton's metrics (with summary metrics enabled for queue and compute durations) showed where time went:

                     before    after (night)   after (peak)
queue time p99       0.3 ms    46 ms           8 ms
compute time p99     31 ms     33 ms           36 ms
avg batch size       1         2.1             11.4

At night, queue time dominated. At peak, batching worked as intended.

Reproducing it

perf_analyzer at low and high concurrency against the batching configuration:

perf_analyzer -m ranker --concurrency-range 1:16:3 --percentile=99 -u localhost:8001 -i grpc

Concurrency: 1,  throughput: 21 infer/sec,  p99 latency: 81,900 usec
Concurrency: 16, throughput: 690 infer/sec, p99 latency: 41,300 usec

Low concurrency had worse latency than high concurrency, the signature of waiting for batches.

What we ruled out

• GPU contention. Compute duration was stable across times of day.
• Network. Client-side and server-side timings agreed within a millisecond.

Root cause

Our model configuration:

dynamic_batching {
  preferred_batch_size: [ 16, 32 ]
  max_queue_delay_microseconds: 50000
}

With preferred batch sizes set, Triton tries to form a batch of a preferred size and waits up to max_queue_delay_microseconds (here 50 ms) for enough requests to arrive. At peak, 16 requests arrive within a few milliseconds. At night, they do not, so most requests waited close to the full 50 ms before being executed in a small batch. The values had been copied from a throughput-oriented example.

What is happening under the hood

Dynamic batching combines individual requests into a batch on the server. A batch is executed when one of these happens:

• A preferred batch size is reached (if configured).
• The maximum batch size is reached.
• The oldest request has waited max_queue_delay_microseconds.

Without preferred_batch_size, Triton forms the largest batch possible from whatever requests are queued when the model becomes available, which already batches well under load. A queue delay above zero adds latency in exchange for bigger batches. The right delay depends on how long a batch takes to compute and on your latency budget, not on throughput alone.

Environment

Minimal reproduction

Any model with a large max_queue_delay_microseconds and preferred batch sizes above the typical queue depth will show higher latency at low load than at high load in perf_analyzer.

The fix

We tuned with perf_analyzer across the real traffic range, measuring p99 at concurrency 1, 4, 16 and 32, and changed the config to:

dynamic_batching {
  max_queue_delay_microseconds: 2000
}
instance_group [ { count: 2, kind: KIND_GPU } ]

• No preferred_batch_size: Triton batches whatever is queued, which naturally yields large batches at peak and immediate execution at night.
• A 2 ms queue delay: enough to gather a few extra requests at medium load, small compared with the 31 ms compute time.
• Two model instances per GPU, so one batch can execute while the next is being formed.

Results from perf_analyzer:

Concurrency  throughput     p99 latency
1            29 infer/sec   34.1 ms
4            118 infer/sec  36.8 ms
16           655 infer/sec  41.0 ms
32           1,080 infer/sec 47.9 ms

Throughput at peak stayed within 5 percent of the aggressive configuration, and p99 stayed under the SLO across the range.

Alternatives and trade-offs

• Priority queues or separate models for latency-critical and batch traffic. Useful when one server handles both online and offline requests.
• Client-side batching. The ranking service already sends several items per request; a larger client batch reduces request overhead but is fixed regardless of load.
• Model Analyzer to search configurations automatically. Good for larger search spaces; our manual sweep covered the two parameters that mattered.

How to test the fix

The perf_analyzer sweep runs in CI for model configuration changes against a staging GPU, and fails if p99 at concurrency 1 or at the expected peak exceeds 45 ms.

Confirming in production

• p99 latency for the ranking call: 36 ms at night, 44 ms at peak, no SLO alerts over the following month.
• GPU count stayed at the reduced level achieved with batching.
• Average batch size at peak is 9.8, slightly lower than before but sufficient.

Regression guard

• Queue duration p99 and average batch size per model on the serving dashboard, with an alert when queue duration p99 exceeds 10 ms.
• Model config changes require the perf sweep results in the pull request.

Security implications

None directly. Triton's HTTP and gRPC endpoints, including the model control and metrics endpoints, are only reachable inside the cluster, and model repository writes are restricted to the deployment pipeline.

Prevention checklist

• Tune batching for your latency budget across the whole traffic range, not only at peak.
• Avoid preferred batch sizes unless measurements show a benefit.
• Keep max_queue_delay_microseconds small relative to compute time.
• Watch queue duration separately from compute duration.
• Re-test when traffic patterns or models change.

References

• Triton model configuration, dynamic batcher — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_configuration.html#dynamic-batcher
• Triton batcher documentation — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/batcher.html
• perf_analyzer — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/README.html
• Triton metrics — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/metrics.html