analyze performance for batching.

[geraldstanje]

hi,

i am trying to understand why latency increases by 4× when going from batch_size = 1 to batch_size = 8 for 512 tokens? - but i dont see an increase in latency for 128 tokens from batch_size = 1 to batch_size = 8
The following shows the GPU trace using NVIDIA Nsight.
in this context, batch_size means that the client sends a batched request and the server has dynamic batching enabled.
given that the SM utilization reaches almost 90% at batch_size = 8 for 512 tokens, does this mean that the GPU is already fully utilized?
what should I look at to determine whether the GPU is truly fully utilized SM and DRAM bandwith?

512 tokens, batch_size=1:
p90 latency: 12ms, rps: 82.49
gpu trace:

512 tokens, batch_size=8:
p90 latency: 39ms, rps: 24.92
gpu trace:

more zoomed out overview:

but when i look at 128 tokens the latency does not increase in latency.

128 tokens, batch_size=1:
p90 latency: 11ms, rps: 91
gpu trace:

512 tokens, batch_size=8:
p90 latency: 7ms, rps: 146
gpu trace:

cc @yuanyao-nv @brb-nv @lix19937 @YouSenRong @zhenhuaw-me @zhenhuaw-me @JuncFang-git @ttyio @ttyio @pranavm-nvidia

[zhenhuaw-me]

given that the SM utilization reaches almost 90% at batch_size = 8 for 512 tokens, does this mean that the GPU is already fully utilized?

I have a similar intuition. You can profiling more to confirm if the latency starts to increase when the occupation is sufficiently high.

[zhenhuaw-me]

Closing since it's like a general question not an issue reporting.

Please feel free to reopen if you see any issue that requires our support. Thanks!

[geraldstanje]

Hi @zhenhuaw-me i see the same with fp16 quantization - the SM was only 89% for batch_size of 8 - any idea?

512 tokens, batch_size: 1:
p90 latency: 4ms, rps: 270
gpu trace:

512 tokens, batch_size: 8:
p90 latency: 10ms, rps: 100 <--- latency also increased

also why are these long pauses here (e.g. between 940ms and 942ms)?

[zhenhuaw-me]

@nvpohanh Could you please nominate someone to support this issue?

@kevinch-nv Can we enable the "Discussions" for our repo? I think issues like this are good candidates to be converted to discussions.