DEEPSEEK-V4-PRO ON H20-141G*8 SINGLE NODE ERROR

[yiminghub2024]

DEEPSEEK-V4-PRO ON H20-141G*8 SINGLE NODE ERROR

GUIDE https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro?features=tool_calling%2Creasoning%2Cspec_decoding&strategy=single_node_dep

ERROR

RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/../jit_kernels/impls/smxx_fp8_fp4_paged_mqa_logits.hpp:223): next_n == 1 or next_n == 2

I try both ep+dp=8 and ep+tp=8 both error is this

could any export help me ?

many thanks

[zyongye]

Please update your deep_gemm version to top of tree. If you're using docker you shouldn't see this.

[yiminghub2024]

i use docker as your guide , its not work ,its not deep_gemm's bussiness
after delete --speculative_config '{"method":"mtp","num_speculative_tokens":2}' ,it works .so its vllm with mtp error ,please check and fix it ,thanks @zyongye

[yiminghub2024]

i try both vllm/vllm-openai:deepseekv4-cu130 and vllm/vllm-openai:deepseekv4-cu129 ,its not work with mtp

[zyongye]

lemme take a look. Thanks for reporting this

[yiminghub2024]

annother bug : ,vllm console log only have "POST /v1/chat/completions HTTP/1.1" 200 OK ,not running request : xxx waiting request :xxx AVG TPS display ,please check @zyongye

[yiminghub2024]

@zyongye do you have wechat? i have a lot of issure want to report directly ?

[zyongye]

annother bug : ,vllm console log only have "POST /v1/chat/completions HTTP/1.1" 200 OK ,not running request : xxx waiting request :xxx AVG TPS display ,please check @zyongye

This is expected in DEP=8 case because we spawn multiply api server. There's no directly to show stats with this. But you can try reducing api server count by adding --api-server-count=1.

[yiminghub2024]

what's the benifit of design muti api-server? @zyongye

[yiminghub2024]

if i use --api-server-count=1 ,what's the problem i will meet ? @zyongye

[zyongye]

I think just one api-server will become bottleneck. you can always run /metrics to see everything you said

[yiminghub2024]

another bug , kvcache size and nX not match , my kvcache total is 3K,but your console says: it can support 800K 1.04X , please fix it @zyongye

[yiminghub2024]

I think just one api-server will become bottleneck. you can always run /metrics to see everything you said

if this is true ,all other model i use tp=8 , so vllm will become bottleneck? tp=8 have only one api-server? @zyongye

[yiminghub2024]

I think just one api-server will become bottleneck. you can always run /metrics to see everything you said

yes, grafana is correct .

[JeremyGe07]

想问下单台H20上速度咋样，老哥，毕竟不是原生支持FP4 @yiminghub2024

[yiminghub2024]

@JeremyGe07 不开MTP TPS:40 开了MTP=1 TPS:60-70