diff --git a/docs/npu/CAM_ASYNC_CONNECTOR_USER_GUIDE.md b/docs/npu/CAM_ASYNC_CONNECTOR_USER_GUIDE.md index 0a9140a1..a697717b 100644 --- a/docs/npu/CAM_ASYNC_CONNECTOR_USER_GUIDE.md +++ b/docs/npu/CAM_ASYNC_CONNECTOR_USER_GUIDE.md @@ -9,7 +9,9 @@ This guide describes the supported deployment shape, configuration contract, rank mapping, data flow, startup requirements, and current limitations. The [DeepSeek-V3.2 recipe](../../recipe/npu/CAMAsyncAFDConnector/deepseek_v3_2/README.md) links the current v0.26 launch scripts and retains the historical v0.19 -multi-node commands and measurements for provenance. +multi-node commands and measurements for provenance. The experimental +[DeepSeek-V4-Flash recipe](../../recipe/npu/CAMAsyncAFDConnector/deepseek_v4_flash/README.md) +documents a single-node DP4TP2 Attention + EP8 FFN FlashComm1 deployment. > [!WARNING] > The vLLM 0.26 CAM async port remains experimental. The linked PCP8 recipe and diff --git a/recipe/README.md b/recipe/README.md index 2b9f433a..3a685dbe 100644 --- a/recipe/README.md +++ b/recipe/README.md @@ -16,7 +16,8 @@ recipe/ │ └── deepseek_v2_lite/ └── npu/ ├── CAMAsyncAFDConnector/ - │ └── deepseek_v3_2/ + │ ├── deepseek_v3_2/ + │ └── deepseek_v4_flash/ └── CAMP2pAFDConnector/ └── deepseek_v3_2/ ``` @@ -36,6 +37,7 @@ Directory names follow these conventions: | GPU | `P2pNcclAFDConnector` | DeepSeek-V2-Lite | Decode | Validated | [Launch examples](gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md) | | Ascend NPU | `CAMP2pAFDConnector` | DeepSeek-V3.2 | Decode | Validated | [Synchronous decode](npu/CAMP2pAFDConnector/deepseek_v3_2/README.md) | | Ascend NPU | `CAMAsyncAFDConnector` | DeepSeek-V3.2 | Prefill / decode | Experimental v0.26 DP+TP/SP path; post-fix DP2TP8+EP16 token split reached `0.9522` strict match on the complete GSM8K evaluation; legacy PCP8 results are v0.19-only | [Async CAM](npu/CAMAsyncAFDConnector/deepseek_v3_2/README.md) | +| Ascend NPU | `CAMAsyncAFDConnector` | DeepSeek-V4-Flash W8A8 | Prefill / decode | Experimental v0.26 single-node DP4TP2 Attention + EP8 FFN tool stack for issue #227 performance validation | [Async CAM](npu/CAMAsyncAFDConnector/deepseek_v4_flash/README.md) | Open the model-level README before running a recipe. It documents the required hardware and runtime baseline, topology, environment variables, launch order, diff --git a/recipe/npu/CAMAsyncAFDConnector/deepseek_v4_flash/README.md b/recipe/npu/CAMAsyncAFDConnector/deepseek_v4_flash/README.md new file mode 100644 index 00000000..b88fa20b --- /dev/null +++ b/recipe/npu/CAMAsyncAFDConnector/deepseek_v4_flash/README.md @@ -0,0 +1,195 @@ +# DeepSeek-V4-Flash Async CAM Recipe + +This recipe launches DeepSeek-V4-Flash W8A8 on one 16-NPU Ascend 910C node: + +- NPUs 0-7: Attention `DP4 x TP2`, FlashComm1 sequence parallelism enabled; +- NPUs 8-15: FFN `DP8 x TP1 / EP8`, FlashComm1 disabled; +- one 16-rank CAM world containing 8 Attention and 8 FFN ranks; +- AFD-managed two-stage token split, dynamic CAM quantization, eager execution, + chunked prefill, and the vLLM async scheduler; +- `max_num_batched_tokens=65536`, shared compressor workspace, and a 4096 MB + CAM HCCL buffer. + +The two roles share one node but run as separate vLLM processes. Only the +Attention process owns the public inference endpoint. + +> [!WARNING] +> DeepSeek-V4 support over `CAMAsyncAFDConnector` remains experimental. This is +> the user-facing launch tool stack for the performance work tracked by +> [AFD issue #227](https://github.com/vllm-project/afd-plugin/issues/227), +> rewritten against the configuration contract on +> `vllm-project/afd-plugin:main`. The performance numbers below are historical +> evidence for the recorded validation cell; this recipe itself has not been +> requalified on NPU hardware. + +For the connector contract and package installation instructions, read the +[CAM Async Connector User Guide](../../../../docs/npu/CAM_ASYNC_CONNECTOR_USER_GUIDE.md). + +## Runtime and model + +The issue #227 performance cell used: + +- Ascend 910C, Python 3.12, and CANN 9.0.1; +- vLLM commit `568afb3a1`; +- vLLM-Ascend commit `80d8c194f`, the `2e01d4c5` head commit from the + still-unmerged + [vLLM-Ascend PR #15452](https://github.com/vllm-project/vllm-ascend/pull/15452), + and the follow-up slot-mapping fix `e19e14da`; +- CAM 209.x operator packages documented in the connector guide; +- the complete `DeepSeek-V4-Flash-w8a8-mtp` checkpoint. + +PR #15452 commit `2e01d4c5` provides the DeepSeek-V4 compressor-tail workspace +reuse; `e19e14da` applies its slot-mapping correction on top. Check out or +apply those exact commits in order. PR #15452 is still open, so do not assume a +released vLLM-Ascend checkout already contains CWS support. + +The checkpoint's `config.json` must select `model_type=deepseek_v4` or the +`DeepseekV4ForCausalLM` architecture. The checkpoint may contain MTP weights, +but this recipe does not enable speculative decoding. + +Install the AFD plugin from the target `main` checkout and install the CAM +operators before launching. Source the matching CANN environment in each +terminal. Verify that the four CAM async operators are registered: + +```bash +python3 - <<'PY' +import torch +import umdk_cam_op_lib # noqa: F401 + +for name in ( + "async_dispatch_send", + "async_dispatch_recv", + "async_combine_send", + "async_combine_recv", +): + assert hasattr(torch.ops.umdk_cam_op_lib, name), name +print("CAM async operators are available") +PY +``` + +## Why these settings + +The issue #227 performance matrix established the operational contract used +here: + +- FlashComm1 must be enabled only for Attention; FFN TP1 rejects it. +- Attention `DP4TP2` plus FFN `EP8` was the best measured single-node layout. +- Token split outperformed request split under sustained load in that layout. +- `max_num_batched_tokens=65536` requires the DeepSeek-V4 shared compressor + workspace from vLLM-Ascend `e19e14da7`. +- The `DP4TP2 / EP8`, MBT 65536 run failed with a smaller HCCL buffer and was + stable with 4096 MB. +- Both roles must use identical CAM rank counts, split settings, rendezvous + address, and `max_num_batched_tokens`. + +Do not add vLLM native DBO flags. `async_moe_ubatching` is the separate, +AFD-managed MoE pipeline. The recipe also deliberately leaves the experimental +`prefill_token_sum` Attention DPLB policy unset; the issue #227 measurements +found no throughput benefit over the request-count policy for the measured +workload, and that policy is not part of the target `main` configuration +contract. + +## Launch + +Run both commands on the same 16-NPU node. `LOCAL_IP` must be the address on +`NIC_NAME`, and the same reachable address is used as the CAM rendezvous host. +The launcher defaults to Attention devices 0-7 and FFN devices 8-15. + +Start FFN in terminal 1: + +```bash +source /usr/local/Ascend/cann-9.0.1/set_env.sh + +MODEL_PATH=/path/to/DeepSeek-V4-Flash-w8a8-mtp \ +LOCAL_IP= \ +NIC_NAME= \ +bash recipe/npu/CAMAsyncAFDConnector/deepseek_v4_flash/launch.sh ffn +``` + +Start Attention in terminal 2: + +```bash +source /usr/local/Ascend/cann-9.0.1/set_env.sh + +MODEL_PATH=/path/to/DeepSeek-V4-Flash-w8a8-mtp \ +LOCAL_IP= \ +NIC_NAME= \ +bash recipe/npu/CAMAsyncAFDConnector/deepseek_v4_flash/launch.sh attention +``` + +Wait for the Attention endpoint: + +```bash +until curl --fail --silent http://127.0.0.1:8000/v1/models >/dev/null; do + sleep 5 +done +``` + +Then send a smoke request: + +```bash +curl http://127.0.0.1:8000/v1/completions \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "deepseek_v4_flash", + "prompt": [1, 2, 3, 4, 5, 6, 7, 8], + "max_tokens": 1 + }' +``` + +## Overrides + +The launcher accepts these optional environment variables: + +| Variable | Default | Meaning | +| --- | --- | --- | +| `AFD_HOST` | `LOCAL_IP` | CAM rendezvous host; must identify Attention rank 0. | +| `AFD_PORT` | `1239` | CAM rendezvous port. | +| `ATTENTION_PORT` / `FFN_PORT` | `8000` / `8001` | Role-local API ports. | +| `ATTENTION_DEVICES` / `FFN_DEVICES` | `0-7` / `8-15` | Comma-separated device lists. | +| `MAX_MODEL_LEN` | `70000` | Maximum sequence length. | +| `MAX_NUM_BATCHED_TOKENS` | `65536` | Scheduler MBT; it must match on both roles. | +| `MAX_NUM_SEQS` | `128` | Maximum scheduled sequences. | +| `GPU_MEMORY_UTILIZATION` | `0.8` | Per-role device memory utilization. | +| `HCCL_BUFFER_SIZE_MB` | `4096` | Connector-scoped CAM HCCL buffer size. | +| `CAM_VENDOR_PATH` | CANN 9.0.1 CAM vendor path | Installed CAM operator root. | + +Keep the topology and model-semantic settings unchanged unless the new +combination is validated independently. + +## Issue #227 performance summary + +The matching single-node `DP4TP2 / EP8`, token-split, MBT 65536 matrix used a +512-request long-prefill workload. Every rate completed 512/512 requests with +no failures: + +| Offered rate | Effective input tokens/s | TTFT p99 | Peak 15s service rate | +| --- | ---: | ---: | ---: | +| 0.5x | 17,075 | 9.0s | 30K tokens/s | +| 0.75x | 25,230 | 10.7s | 41K tokens/s | +| 1.0x | 32,830 | 15.6s | 45K tokens/s | +| 1.25x | 37,321 | 25.6s | 47K tokens/s | +| 1.5x | 40,096 | 34.3s | 47K tokens/s | + +Against the best baseline point at each offered rate, the recorded AFD cell +improved effective throughput by 4.8% at 1.0x, 6.6% at 1.25x, and 8.6% at +1.5x. At 1.0x, TTFT p50 improved by 37.4%, TTFT p99 by 22.1%, and the fraction +of requests meeting a 10-second TTFT SLO increased from 65.2% to 85.2%. + +These values describe the exact issue #227 software, checkpoint, and workload +cell. They are not a portable performance guarantee or a validation result for +a different checkout. + +## Limitations + +Current constraints include: + +- eager execution only; ACL graph mode is unsupported; +- FlashComm1 only on Attention and only with TP greater than one; +- no context parallelism, prefix caching, speculative decoding, KV transfer, + or native DBO; +- exactly two AFD MoE stages; +- W8A8 CAM dynamic quantization is required for DeepSeek-V4-Flash; +- all ranks need the same CAM/CANN packages, connector settings, and model; +- teardown must remove both vLLM process trees before relaunching, or stale + workers can retain NPU memory and CAM port 1239. diff --git a/recipe/npu/CAMAsyncAFDConnector/deepseek_v4_flash/launch.sh b/recipe/npu/CAMAsyncAFDConnector/deepseek_v4_flash/launch.sh new file mode 100755 index 00000000..729d9ea9 --- /dev/null +++ b/recipe/npu/CAMAsyncAFDConnector/deepseek_v4_flash/launch.sh @@ -0,0 +1,158 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project + +set -euo pipefail + +usage() { + echo "Usage: MODEL_PATH= LOCAL_IP= NIC_NAME= $0 " >&2 + echo "Launch each role in a separate terminal." >&2 +} + +if [[ $# -eq 1 && ( "$1" == "-h" || "$1" == "--help" ) ]]; then + usage + exit 0 +fi +if [[ $# -ne 1 ]]; then + usage + exit 2 +fi +ROLE="$1" + +: "${MODEL_PATH:?Set MODEL_PATH to the DeepSeek-V4-Flash W8A8 checkpoint}" +: "${LOCAL_IP:?Set LOCAL_IP to this node communication IP}" +: "${NIC_NAME:?Set NIC_NAME to the NPU network interface}" + +AFD_HOST="${AFD_HOST:-$LOCAL_IP}" +AFD_PORT="${AFD_PORT:-1239}" +ATTENTION_PORT="${ATTENTION_PORT:-8000}" +FFN_PORT="${FFN_PORT:-8001}" +MAX_MODEL_LEN="${MAX_MODEL_LEN:-70000}" +MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-65536}" +MAX_NUM_SEQS="${MAX_NUM_SEQS:-128}" +GPU_MEMORY_UTILIZATION="${GPU_MEMORY_UTILIZATION:-0.8}" +HCCL_BUFFER_SIZE_MB="${HCCL_BUFFER_SIZE_MB:-4096}" +ATTENTION_DEVICES="${ATTENTION_DEVICES:-0,1,2,3,4,5,6,7}" +FFN_DEVICES="${FFN_DEVICES:-8,9,10,11,12,13,14,15}" +CAM_VENDOR_PATH="${CAM_VENDOR_PATH:-/usr/local/Ascend/cann-9.0.1/opp/vendors/CAM}" + +if [[ ! -f "$MODEL_PATH/config.json" ]]; then + echo "DeepSeek-V4 config not found: $MODEL_PATH/config.json" >&2 + exit 1 +fi + +case "$ROLE" in + attention) + API_PORT="$ATTENTION_PORT" + VISIBLE_DEVICES="$ATTENTION_DEVICES" + FLASHCOMM1=1 + WORKER_CLASS=afd_plugin.v1.worker.npu.AFDNPUAttentionWorker + PARALLEL_ARGS=( + --data-parallel-size 4 + --tensor-parallel-size 2 + ) + ;; + ffn) + API_PORT="$FFN_PORT" + VISIBLE_DEVICES="$FFN_DEVICES" + FLASHCOMM1=0 + WORKER_CLASS=afd_plugin.v1.worker.npu.AFDNPUFFNWorker + PARALLEL_ARGS=( + --data-parallel-size 8 + --tensor-parallel-size 1 + ) + ;; + *) + echo "ROLE must be attention or ffn, got: $ROLE" >&2 + exit 2 + ;; +esac + +CAM_OPAPI_DIR="$CAM_VENDOR_PATH/op_api/lib" +if [[ -f "$CAM_OPAPI_DIR/libopapi.so" ]]; then + CAM_OPAPI="$CAM_OPAPI_DIR/libopapi.so" + CAM_OPAPI_LOAD_PATH="$CAM_OPAPI_DIR" +elif [[ -f "$CAM_OPAPI_DIR/libcust_opapi.so" ]]; then + # CAM 209.x may ship only libcust_opapi.so, while UMDK resolves the vendor + # implementation by the libopapi.so name. Provide a process-local alias. + CAM_OPAPI_COMPAT_DIR="${CAM_OPAPI_COMPAT_DIR:-${TMPDIR:-/tmp}/afd-cam-opapi}" + mkdir -p "$CAM_OPAPI_COMPAT_DIR" + ln -sfn "$CAM_OPAPI_DIR/libcust_opapi.so" "$CAM_OPAPI_COMPAT_DIR/libopapi.so" + CAM_OPAPI="$CAM_OPAPI_COMPAT_DIR/libopapi.so" + CAM_OPAPI_LOAD_PATH="$CAM_OPAPI_COMPAT_DIR:$CAM_OPAPI_DIR" +else + echo "CAM op-api library not found under $CAM_OPAPI_DIR" >&2 + exit 1 +fi + +export ASCEND_RT_VISIBLE_DEVICES="$VISIBLE_DEVICES" +export ASCEND_CUSTOM_OPP_PATH="$CAM_VENDOR_PATH:${ASCEND_CUSTOM_OPP_PATH:-}" +export LD_LIBRARY_PATH="$CAM_OPAPI_LOAD_PATH:$CAM_VENDOR_PATH/op_api:${LD_LIBRARY_PATH:-}" +export CAM_CUST_OPAPI_LIB_PATH="$CAM_OPAPI" +export LD_PRELOAD="$CAM_OPAPI${LD_PRELOAD:+:$LD_PRELOAD}" +export VLLM_PLUGINS="${VLLM_PLUGINS:-ascend,afd}" +export VLLM_WORKER_MULTIPROC_METHOD="${VLLM_WORKER_MULTIPROC_METHOD:-spawn}" +export AFD_FORCE_SPAWN_MULTIPROCESSING="${AFD_FORCE_SPAWN_MULTIPROCESSING:-1}" +export VLLM_ENGINE_READY_TIMEOUT_S="${VLLM_ENGINE_READY_TIMEOUT_S:-3600}" +export PYTORCH_NPU_ALLOC_CONF="${PYTORCH_NPU_ALLOC_CONF:-expandable_segments:True}" +export OMP_PROC_BIND="${OMP_PROC_BIND:-false}" +export OMP_NUM_THREADS="${OMP_NUM_THREADS:-10}" +export HCCL_IF_IP="$LOCAL_IP" +export GLOO_SOCKET_IFNAME="$NIC_NAME" +export TP_SOCKET_IFNAME="$NIC_NAME" +export HCCL_SOCKET_IFNAME="$NIC_NAME" +export HCCL_OP_EXPANSION_MODE="${HCCL_OP_EXPANSION_MODE:-AIV}" +# The recorded DP4TP2/EP8 run required 4096 MB for the mbt=65536 CAM domain. +# Keep the process-wide fallback aligned with the connector-scoped value. +export HCCL_BUFFSIZE="${HCCL_BUFFSIZE:-$HCCL_BUFFER_SIZE_MB}" +# FlashComm1/SP is Attention-local. FFN TP1 rejects it. +export VLLM_ASCEND_ENABLE_FLASHCOMM1="$FLASHCOMM1" +export AFD_FORCE_BALANCED_TOPK_IDS=0 + +ADDITIONAL_CONFIG="$( + printf '%s' "{ + \"enable_force_load_balance\": false, + \"multistream_dsv4_dsa_overlap\": false, + \"enable_dsv4_shared_compressor_workspace\": true, + \"afd\": { + \"role\": \"$ROLE\", + \"connector\": \"CAMAsyncAFDConnector\", + \"async\": true, + \"host\": \"$AFD_HOST\", + \"port\": $AFD_PORT, + \"num_attention_ranks\": 8, + \"num_ffn_ranks\": 8, + \"compute_gate_on_attention\": true, + \"connector_extra_config\": { + \"dynamicQuant\": 1, + \"attn_ranks_per_dp\": 2, + \"async_moe_ubatching\": true, + \"async_moe_num_ubatches\": 2, + \"async_moe_split\": \"token\", + \"hccl_buffer_size\": $HCCL_BUFFER_SIZE_MB + } + } + }" +)" + +exec env VLLM_USE_V1=1 vllm serve "$MODEL_PATH" \ + --host 0.0.0.0 \ + --port "$API_PORT" \ + --served-model-name deepseek_v4_flash \ + --worker-cls "$WORKER_CLASS" \ + "${PARALLEL_ARGS[@]}" \ + --enable-expert-parallel \ + --enforce-eager \ + --quantization ascend \ + --tokenizer-mode deepseek_v4 \ + --block-size 128 \ + --max-model-len "$MAX_MODEL_LEN" \ + --max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" \ + --max-num-seqs "$MAX_NUM_SEQS" \ + --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" \ + --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 16}' \ + --seed 1024 \ + --trust-remote-code \ + --no-enable-prefix-caching \ + --enable-chunked-prefill \ + --additional-config "$ADDITIONAL_CONFIG"