vllm-project · ShwStone · Sep 4, 2026
@@ -9,7 +9,9 @@ This guide describes the supported deployment shape, configuration contract,
 rank mapping, data flow, startup requirements, and current limitations. The
 [DeepSeek-V3.2 recipe](../../recipe/npu/CAMAsyncAFDConnector/deepseek_v3_2/README.md)
 links the current v0.26 launch scripts and retains the historical v0.19
-multi-node commands and measurements for provenance.
+multi-node commands and measurements for provenance. The experimental
+[DeepSeek-V4-Flash recipe](../../recipe/npu/CAMAsyncAFDConnector/deepseek_v4_flash/README.md)
+documents a single-node DP4TP2 Attention + EP8 FFN FlashComm1 deployment.
 
 > [!WARNING]
 > The vLLM 0.26 CAM async port remains experimental. The linked PCP8 recipe and

@@ -16,7 +16,8 @@ recipe/
 │       └── deepseek_v2_lite/
 └── npu/
     ├── CAMAsyncAFDConnector/
-    │   └── deepseek_v3_2/
+    │   ├── deepseek_v3_2/
+    │   └── deepseek_v4_flash/
     └── CAMP2pAFDConnector/
         └── deepseek_v3_2/
 ```
@@ -36,6 +37,7 @@ Directory names follow these conventions:
 | GPU | `P2pNcclAFDConnector` | DeepSeek-V2-Lite | Decode | Validated | [Launch examples](gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md) |
 | Ascend NPU | `CAMP2pAFDConnector` | DeepSeek-V3.2 | Decode | Validated | [Synchronous decode](npu/CAMP2pAFDConnector/deepseek_v3_2/README.md) |
 | Ascend NPU | `CAMAsyncAFDConnector` | DeepSeek-V3.2 | Prefill / decode | Experimental v0.26 DP+TP/SP path; post-fix DP2TP8+EP16 token split reached `0.9522` strict match on the complete GSM8K evaluation; legacy PCP8 results are v0.19-only | [Async CAM](npu/CAMAsyncAFDConnector/deepseek_v3_2/README.md) |
+| Ascend NPU | `CAMAsyncAFDConnector` | DeepSeek-V4-Flash W8A8 | Prefill / decode | Experimental v0.26 single-node DP4TP2 Attention + EP8 FFN tool stack for issue #227 performance validation | [Async CAM](npu/CAMAsyncAFDConnector/deepseek_v4_flash/README.md) |
 
 Open the model-level README before running a recipe. It documents the required
 hardware and runtime baseline, topology, environment variables, launch order,

@@ -0,0 +1,195 @@
+# DeepSeek-V4-Flash Async CAM Recipe
+
+This recipe launches DeepSeek-V4-Flash W8A8 on one 16-NPU Ascend 910C node:
+
+- NPUs 0-7: Attention `DP4 x TP2`, FlashComm1 sequence parallelism enabled;
+- NPUs 8-15: FFN `DP8 x TP1 / EP8`, FlashComm1 disabled;
+- one 16-rank CAM world containing 8 Attention and 8 FFN ranks;
+- AFD-managed two-stage token split, dynamic CAM quantization, eager execution,
+  chunked prefill, and the vLLM async scheduler;
+- `max_num_batched_tokens=65536`, shared compressor workspace, and a 4096 MB
+  CAM HCCL buffer.
+
+The two roles share one node but run as separate vLLM processes. Only the
+Attention process owns the public inference endpoint.
+
+> [!WARNING]
+> DeepSeek-V4 support over `CAMAsyncAFDConnector` remains experimental. This is
+> the user-facing launch tool stack for the performance work tracked by
+> [AFD issue #227](https://github.com/vllm-project/afd-plugin/issues/227),
+> rewritten against the configuration contract on
+> `vllm-project/afd-plugin:main`. The performance numbers below are historical
+> evidence for the recorded validation cell; this recipe itself has not been
+> requalified on NPU hardware.
+
+For the connector contract and package installation instructions, read the
+[CAM Async Connector User Guide](../../../../docs/npu/CAM_ASYNC_CONNECTOR_USER_GUIDE.md).
+
+## Runtime and model
+
+The issue #227 performance cell used:
+
+- Ascend 910C, Python 3.12, and CANN 9.0.1;
+- vLLM commit `568afb3a1`;
+- vLLM-Ascend commit `80d8c194f`, the `2e01d4c5` head commit from the
+  still-unmerged
+  [vLLM-Ascend PR #15452](https://github.com/vllm-project/vllm-ascend/pull/15452),
+  and the follow-up slot-mapping fix `e19e14da`;
+- CAM 209.x operator packages documented in the connector guide;
+- the complete `DeepSeek-V4-Flash-w8a8-mtp` checkpoint.
+
+PR #15452 commit `2e01d4c5` provides the DeepSeek-V4 compressor-tail workspace
+reuse; `e19e14da` applies its slot-mapping correction on top. Check out or
+apply those exact commits in order. PR #15452 is still open, so do not assume a
+released vLLM-Ascend checkout already contains CWS support.
+
+The checkpoint's `config.json` must select `model_type=deepseek_v4` or the
+`DeepseekV4ForCausalLM` architecture. The checkpoint may contain MTP weights,
+but this recipe does not enable speculative decoding.
+
+Install the AFD plugin from the target `main` checkout and install the CAM
+operators before launching. Source the matching CANN environment in each
+terminal. Verify that the four CAM async operators are registered:
+
+```bash
+python3 - <<'PY'
+import torch
+import umdk_cam_op_lib  # noqa: F401
+
+for name in (
+    "async_dispatch_send",
+    "async_dispatch_recv",
+    "async_combine_send",
+    "async_combine_recv",
+):
+    assert hasattr(torch.ops.umdk_cam_op_lib, name), name
+print("CAM async operators are available")
+PY
+```
+
+## Why these settings
+
+The issue #227 performance matrix established the operational contract used
+here:
+
+- FlashComm1 must be enabled only for Attention; FFN TP1 rejects it.
+- Attention `DP4TP2` plus FFN `EP8` was the best measured single-node layout.
+- Token split outperformed request split under sustained load in that layout.
+- `max_num_batched_tokens=65536` requires the DeepSeek-V4 shared compressor
+  workspace from vLLM-Ascend `e19e14da7`.
+- The `DP4TP2 / EP8`, MBT 65536 run failed with a smaller HCCL buffer and was
+  stable with 4096 MB.
+- Both roles must use identical CAM rank counts, split settings, rendezvous
+  address, and `max_num_batched_tokens`.
+
+Do not add vLLM native DBO flags. `async_moe_ubatching` is the separate,
+AFD-managed MoE pipeline. The recipe also deliberately leaves the experimental
+`prefill_token_sum` Attention DPLB policy unset; the issue #227 measurements
+found no throughput benefit over the request-count policy for the measured
+workload, and that policy is not part of the target `main` configuration
+contract.
+
+## Launch
+
+Run both commands on the same 16-NPU node. `LOCAL_IP` must be the address on
+`NIC_NAME`, and the same reachable address is used as the CAM rendezvous host.
+The launcher defaults to Attention devices 0-7 and FFN devices 8-15.
+
+Start FFN in terminal 1:
+
+```bash
+source /usr/local/Ascend/cann-9.0.1/set_env.sh
+
+MODEL_PATH=/path/to/DeepSeek-V4-Flash-w8a8-mtp \
+LOCAL_IP=<node-ip> \
+NIC_NAME=<npu-nic> \
+bash recipe/npu/CAMAsyncAFDConnector/deepseek_v4_flash/launch.sh ffn
+```
+
+Start Attention in terminal 2:
+
+```bash
+source /usr/local/Ascend/cann-9.0.1/set_env.sh
+
+MODEL_PATH=/path/to/DeepSeek-V4-Flash-w8a8-mtp \
+LOCAL_IP=<node-ip> \
+NIC_NAME=<npu-nic> \
+bash recipe/npu/CAMAsyncAFDConnector/deepseek_v4_flash/launch.sh attention
+```
+
+Wait for the Attention endpoint:
+
+```bash
+until curl --fail --silent http://127.0.0.1:8000/v1/models >/dev/null; do
+  sleep 5
+done
+```
+
+Then send a smoke request:
+
+```bash
+curl http://127.0.0.1:8000/v1/completions \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "deepseek_v4_flash",
+    "prompt": [1, 2, 3, 4, 5, 6, 7, 8],
+    "max_tokens": 1
+  }'
+```
+
+## Overrides
+
+The launcher accepts these optional environment variables:
+
+| Variable | Default | Meaning |
+| --- | --- | --- |
+| `AFD_HOST` | `LOCAL_IP` | CAM rendezvous host; must identify Attention rank 0. |
+| `AFD_PORT` | `1239` | CAM rendezvous port. |
+| `ATTENTION_PORT` / `FFN_PORT` | `8000` / `8001` | Role-local API ports. |
+| `ATTENTION_DEVICES` / `FFN_DEVICES` | `0-7` / `8-15` | Comma-separated device lists. |
+| `MAX_MODEL_LEN` | `70000` | Maximum sequence length. |
+| `MAX_NUM_BATCHED_TOKENS` | `65536` | Scheduler MBT; it must match on both roles. |
+| `MAX_NUM_SEQS` | `128` | Maximum scheduled sequences. |
+| `GPU_MEMORY_UTILIZATION` | `0.8` | Per-role device memory utilization. |
+| `HCCL_BUFFER_SIZE_MB` | `4096` | Connector-scoped CAM HCCL buffer size. |
+| `CAM_VENDOR_PATH` | CANN 9.0.1 CAM vendor path | Installed CAM operator root. |
+
+Keep the topology and model-semantic settings unchanged unless the new
+combination is validated independently.
+
+## Issue #227 performance summary
+
+The matching single-node `DP4TP2 / EP8`, token-split, MBT 65536 matrix used a
+512-request long-prefill workload. Every rate completed 512/512 requests with
+no failures:
+
+| Offered rate | Effective input tokens/s | TTFT p99 | Peak 15s service rate |
+| --- | ---: | ---: | ---: |
+| 0.5x | 17,075 | 9.0s | 30K tokens/s |
+| 0.75x | 25,230 | 10.7s | 41K tokens/s |
+| 1.0x | 32,830 | 15.6s | 45K tokens/s |
+| 1.25x | 37,321 | 25.6s | 47K tokens/s |
+| 1.5x | 40,096 | 34.3s | 47K tokens/s |
+
+Against the best baseline point at each offered rate, the recorded AFD cell
+improved effective throughput by 4.8% at 1.0x, 6.6% at 1.25x, and 8.6% at
+1.5x. At 1.0x, TTFT p50 improved by 37.4%, TTFT p99 by 22.1%, and the fraction
+of requests meeting a 10-second TTFT SLO increased from 65.2% to 85.2%.
+
+These values describe the exact issue #227 software, checkpoint, and workload
+cell. They are not a portable performance guarantee or a validation result for
+a different checkout.
+
+## Limitations
+
+Current constraints include:
+
+- eager execution only; ACL graph mode is unsupported;
+- FlashComm1 only on Attention and only with TP greater than one;
+- no context parallelism, prefix caching, speculative decoding, KV transfer,
+  or native DBO;
+- exactly two AFD MoE stages;
+- W8A8 CAM dynamic quantization is required for DeepSeek-V4-Flash;
+- all ranks need the same CAM/CANN packages, connector settings, and model;
+- teardown must remove both vLLM process trees before relaunching, or stale
+  workers can retain NPU memory and CAM port 1239.
@@ -0,0 +1,158 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project
+
+set -euo pipefail
+
+usage() {
+  echo "Usage: MODEL_PATH=<path> LOCAL_IP=<ip> NIC_NAME=<nic> $0 <attention|ffn>" >&2
+  echo "Launch each role in a separate terminal." >&2
+}
+
+if [[ $# -eq 1 && ( "$1" == "-h" || "$1" == "--help" ) ]]; then
+  usage
+  exit 0
+fi
+if [[ $# -ne 1 ]]; then
+  usage
+  exit 2
+fi
+ROLE="$1"
+
+: "${MODEL_PATH:?Set MODEL_PATH to the DeepSeek-V4-Flash W8A8 checkpoint}"
+: "${LOCAL_IP:?Set LOCAL_IP to this node communication IP}"
+: "${NIC_NAME:?Set NIC_NAME to the NPU network interface}"
+
+AFD_HOST="${AFD_HOST:-$LOCAL_IP}"
+AFD_PORT="${AFD_PORT:-1239}"
+ATTENTION_PORT="${ATTENTION_PORT:-8000}"
+FFN_PORT="${FFN_PORT:-8001}"
+MAX_MODEL_LEN="${MAX_MODEL_LEN:-70000}"
+MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-65536}"
+MAX_NUM_SEQS="${MAX_NUM_SEQS:-128}"
+GPU_MEMORY_UTILIZATION="${GPU_MEMORY_UTILIZATION:-0.8}"
+HCCL_BUFFER_SIZE_MB="${HCCL_BUFFER_SIZE_MB:-4096}"
+ATTENTION_DEVICES="${ATTENTION_DEVICES:-0,1,2,3,4,5,6,7}"
+FFN_DEVICES="${FFN_DEVICES:-8,9,10,11,12,13,14,15}"
+CAM_VENDOR_PATH="${CAM_VENDOR_PATH:-/usr/local/Ascend/cann-9.0.1/opp/vendors/CAM}"
+
+if [[ ! -f "$MODEL_PATH/config.json" ]]; then
+  echo "DeepSeek-V4 config not found: $MODEL_PATH/config.json" >&2
+  exit 1
+fi
+
+case "$ROLE" in
+  attention)
+    API_PORT="$ATTENTION_PORT"
+    VISIBLE_DEVICES="$ATTENTION_DEVICES"
+    FLASHCOMM1=1
+    WORKER_CLASS=afd_plugin.v1.worker.npu.AFDNPUAttentionWorker
+    PARALLEL_ARGS=(
+      --data-parallel-size 4
+      --tensor-parallel-size 2
+    )
+    ;;
+  ffn)
+    API_PORT="$FFN_PORT"
+    VISIBLE_DEVICES="$FFN_DEVICES"
+    FLASHCOMM1=0
+    WORKER_CLASS=afd_plugin.v1.worker.npu.AFDNPUFFNWorker
+    PARALLEL_ARGS=(
+      --data-parallel-size 8
+      --tensor-parallel-size 1
+    )
+    ;;
+  *)
+    echo "ROLE must be attention or ffn, got: $ROLE" >&2
+    exit 2
+    ;;
+esac
+
+CAM_OPAPI_DIR="$CAM_VENDOR_PATH/op_api/lib"
+if [[ -f "$CAM_OPAPI_DIR/libopapi.so" ]]; then
+  CAM_OPAPI="$CAM_OPAPI_DIR/libopapi.so"
+  CAM_OPAPI_LOAD_PATH="$CAM_OPAPI_DIR"
+elif [[ -f "$CAM_OPAPI_DIR/libcust_opapi.so" ]]; then
+  # CAM 209.x may ship only libcust_opapi.so, while UMDK resolves the vendor
+  # implementation by the libopapi.so name. Provide a process-local alias.
+  CAM_OPAPI_COMPAT_DIR="${CAM_OPAPI_COMPAT_DIR:-${TMPDIR:-/tmp}/afd-cam-opapi}"
+  mkdir -p "$CAM_OPAPI_COMPAT_DIR"
+  ln -sfn "$CAM_OPAPI_DIR/libcust_opapi.so" "$CAM_OPAPI_COMPAT_DIR/libopapi.so"
+  CAM_OPAPI="$CAM_OPAPI_COMPAT_DIR/libopapi.so"
+  CAM_OPAPI_LOAD_PATH="$CAM_OPAPI_COMPAT_DIR:$CAM_OPAPI_DIR"
+else
+  echo "CAM op-api library not found under $CAM_OPAPI_DIR" >&2
+  exit 1
+fi
+
+export ASCEND_RT_VISIBLE_DEVICES="$VISIBLE_DEVICES"
+export ASCEND_CUSTOM_OPP_PATH="$CAM_VENDOR_PATH:${ASCEND_CUSTOM_OPP_PATH:-}"
+export LD_LIBRARY_PATH="$CAM_OPAPI_LOAD_PATH:$CAM_VENDOR_PATH/op_api:${LD_LIBRARY_PATH:-}"
+export CAM_CUST_OPAPI_LIB_PATH="$CAM_OPAPI"
+export LD_PRELOAD="$CAM_OPAPI${LD_PRELOAD:+:$LD_PRELOAD}"
+export VLLM_PLUGINS="${VLLM_PLUGINS:-ascend,afd}"
+export VLLM_WORKER_MULTIPROC_METHOD="${VLLM_WORKER_MULTIPROC_METHOD:-spawn}"
+export AFD_FORCE_SPAWN_MULTIPROCESSING="${AFD_FORCE_SPAWN_MULTIPROCESSING:-1}"
+export VLLM_ENGINE_READY_TIMEOUT_S="${VLLM_ENGINE_READY_TIMEOUT_S:-3600}"
+export PYTORCH_NPU_ALLOC_CONF="${PYTORCH_NPU_ALLOC_CONF:-expandable_segments:True}"
+export OMP_PROC_BIND="${OMP_PROC_BIND:-false}"
+export OMP_NUM_THREADS="${OMP_NUM_THREADS:-10}"
+export HCCL_IF_IP="$LOCAL_IP"
+export GLOO_SOCKET_IFNAME="$NIC_NAME"
+export TP_SOCKET_IFNAME="$NIC_NAME"
+export HCCL_SOCKET_IFNAME="$NIC_NAME"
+export HCCL_OP_EXPANSION_MODE="${HCCL_OP_EXPANSION_MODE:-AIV}"
+# The recorded DP4TP2/EP8 run required 4096 MB for the mbt=65536 CAM domain.
+# Keep the process-wide fallback aligned with the connector-scoped value.
+export HCCL_BUFFSIZE="${HCCL_BUFFSIZE:-$HCCL_BUFFER_SIZE_MB}"
+# FlashComm1/SP is Attention-local. FFN TP1 rejects it.
+export VLLM_ASCEND_ENABLE_FLASHCOMM1="$FLASHCOMM1"
+export AFD_FORCE_BALANCED_TOPK_IDS=0
+
+ADDITIONAL_CONFIG="$(
+  printf '%s' "{
+    \"enable_force_load_balance\": false,
+    \"multistream_dsv4_dsa_overlap\": false,
+    \"enable_dsv4_shared_compressor_workspace\": true,
+    \"afd\": {
+      \"role\": \"$ROLE\",
+      \"connector\": \"CAMAsyncAFDConnector\",
+      \"async\": true,
+      \"host\": \"$AFD_HOST\",
+      \"port\": $AFD_PORT,
+      \"num_attention_ranks\": 8,
+      \"num_ffn_ranks\": 8,
+      \"compute_gate_on_attention\": true,
+      \"connector_extra_config\": {
+        \"dynamicQuant\": 1,
+        \"attn_ranks_per_dp\": 2,
+        \"async_moe_ubatching\": true,
+        \"async_moe_num_ubatches\": 2,
+        \"async_moe_split\": \"token\",
+        \"hccl_buffer_size\": $HCCL_BUFFER_SIZE_MB
+      }
+    }
+  }"
+)"
+
+exec env VLLM_USE_V1=1 vllm serve "$MODEL_PATH" \
+  --host 0.0.0.0 \
+  --port "$API_PORT" \
+  --served-model-name deepseek_v4_flash \
+  --worker-cls "$WORKER_CLASS" \
+  "${PARALLEL_ARGS[@]}" \
+  --enable-expert-parallel \
+  --enforce-eager \
+  --quantization ascend \
+  --tokenizer-mode deepseek_v4 \
+  --block-size 128 \
+  --max-model-len "$MAX_MODEL_LEN" \
+  --max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" \
+  --max-num-seqs "$MAX_NUM_SEQS" \
+  --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" \
+  --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 16}' \
+  --seed 1024 \
+  --trust-remote-code \
+  --no-enable-prefix-caching \
+  --enable-chunked-prefill \
+  --additional-config "$ADDITIONAL_CONFIG"