Skip to content

feat(npu): run dsv4-flash w8a8 8p perf case standalone in single-node e2e workflow - #1236

Open
pllimax wants to merge 13 commits into
Ascend:testcases-260813from
pllimax:dsv4-flash-single-node-e2e
Open

pllimax wants to merge 13 commits into
Ascend:testcases-260813from
pllimax:dsv4-flash-single-node-e2e

Conversation

@pllimax

@pllimax pllimax commented Sep 10, 2026

Copy link
Copy Markdown

Purpose

从 PR #1222 中抽出 DeepSeek-V4-Flash W8A8 8P 性能用例(test_npu_deepseek_v4_flash_w8a8_8p_in8k_out1k_50ms.py)单独执行所需的最小改动集合,形成独立 PR。

Changes

  1. 新增 .github/workflows/nightly-test-npu-e2e-single-node.yml\:单用例单机执行的 workflow 模板(取自 PR Patch 3 #1222),通过 \workflow_call\ 接收 runner / test_type / test_config_name / test_case 等输入。
  2. 修改 .github/workflows/full-test-npu.yml\:
    • 启用 \pull_request\ 触发器,paths 过滤 \ est/registered/npu/performance/deepseek_v4_flash/**\ 及相关 workflow 文件;
    • 新增 ightly-poc-single-node-tests\ job(框架拉起服务方式),matrix 中为 dsv4-flash w8a8 8p 性能用例,runner 为 \linux-aarch64-a3-800t-16\;
    • 将该 job 加入 \check-all-jobs\ 的 needs 列表。
  3. 修改 \ est/registered/npu/performance/deepseek_v4_flash/test_npu_deepseek_v4_flash_w8a8_8p_in8k_out1k_50ms.py\(取自 PR Patch 3 #1222,保持其 diff 一致):
    • CI 注册调整:移除 \�ase-c-test-perf-16-npu-a3\ 套件,\est_time\ 1200 -> 1800;
    • 环境变量调整:新增 \SGLANG_ENABLE_WAR_BARRIER\ / \SGLANG_FORCE_COARSE_WAR_BARRIER\ / \SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK\ / \LD_DEBUG\;
    • 服务启动参数调整:\mem-fraction-static\ 0.6 -> 0.68、\chunked-prefill-size\ -1 -> 131072 等;
    • 压测参数调整:新增固定 \dataset_path\、\warmup_requests\ 0 -> 16、\max_attempts=3\、\output_token_throughput\ 1708 -> 2825;
    • 新增 \SGLANG_EXTERNAL_SERVER_URL\ 外部服务模式开关(不设置时保持原有框架拉起行为,完全向后兼容)。

Test

  • \python -m py_compile\ 通过;两个 workflow YAML 解析通过。
  • 提交后由 paths 过滤自动触发 ightly-poc-single-node-tests\(16 卡 a3 runner)验证。

CI States

Latest PR Test (Base): ❌ Run #34800191333
Latest PR Test (Extra): ❌ Run #34800191163
Latest PR Test (AMD ROCm 7.2): ❌ Run #34800191317

… e2e workflow

- Add .github/workflows/nightly-test-npu-e2e-single-node.yml template for running a single registered NPU test case on one node

- Register nightly-poc-single-node-tests job in full-test-npu.yml for test_npu_deepseek_v4_flash_w8a8_8p_in8k_out1k_50ms and enable PR trigger with paths filter

- Update the dsv4-flash w8a8 8p perf case: tune envs/server args/benchmark params and support SGLANG_EXTERNAL_SERVER_URL external-server mode
@ascend-robot

Copy link
Copy Markdown

CLA Signature Pass

pllimax, thanks for your pull request. All authors of the commits have signed the CLA. 👍

@ascend-robot

Copy link
Copy Markdown

CLA Signature Guide

@pllimax , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit Reason
c9db6f03 Update test_npu_deepseek_v4_flas... the email used in the commit is not linked to a signed CLA!
please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

…t trigger

PR Ascend#1236 targets testcases-260813 but the trigger only matched main, so the nightly-poc-single-node-tests job never ran for this PR.
@ascend-robot

Copy link
Copy Markdown

CLA Signature Guide

@pllimax , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit Reason
c9db6f03 Update test_npu_deepseek_v4_flas... the email used in the commit is not linked to a signed CLA!
please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

@pllimax pllimax closed this Sep 10, 2026
@pllimax pllimax reopened this Sep 10, 2026
@ascend-robot

Copy link
Copy Markdown

CLA Signature Guide

@pllimax , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit Reason
c9db6f03 Update test_npu_deepseek_v4_flas... the email used in the commit is not linked to a signed CLA!
please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

1 similar comment
@ascend-robot

Copy link
Copy Markdown

CLA Signature Guide

@pllimax , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit Reason
c9db6f03 Update test_npu_deepseek_v4_flas... the email used in the commit is not linked to a signed CLA!
please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

@ascend-robot

Copy link
Copy Markdown

CLA Signature Guide

@pllimax , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit Reason
c9db6f03 Update test_npu_deepseek_v4_flas... the email used in the commit is not linked to a signed CLA!
please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

- Remove DEEPSEEK_V4_FLASH_0731_W8A8_MODEL_PATH import from
  test_npu_performance_utils (not defined there, caused ImportError in
  CI run 34506648448); define it locally in the test file instead.
- Route LD_DEBUG output via LD_DEBUG_OUTPUT to standalone files instead
  of flooding stderr, and archive them into the per-case metrics
  directory (ld_debug.tar.gz) in tearDownClass before the process tree
  is killed.
@ascend-robot

Copy link
Copy Markdown

CLA Signature Guide

@pllimax , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit Reason
c9db6f03 Update test_npu_deepseek_v4_flas... the email used in the commit is not linked to a signed CLA!
please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

The per-case tearDownClass archives LD_DEBUG output into
<metrics_data_file>/ld_debug.tar.gz on the runner, but the Upload
metrics step only pushes /tmp/metrics.json. Add an upload step so the
so-loading logs are retrievable from the Actions artifacts.
@ascend-robot

Copy link
Copy Markdown

CLA Signature Guide

@pllimax , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit Reason
c9db6f03 Update test_npu_deepseek_v4_flas... the email used in the commit is not linked to a signed CLA!
please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

… e2e workflow

- grep -c prints 0 and exits 1 on no match, so `|| echo 0` produced a
  two-line value and broke `[ ... -gt 0 ]`; use a default expansion.
- The workflow-level `export LD_DEBUG=libs` had no LD_DEBUG_OUTPUT, so
  the pytest driver process dumped its whole dlopen/fini trace into the
  job log. Redirect it to /tmp/wf_ld_debug and upload it as artifact.
@ascend-robot

Copy link
Copy Markdown

CLA Signature Guide

@pllimax , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit Reason
c9db6f03 Update test_npu_deepseek_v4_flas... the email used in the commit is not linked to a signed CLA!
please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

…f popen_launch_server

Bypass the framework's "sglang serve" entry and its offline/cache
pre-checks by launching the server directly with
"python -m sglang.launch_server" in setUpClass, keeping the same
envs/other_args injection. Wait loop detects early process exit instead
of polling a dead server until timeout, and server stdout/stderr are
captured to server.log under the per-case output dir for debugging.
External-server mode (SGLANG_EXTERNAL_SERVER_URL) is unchanged.
@ascend-robot

Copy link
Copy Markdown

CLA Signature Guide

@pllimax , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit Reason
c9db6f03 Update test_npu_deepseek_v4_flas... the email used in the commit is not linked to a signed CLA!
please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

- sglang.launch_server takes the model via the --model-path option, not
  as a positional argument; the positional form made argparse exit with
  code 2 within seconds. Match the format used by popen_launch_server.
- Add /root/.cache/tests/output/**/server.log to the ld-debug artifact
  upload so early-exit reasons are visible without runner shell access.
@ascend-robot

Copy link
Copy Markdown

CLA Signature Guide

@pllimax , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit Reason
c9db6f03 Update test_npu_deepseek_v4_flas... the email used in the commit is not linked to a signed CLA!
please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

- Flip install_sglang_from_source to true so the workflow uses the PR
  branch runtime instead of the image's stale sglang package, which
  lacked the preallocated NPU swa_indices fix (SWA mask device error
  during verify graph capture).
- Drop the case-level LD_DEBUG env and its tearDownClass archival hook:
  debugging served its purpose, and the workflow-level wf_ld_debug
  redirection already covers the pytest parent process.
- Drop the server.log artifact upload path added earlier; the server
  early-exit message points to the on-runner path, which is enough.
@ascend-robot

Copy link
Copy Markdown

CLA Signature Guide

@pllimax , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit Reason
c9db6f03 Update test_npu_deepseek_v4_flas... the email used in the commit is not linked to a signed CLA!
please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

Server died during startup under source-install mode and the job log
has no stage output, so the artifact is needed to diagnose the exit.
@ascend-robot

Copy link
Copy Markdown

CLA Signature Guide

@pllimax , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit Reason
c9db6f03 Update test_npu_deepseek_v4_flas... the email used in the commit is not linked to a signed CLA!
please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants