Skip to content

Support d != dv test, add varlen test.#14

Open
baoqiwen wants to merge 1 commit into
umiswing:masterfrom
baoqiwen:bqw_test
Open

Support d != dv test, add varlen test.#14
baoqiwen wants to merge 1 commit into
umiswing:masterfrom
baoqiwen:bqw_test

Conversation

@baoqiwen
Copy link
Copy Markdown
Contributor

@baoqiwen baoqiwen commented Apr 13, 2026

功能支持

  1. 测试时支持 d != dv。主要是 d=192, dv=128 的场景。
  2. 新增 fa4 varlen torch 的性能测试代码 -- benchmark_fa4_varlen_torch.py
    • 用 fa4 官方代码测:
      • from flash_attn.cute.interface import _flash_attn_fwd, _flash_attn_bwd
    • 用迁移到 fm 仓库的 fa4 代码测:
      • from flash_mask.flash_attn_v4.torch.interface import _flash_attn_fwd, _flash_attn_bwd
  3. 新增 fa4 varlen paddle 性能测试代码 -- benchmark_fa4_varlen_paddle.py
  4. 新增 fa4 varlen paddle 精度测代码 -- test_flashmask_varlen.py
    • from flash_mask import flashmask_attention

tips

  1. benchmark_fa4_mask_mod.py

    • d=192,dv=128时,fa4 不支持 mask_mod is not None 的场景。因此,只测 Full、Causal。
    • 如果有 triton 内置 ptxas 不认识 sm103 相关的错,记得 pip3 install triton==3.6.0。
  2. fa4 官方并未合入 d=256 dv=256 代码。

@baoqiwen baoqiwen force-pushed the bqw_test branch 10 times, most recently from 1ad39a9 to 56dd476 Compare April 16, 2026 10:31
@baoqiwen baoqiwen changed the title Support d=192 dv=128 test Support d != dv test, add varlen test. Apr 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant