Skip to content

[Bugfix][TE] Fix Ascend wildcard location fallback and 64KB memory alignment - #4053

Closed
gygdh-001 wants to merge 1 commit into
kvcache-ai:mainfrom
gygdh-001:fix-ascend-wildcard-fallback-alignment
Closed

[Bugfix][TE] Fix Ascend wildcard location fallback and 64KB memory alignment#4053
gygdh-001 wants to merge 1 commit into
kvcache-ai:mainfrom
gygdh-001:fix-ascend-wildcard-fallback-alignment

Conversation

@gygdh-001

@gygdh-001 gygdh-001 commented Sep 11, 2026

Copy link
Copy Markdown

Description

Fix two defects that prevent host-memory registration on the Ascend transfer engine (see #4046):

  1. Wildcard location probe falls back to host memory (ascend_direct_transport.cpp)

    ResolveAscendMemType() previously used CHECK_ACL, treating an
    aclrtPointerGetAttributes() failure as a hard error (returning -1). As a result,
    registering host memory (hugepage / shm / malloc) with the wildcard location "*"
    failed on Ascend. CUDA already falls back when the wildcard probe cannot identify the
    memory; this change aligns Ascend by treating the probe failure as host memory
    (adxl::MEM_HOST). This matches the existing fix for the heterogeneous RDMA transport
    (fix: fix heterogeneous rdma transport error #1657).

  2. 64KB-aligned default allocator (transfer_engine_py.cpp)

    initMemoryAllocator()'s fallback path used plain malloc, whose address is not
    64KB page-aligned. Ascend SVM (_devmm_mem_remote_map) requires a 64KB-aligned
    src_va, so aclrtHostRegister failed with EINVAL. The fallback now uses
    posix_memalign(64 * 1024, ...).

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Reshard (mooncake-reshard)
  • Mooncake EP (mooncake-ep)
  • Mooncake PG (mooncake-pg)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • Common (mooncake-common)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

Manual verification on Ascend 910B (bare mooncake.engine TE path):

  • Cross-card transfer runs: rdma path reaches 7.2 GB/s (1MB block × batch16 × 10
    rounds, all ok).
  • With the alignment fix, the fallback allocator returns 64KB-aligned addresses and
    aclrtHostRegister no longer fails with EINVAL.
  • The wildcard fallback fix follows the same pattern already merged for the
    heterogeneous-RDMA transport (fix: fix heterogeneous rdma transport error #1657).

Test commands:

# Manual verification on Ascend 910B

Test results:

  • Unit tests pass
  • Integration tests pass (if applicable)
  • Manual testing done (describe below)

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh
  • I have run pre-commit on the files changed in this PR and all hooks pass
  • I have updated the documentation (if applicable)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (specify below)

ResolveAscendMemType() treated aclrtPointerGetAttributes() failure as a hard
error (CHECK_ACL returned -1), so registering host memory with the wildcard
location ("*") failed on Ascend. CUDA already falls back when the wildcard
probe cannot identify the memory; align Ascend by treating probe failure as
host memory (adxl::MEM_HOST).

initMemoryAllocator()'s fallback path used plain malloc(), whose address is
not 64KB page-aligned. Ascend SVM (_devmm_mem_remote_map) requires a
64KB-aligned src_va, so aclrtHostRegister failed with EINVAL. Use
posix_memalign(64KB) for the default allocator instead.
@gygdh-001

Copy link
Copy Markdown
Author

Superseded - split into #4056 (wildcard location probe fallback) and #4057 (64KB alignment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant