Skip to content

[Bugfix][TE] Fix Ascend wildcard location probe fallback to host memory - #4056

Open
gygdh-001 wants to merge 1 commit into
kvcache-ai:mainfrom
gygdh-001:fix-ascend-wildcard-probe-fallback
Open

[Bugfix][TE] Fix Ascend wildcard location probe fallback to host memory#4056
gygdh-001 wants to merge 1 commit into
kvcache-ai:mainfrom
gygdh-001:fix-ascend-wildcard-probe-fallback

Conversation

@gygdh-001

@gygdh-001 gygdh-001 commented Sep 11, 2026

Copy link
Copy Markdown

Description

ResolveAscendMemType() previously used CHECK_ACL, treating an
aclrtPointerGetAttributes() failure as a hard error (returning -1). As a result, registering
host memory (hugepage / shm / malloc) with the wildcard location "*" failed on Ascend. CUDA
already falls back when the wildcard probe cannot identify the memory; this change aligns Ascend
by treating the probe failure as host memory (adxl::MEM_HOST). This matches the existing fix
for the heterogeneous RDMA transport (#1657).

Fixes #4054

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Reshard (mooncake-reshard)
  • Mooncake EP (mooncake-ep)
  • Mooncake PG (mooncake-pg)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • Common (mooncake-common)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

Manual verification on Ascend 910B (bare mooncake.engine TE path). The wildcard fallback fix
follows the same pattern already merged for the heterogeneous-RDMA transport (#1657).

Test results:

  • Unit tests pass
  • Integration tests pass (if applicable)
  • Manual testing done (describe below)

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh
  • I have run pre-commit on the files changed in this PR and all hooks pass
  • I have updated the documentation (if applicable)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (specify below)

ResolveAscendMemType() treated aclrtPointerGetAttributes() failure as a hard
error (CHECK_ACL returned -1), so registering host memory with the wildcard
location ("*") failed on Ascend. CUDA already falls back when the wildcard
probe cannot identify the memory; align Ascend by treating probe failure as
host memory (adxl::MEM_HOST).
@ascend-direct-dev

Copy link
Copy Markdown
Collaborator

Thanks for the report. We probed aclrtPointerGetAttributes() on Ascend 910 / CANN 9.2.0 for the host allocation paths mentioned here (hugepage / shm / malloc). The ACL call does not fail for these buffers, so the CHECK_ACL change is not needed.

Current ResolveAscendMemType() already covers this: a successful probe with location.type that is neither HOST nor DEVICE falls through to the existing else and is treated as host memory.

Allocation ACL ret location.type Current code path
malloc / new[] / posix_memalign 0 UNREGISTERED(2) else → HOST
anonymous mmap / THP 0 UNREGISTERED(2) else → HOST
shm_open + mmap / memfd 0 UNREGISTERED(2) else → HOST
MAP_HUGETLB 0 UNREGISTERED(2) else → HOST
aclrtMallocHost 0 HOST(0) HOST branch
aclrtMalloc (device) 0 DEVICE(1) DEVICE branch
nullptr 100000 CHECK_ACL fails

ACL is not CUDA here: CUDA may fail the probe on unregistered host pointers, but ACL returns ACL_MEM_LOCATION_TYPE_UNREGISTERED successfully. Swallowing ACL errors would also turn a real invalid pointer (nullptr) into host memory.

Please keep the existing else HOST fallback and drop this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Ascend wildcard location probe fails on host memory instead of falling back

2 participants