Merged
Conversation
Rebased onto latest test. setup.py: kept test's merged Ascend build machinery and extended it to compile mixed .asc + C++ pybind sources (npu_module.cpp from this PR) per-source before linking. Signed-off-by: zhangj1an <jianmusings@gmail.com>
feat(ascend): add RoPE kernel
Sequential rebase onto latest test (which includes #378): registry.py and _C_npu.pyi union with the rope entries; deterministic_attention_ascend binding consolidated into npu_module.cpp (single PYBIND11_MODULE) per the ws1 skill; recursive **/*.asc glob retained for the attention subdirectory. Signed-off-by: zhangj1an <jianmusings@gmail.com>
…attention-test [WS1][kernels] Deterministic attention Ascend C kernel
Sequential rebase onto latest test (includes #320): ops_npu.asc kept as the comment-only aggregator; prefix_shared_attention_ascend binding consolidated into npu_module.cpp; _C_npu.pyi and the ascend attention __init__ unioned with the deterministic-attention entries. Signed-off-by: zhangj1an <jianmusings@gmail.com>
…ntion feat(ascend): add prefix-shared attention Ascend C kernel
Sequential rebase onto latest test (includes #320, #340): ops_npu.asc kept as the comment-only aggregator; deterministic_collective_* bindings consolidated into npu_module.cpp; _C_npu.pyi unioned; collectives.py and setup.py auto-merged (CUDA fused paths kept, NPU staged flow added). Signed-off-by: zhangj1an <jianmusings@gmail.com>
…ective feat(ascend): add deterministic collective Ascend C kernel
Sequential rebase onto latest test (includes #320/#340/#355): rmsnorm binding consolidated into npu_module.cpp; _C_npu.pyi, registry npu priority map, ascend __init__, test_dispatch, check_operator.py unioned with the existing rope/attention/collective entries; setup.py mixed-build support already present from #378. Signed-off-by: zhangj1an <jianmusings@gmail.com>
feat(ascend): add batch-invariant RMSNorm Ascend C operator
…dding [WS1][Ascend] [Qwen3-8b] Embedding ops
[WS1][Ascend] [Qwen3-8b] Fused logp ops
Sequential rebase onto latest test (no conflicts). Signed-off-by: zhangj1an <jianmusings@gmail.com>
Re-resolved onto latest test (includes #370's logp entries): lm_head binding, registry enum + priority map, _C_npu.pyi, linear __init__, test_dispatch unioned with the existing ascend entries. Signed-off-by: zhangj1an <jianmusings@gmail.com>
[WS1][Ascend] [Qwen3-8b] LM head ops
…ar-logp [WS1][Ascend] [Qwen3-8b] Fused linear logp ops
feat(ascend): add SwiGLU forward and backward kernels
Port the WS1 deterministic GEMM (issue #146) to Ascend NPU, mirroring the CUDA det_gemm_kernel.cu contract: - Ascend C kernel (csrc/ascend/gemm/det_gemm_ascend.asc) with BF16 in / FP32 accumulation / BF16 out, no split-K, fixed ascending 32-element leaf order, and the CUDA mid-split BF16-add tree, so a contiguous half-K GEMM is one tree child and simulated TP=2 matches TP=1 bitwise. - Every output row-tile is reduced end-to-end by one AI-core block with a MAX_BLOCKS-capped strided launch -> batch-invariant numerics. - Six entry points mirror the CUDA surface 1:1: fwd, fwd_rhs_transposed, fwd_fp32, da, db, db_transposed (backward reuses the forward kernel on transposed operands, like CUDA). - DetGemmAscendOp (rl_engine/kernels/ops/ascend/matmul/det_gemm.py) with autograd forward/backward and native [N,K] linear support. - Registered in the kernel registry (npu det_gemm dispatch), gtest operator_specs ascend candidate, _C_npu.pyi stubs, docs, and tests/test_det_gemm_ascend.py (tree-reference correctness, batch and TP-shard bitwise invariance, backward correctness/layout contracts). Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Zhang Jian <jianmusings@gmail.com>
…kernel
bisheng rejects the three DataCopyPadExtParams<bfloat16_t> brace-inits that
pass a uint32_t byte count as the rightPadding field, which is uint8_t:
det_gemm_ascend.asc:174,189 LeafPadBytes(leafLen)
det_gemm_ascend.asc:204 colPad
error: non-constant-expression cannot be narrowed from type 'uint32_t'
to 'uint8_t' in initializer list [-Wc++11-narrowing]
Add explicit static_cast<uint8_t> at the three sites. Both values are
bounded by the fixed tile geometry -- LeafPadBytes in [0, 62] and colPad in
[0, 254], against a uint8_t ceiling of 255 -- so the cast is
semantics-preserving and cannot truncate.
Compile-only change: bisheng now builds det_gemm_ascend.asc and the
_C_npu extension links cleanly (verified with the documented
KERNEL_ALIGN_FORCE_ASCEND=1 pip install). It does NOT make the kernel
correct at runtime: on davinci0 all 24 tests in test_det_gemm_ascend.py
still fail with a vector core exception (507035), root cause not yet
identified.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: Zhang Jian <jianmusings@gmail.com>
The first on-device bring-up of det_gemm_ascend.asc exposed a cluster of
CANN 9.0.0 / Atlas A2 data-movement errors that killed every launch with
ACL_ERROR_RT_VECTOR_CORE_EXCEPTION (507035, "The write address of the MTE
instruction is out of range") and, once the kernel ran, corrupted the
kNK and transposed-output layouts:
- DataCopyPad stride semantics: srcStride/dstStride are the GAP after each
block (dstStride in 32 B units), not the block pitch. Passing the pitch
overran the UB window at block 25+ and raised the MTE fault. Both fast
paths now express the real gaps; contiguous tiles use 0/0.
- DataCopyPad padding: blockLen is the valid byte count excluding padding,
and left/rightPadding are element counts capped at 32 B of padding. Tail
leaves and partial tiles now pad only the final partial 32 B block, after
explicitly zero-filling the staging buffers each leaf.
- kNK loads landed the rhs tile as [N_TILE, TREE_LEAF] but the leaf sum
read it as [TREE_LEAF, N_TILE]; the per-k row is now gathered from the
column-major tile.
- kNK slow path used the global column index as the UB offset (tile 1+
wrote past the buffer); loads are now bounded by valid rows/columns.
- Cast fp32->fp32 with CAST_NONE emits no instruction on A2, leaving the
tree stack and FP32 outputs uninitialized; same-type copies now use
AscendC::Copy.
- Contiguous stores use one exact-length copy from an aligned base;
transposed stores expand each value into its own 32 B slot (MTE3 sources
must be 32 B aligned).
- MTE2 destinations moved to VECIN buffers and MTE3 sources to VECOUT
buffers per the DataCopyPad contract; dispatch pins the input device
before allocation.
The test reference splits the tree in leaf space now: the element-space
recursion produced sub-32-element leaves whenever K was not 32 * 2**j
(e.g. K=12288 -> 24-element leaves), a different tree than the kernel
evaluates, failing tolerance on large non-power-of-two reductions.
Validated on device: 34 pytest cases pass, including the regression sweep
R={1,17,33,65,96,32768} x N={129,257}, both rhs layouts, FP32 output,
batch and TP-shard bitwise invariance; fwd(A,B) equals
fwd_rhs_transposed(A,B.t()) and db_transposed equals db.t() bitwise, and
the device output matches an independent exact simulation of the leaf-space
tree (ascending FP32 leaf sums, BF16 RNE at every node) with maxdiff 0.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: Zhang Jian <jianmusings@gmail.com>
[WS1][kernels] Batch-invariant deterministic GEMM Ascend C kernel
…rofile Issue #266 is the WS1 acceptance entry (C1-C11 / #267-#277) for single-GPU model-level train-inference consistency on full Qwen3-8B Dense, with cuda_bf16 and triton_cuda_bf16 as its required profiles. This adds the Ascend version: ascend_bf16 (backend family "ascend") carried through every one of C1-C11 on the same shared contract and the same harnesses. ascend_bf16 is required, not optional: a missing or unexecuted Ascend cell is red, never N/A and never a fallback to another vendor's kernels. Kernel gaps closed first: - silu is a required C2 chain node with no Ascend kernel. Added to csrc/ascend/activation.asc next to SwiGLU, sharing its tile geometry and FP32 sigmoid sequence, so silu(x) is bitwise equal to swiglu(x, ones). Dispatching SwiGLU-with-a-unit-operand instead would report SwiGLU provenance, which C1 treats as an undeclared backend. - The canonical row-fold VJP needs a deterministic FP32-in GEMM; the Ascend det_gemm kernel is BF16-in only. det_gemm_rowwise_ascend_fwd_fp32 exposes the existing lm_head_ascend kernel (FP32 input, one fixed per-row reduction order) as a general GEMM via B^T - the same construction CUDA uses to build det_gemm_rowwise_fwd_fp32 from its SM90 lm_head kernel. Casting the VJP to BF16 would have broken the contract's FP32-accumulation rule. C1-C11: - C1 tolerance_contract.json declares ascend_bf16 -> family "ascend"; tolerance.py requires it. No Ascend-private tolerance relaxation. TF32 holds by construction (Ascend has no TF32 mode). - C2 ws1_manifest.json gains the profile with all 11 nodes declared and 23 representative cases mirroring the CUDA set, each pinning a real .asc entry point. version -> ws1-c2-v8, identity regenerated; workload_id is unchanged so existing CUDA/Triton evidence stays bound to the workload. - C3/C4 check_forward_invariance.py / check_gradient_invariance.py take --backend-profile ascend_bf16 and run on the profile's own device. - C5 elementwise_inventory gains an ascend_verdict column. - C6/C7 kv_consistency and its CLIs resolve the device from the profile. - C8 four_judgment_matrix covers the profile and can be scoped per host. - C9 qwen3_dense is device-agnostic; canonical backward paths gained Ascend branches recording family="ascend". - C10 chain_gate and ws1_chain_gate.py run the full #150 matrix on the NPU. - C11 ci/run_ws1_ascend_ci.sh plus .github/workflows/ws1-chain-npu.yml; ci/run_ws1_chain_gate.sh is parameterised through WS1_PROFILES. A host has a GPU or an NPU, not both, so the C8 sweep, candidate-evidence script and chain-gate CI script take an explicit profile list and each vendor's job proves its own profiles. C11 closes only when every required profile has gone green on its own hardware. rl_engine/kernels/gtest/accelerator.py holds the vendor-dependent facts the gates need and fails closed: no NPU means AcceleratorUnavailable, and pointing an Ascend profile at cuda:0 is rejected before any device probe. On-device evidence is not collected yet - it needs an Ascend host. Until then the Ascend C5 rows are tracked_red and the C8 Ascend cells are red, which is the correct pre-execution state. Includes PR #405 (Ascend deterministic GEMM), which this builds on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PzQmyerKmyyiMPSyPGspCH Signed-off-by: Zhang Jian <jianmusings@gmail.com>
On-device bring-up of the #266 closeout (feat/ws1-ascend-closeout) exposed two structural comparison mismatches and two missing canonical backward hooks; the C8 four-judgment sweep went from 10 red cells to 0 and the C2 candidate evidence from 21/23 to 23/23: - det_gemm accuracy gold: the deterministic GEMM rounds every 32-element leaf and every tree merge node to BF16, so comparing a candidate against the single-rounding torch.matmul gold fails structurally at near- cancellation outputs on random inputs (max_abs 2.0-4.0, reproducible in pure numpy on the manifest fixture data). The gold is now DetGemmTreeReferenceOp, the exact leaf-space mid-split tree. - det_gemm gradients: the autograd backward now uses the canonical FP32-accumulation rowwise VJP (det_gemm_rowwise_ascend_fwd_fp32) instead of the BF16 tree da/db kernels, so gradient_accuracy matches the unrounded FP32 reference grads to ULP. Determinism is preserved: the rowwise kernel reduces each output row in one fixed per-row order. - C4 gradient invariance: DetGemmAscendOp and RMSNormAscendOp now expose parameter_vjp_contributions_fp32 (the CUDA twin): per-row FP32 contributions that the harness accumulates in FP32 across call spans, so chunked / padded / permuted / singleton-aggregated weight gradients are bitwise identical (previously 1.5e-4 - 3.1e-4 drifts). - canonical embedding accepts the ascend family (the Ascend embedding's deterministic grad-weight reuses the CUDA construction bit-for-bit). - FP32-output attention: the model's FP32 composite attention edge had no Ascend path. The Ascend C kernel now accepts an outFp32 flag and emits the exact FP32 accumulator; DeterministicAttentionAscendOp.forward_fp32 exposes it (the twin of the CUDA op's forward_fp32). - tests/test_det_gemm_ascend.py backward reference updated to the FP32 matmul VJP (the gradient-accuracy gold semantics). Verified on device (Ascend 910B, CANN 9.0.0): C2 23/23, C8 88 green / 0 red / 8 N/A, C3/C4 invariance bitwise, det_gemm/attention operator suites green (34 + 26), and the C10 full-model gate now runs all eight cells with real backward; parity aggregates pass (max_abs_dlogp 1.7e-6). The C10 selected_logp config invariance still drifts on three pairs (BN/padded_left 0.12, B1-singleton/chunked 0.07, B1-singleton/full 1 ULP) - candidate-side model-wiring gaps tracked for follow-up. The contract's FP32-reference cell cannot run on 64 GB HBM (needs ~4.7 GiB more; the CUDA reference ran on an 80 GB H20). Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Zhang Jian <jianmusings@gmail.com>
The model feeds the selected-logp a non-contiguous slice (score_logits[:, :-1]), and FusedLogpAscendOp.apply silently fell back to the native torch log-softmax for non-contiguous inputs. The native path's per-row numerics depend on the batch layout, so the B1-singleton-aggregate cell's logp differed from the BN cell's by 1-2 fp32 ULP (19/27 tokens) and broke the C10 forward_invariance judgment (bitwise required). The wrapper now materializes the logits so the batch-invariant Ascend kernel runs for every NPU input; the B1-vs-BN selected_logp comparison is bitwise (0/27 diffs on the full-model gate cells). Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Zhang Jian <jianmusings@gmail.com>
… tiling) Left padding shifted the physical 64-key tile boundaries, and the softmax denominator (sumExp) is reduced per physical tile before the per-tile results are summed -- so the FP32 addition grouping of the valid keys depended on where the padding sat, and the FP32 composite edges amplified the 1-ULP-level difference across the 36 layers into a visible logp drift (BN/padded_left selected_logp max_abs 0.1197). The tiles are now anchored to the first valid key (keyBegin), so the valid keys always start at the first position of the first tile; the masked lanes are additionally zeroed after the Exp so they contribute exactly zero regardless of the vector Exp's behavior on the -FLT_MAX sentinel. Fully-masked batches fall through to the existing out=0 / lse=-inf path. Verified on device: the attention op's FP32 output and LSE are bitwise identical across left-pad lengths 0/1/63/64/65, and the BN/padded_left selected_logp is bitwise identical to BN/full (0/27 tokens, was 26/27). The B1-singleton/chunked cell still drifts: traced to a 1-bf16-ULP K/V divergence at layer 3 key 2 (the q_proj matches bitwise while k_proj / v_proj differ by one ULP) -- the chunked path's projection chain, not the attention kernel itself; tracked for follow-up. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Zhang Jian <jianmusings@gmail.com>
torch mean/sum select shape-dependent reduction kernels on NPU and flip single-ULP results between batch layouts (verified: 52/140 rows flip between [1,7,H] and [1,20,H] on the same data). The canonical and native RMSNorm forwards computed the rstd with the torch mean, so the chunked path's [1,chunk,H] slices and the full path's [1,20,H] batch produced ULP-different rstd values; the difference entered at layer 3, amplified through the FP32 composite edges, and reached 0.07 at the selected logp (the B1-singleton/chunked C10 invariance failure). The native reference, the Ascend op, and the canonical path now share one shape_invariant_rstd helper: the sum of squares is reduced in FIXED 32-wide chunks first, so the intermediate shapes (and hence the reduction kernels) never depend on the batch layout, and the rstd is bitwise identical for every layout on every device. The chunked cell's internal stateful-prefill consistency check passes, and the BN/padded_left, B1-singleton/full, and B1-singleton/chunked selected_logp maps are all bitwise identical to BN/full (0/27 each; all three were non-zero before). tests/test_rms_norm.py's manual reference uses the shared helper (the implementation's formula changed; the test's independent formula mirrors it). The gtest acceptance checks are untouched. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Zhang Jian <jianmusings@gmail.com>
The forward rstd was made shape-invariant earlier, but the backward's dx dot product s = sum(dy * w * x, dim=-1) still used the plain torch sum, whose reduction kernel is selected by shape on NPU and flips single-ULP results between batch layouts. The resulting dx differences propagated to every upstream parameter gradient: the B1-singleton/chunked cell differed from B1/full on 694/2394 weight gradients (max_abs 0.031). The ordinary and canonical backward now share _rms_norm_backward_rows, whose hidden-dim reduction is an explicit adjacent-pair FP32 tree (_fixed_row_sum) — elementwise adds whose pairing depends only on the hidden dimension, never on the row count. The ordinary backward reduces the dweight rows with the shared reduce_rows_fp32; the canonical backward keeps its session fold over the logical rows unchanged. Adds tests/test_ascend_rmsnorm_backward_partition.py: CPU/NPU partition regressions covering the fixed-pair sum (incl. odd widths and bf16 inputs), a float64-autograd backward oracle, and chunk-boundary independence for the canonical embedding + norm gradients. Verified on device: 20 new tests pass and the chunked cell's weight gradients are now bitwise identical to B1/full (0/399, was 694/2394 differing). The gtest checks and references are untouched. Signed-off-by: Zhang Jian <jianmusings@gmail.com> Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Zhang Jian <jianmusings@gmail.com>
…action The attention's VJP (the fp32 reference backward) ran the torch softmax/matmul on the padded layout, where left/right padding shifts the valid values inside the reduction trees and flips ULPs in dq/dk/dv (up to 7.0 between pad sides for identical logical tokens); the differences then propagated to every upstream parameter gradient (BN/padded_left: 390/2394 weight gradients differing, max_abs 0.031). The backward now compacts the valid tokens into the logical order before the VJP and scatters the gradients back, so the reductions are padding-invariant and the padded positions receive zero grads. Verified on device: the op-level dq/dk/dv are bitwise identical between the pad sides (0.0), the BN/padded_left weight gradients are bitwise identical to BN/full (0/399, was 390 differing), and the attention / rmsnorm / partition suites pass (137 passed, 32 skipped). The gtest checks and references are untouched. Signed-off-by: Zhang Jian <jianmusings@gmail.com> Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Zhang Jian <jianmusings@gmail.com>
…nvariance Replace the torch-compaction VJP with a dedicated Ascend C backward (deterministic_attention_backward_ascend.asc): three stream-ordered launches (rows/dV/dK) that recompute P/dS and the dq/dk/dv VJP with fixed keyBegin-anchored logical reduction orders. Gradients are now bitwise invariant to the batch layout and to where padding sits across physical lengths, closing the remaining PR #406 follow-up (valid Sv=63/64 with pad=5 previously showed 5e-4/2.4e-4 residuals; now 0.0 bitwise for Sv=63/64/65/96/128 on both pad sides). FP16 (an optional contract row) falls back to the torch VJP path, since the kernel is bf16-only. test_backward_grads now resolves the contract's gradient_accuracy/attention row and compares candidate grads against the FP32-kept reference VJP per the WS1 precision standard. Verified on device: 26/26 attention tests; boundary padding invariance bitwise; the full C10 gate reports gradient_invariance 2394/2394 with max_abs 0.0 and forward invariance 7/7 bitwise. Signed-off-by: Zhang Jian <jianmusings@gmail.com> Co-Authored-By: Claude Code <noreply@anthropic.com>
…B HBM The FP32-reference full-model backward OOMs the 64 GB HBM (~4.7 GiB short), so the C10 reference cell runs in BF16 on this host (documented in the PR description): the gold topology resolves family='pytorch' with a plain matmul and the reference model builds in bfloat16. The accuracy judgment must be re-assessed with the official FP32 reference on a larger-memory device. Signed-off-by: Zhang Jian <jianmusings@gmail.com> Co-Authored-By: Claude Code <noreply@anthropic.com>
The FP32-reference full-model backward OOMs the 64 GB HBM with resident weights (~4.7 GiB short). The reference cell now keeps the FP32 weights CPU-resident (Qwen3DenseWeightsOffloaded) and pages each weight onto the NPU per access: autograd holds each copy only until its VJP consumes it, and the FP32 gradients accumulate on the CPU leaves, so the peak HBM is ~36 GB instead of ~70 GB. The paging copies are exact, so the reference numerics are bitwise identical to the resident-FP32 model (verified on device: layer-0/1 forward and all gradients bitwise, offloaded vs resident). The gate now measures the accuracy judgments against the official FP32 reference on this host: selected_logp max_abs 0.0857 (atol 0.06) and 44/798 gradient rows fail at near-zero reference-gradient elements (max_abs 0.11-1.88), a candidate-side accuracy gap against the contract tree reference (CUDA H20: 798/798 at 0.1034). Invariance and parity are unaffected. Signed-off-by: Zhang Jian <jianmusings@gmail.com> Co-Authored-By: Claude Code <noreply@anthropic.com>
[WS1][Ascend] Qwen3-8B Dense train-inference closeout for BF16 execution vs FP32 reference
[skill] add ws1 ascend kernel
Flink-ddd
requested review from
KJLdefeated,
bitborne,
inaniloquentee and
maxiaosong1124
as code owners
September 17, 2026 08:53
|
Important Review skippedToo many files! This PR contains 103 files, which is 3 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (103)
You can disable this status message by setting the Comment |
inaniloquentee
approved these changes
Sep 17, 2026
maxiaosong1124
approved these changes
Sep 17, 2026
Flink-ddd
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.