Repository navigation
update: advance MLX pin and reconcile backend ownership - #2304
Merged
Merged
Conversation
Advance MLX to 243bdf09c0cccbb85ca6a686dab3cc0102cf1f7e, rebase changed CPU/CUDA/Metal overlays against pristine upstream sources, retire the CUDA teardown overlay superseded by upstream #4514, and retain the gathered-row correctness fix while using upstream's Xcode 27 workaround. Retarget the ROCm core overlay and adapt move-only allocation ownership, custom releases and primitive APIs without changing the fork revision. Add host allocation lifetime contracts and preserve reviewed macOS filesystem test portability fixes for candidate CI. Document every retained overlay and hardware validation boundaries. Validation: 27 MLX pin/parser tests, version/dtype/dispatch/llama contracts, 123 Python tooling tests and formatting pass. Native CUDA compilation and candidate Metal execution are coordinated separately; ROCm GPU validation requires an AMD host. Refs #2303
inureyes
added
status:review
The rebased upstream gathered GEMM route called make_cta_tiler with two arguments, while the retained mlxcel selector needs the active CUDA device for its Blackwell tile cap. Pass encoder.device() as the existing quantized GEMM route does, preserving the row count, quantization group size and runtime tile overrides. The missing argument caused the first candidate CUDA build to fail and the later CuTe layout errors were cascading diagnostics. Validation: inspected both selector call sites and checked the diff. Native CUDA compilation is being rerun on the corrected source. Refs #2303
Add 384-wide sorted-prefill reference cases for MXFP4 and MXFP8 with float16 and bfloat16 activations. The existing 320-wide cases fail the new CUDA gathered RHS kernel's output-alignment admission, so they cannot validate its JIT path or the Blackwell 128-row tile policy. Keep those tail cases and reuse the existing expanded routing fixture, host reference and tolerances. Correct the CUDA ops overlay's stale provenance comment. Formatting and diff checks pass; candidate hardware execution follows in the coordinated validation run. Refs #2303
Upstream #4554 adds cp_async_wait<0>() and a CTA barrier immediately before the quantized GEMM epilogue overwrites the shared-memory A/B buffers. The former local #910 pair precedes only a register-fragment conversion, so upstream now supplies the same async-copy completion and cross-thread ordering for both ordinary and gathered RHS kernels. Retire the whole device header overlay after verifying that removing its provenance and local barrier hunk reproduces pristine 243bdf09 byte for byte. Record the proof and updated overlay inventory; retain the existing JIT module revision boundary. Validation: pristine-object equality assertion and diff check pass. The coordinated CUDA run will restore the retired cached header from the pinned Git object before rebuilding. Refs #2303
inureyes
marked this pull request as ready for review
October 14, 2026 01:04
Merge main e642d56 into PR #2304 while preserving the retired CUDA overlays and moving the ROCm allocator header into the new overlay tree. Update the ownership test path, reconcile duplicate ROCm fix identifiers, and refresh overlay paths and counts without dropping main's ROCm or MoE changes. Validate the pin parsers, overlay provenance and ROCm residual, host ownership regression, contract gates, Python tooling, formatting, and CUDA compilation. Numerical execution follows the remaining main synchronization.
Integrate main 5997428 after the mlxcelverse conflict resolution and preserve its request-memory admission behavior. Record successful final CUDA compilation and 112 focused test executions, including MoE paths and direct RHS execution, alongside the prior source-specific hardware evidence.
The final PR CI found RUSTSEC-2026-0346 through RUSTSEC-2026-0349 in the existing rustls 0.23.45 lockfile entry. Update rustls alone to 0.23.46; cargo deny check passes without adding advisory exceptions.
Member
Author
|
Resolved the latest-main conflicts and reviewed the resulting overlay delta. The mlxcelverse migration, main's ROCm grid fixes and MoE changes are preserved; superseded CUDA overlays remain deleted. No unresolved serious correctness, security, or performance finding remains. The new rustls advisories are fixed with 0.23.46, without adding advisory exceptions. Validation:
ROCm hardware and M5-specific validation remain unavailable. Queued or running PR checks are not represented as passes. Actual Kimi inference evidence requires the independent #2296 fix and retains its source attribution in the PR body. |
inureyes
added a commit
that referenced
this pull request
Oct 14, 2026
Merge main after #2304 while retaining its gathered-row correction and portable tests. Remove the obsolete NAX compatibility overlay now supplied by upstream, and preserve MLX license terms with updated source attribution. The remaining diff from main is documentation only. Production source matches the successful final Metal nightly and CUDA library validation recorded in the reports. Refs #1599
inureyes
added a commit
that referenced
this pull request
Oct 14, 2026
Reconcile the Metal nightly fix with main `9bfabbab` after #2304. Main already contains the `gather_qmm_rhs` broadcast-row correction (`B * M`) and the two split-MTP portability fixes. Retain those implementations, and remove the obsolete Xcode 27 NAX overlay because pinned upstream MLX `243bdf09` supplies the correction from ml-explore/mlx#4594. The remaining diff from main contains only MLX attribution and bilingual technical reports. Preserve the full MIT terms and expm1f attribution while removing references to the deleted overlay. Rustls 0.23.46 is already included through #2304. Validation: - Production source is identical to main, whose exact pre-squash source `4befdbc7` passed [Metal nightly 38583586205](https://github.com/lablup/mlxcel/actions/runs/38583586205): fmt, Clippy, 12,315 workspace tests (390 ignored), and 51 FFmpeg video tests. - After reconciliation: six focused CUDA mxfp tests, 160 surgery tests, formatting, license-header and diff checks pass. - Historical M1 Ultra release A/B at `44045522`: a 96-token prompt and 64 greedy output tokens, two runs per arm with fused MoE disabled, produced identical outputs (`cbf1670d0b554a35c029ec4104ae3ecfc8a46e07296d7e33a5f3b2c214a5fa8f`). The baseline retained Xcode compatibility and reverted only the row-count fix. This is not a fresh model run on the new pin. Reports preserve the artifact hashes. Review found no remaining blocking implementation, security, or performance issue. M5 execution remains unverified. Keep #1599 open until the final post-merge main Metal nightly passes. Refs #1599
inureyes
added a commit
that referenced
this pull request
Oct 14, 2026
Preserve the English/Korean assessment and validation reports for the rustls 0.23.46 security update. The dependency fix already landed through #2304; after integrating current main, this PR changes only the reports, not product code or dependency resolution. The original lock-only patch was independently reviewed against all four RustSec advisories and the canonical crate checksum. The ring-only provider qualification remains explicit: no product-level post-quantum protection or loss is claimed. Validation of the original patch at `e9604cba`: cargo-deny, locked metadata/dependency tree, 10 TLS tests, and 71 downloader tests passed; one optional integration test was ignored. Tests used the supported `MLXCEL_BUILD_METAL=OFF` CPU configuration because that historical base had an unrelated Xcode 27 Metal compilation failure. Live external TLS integration was not exercised. These are not fresh whole-product tests on the newer MLX pin. The original WebUI performance check missed its threshold (2.075% degradation versus 2% limit); it is preserved as historical evidence, not relabeled as a pass. The final diff is documentation-only and does not modify or weaken that gate. Closes #2312
This was referenced Oct 14, 2026
inureyes
added a commit
that referenced
this pull request
Oct 15, 2026
…3bdf09 (#2317) ## Summary - Updates `src/lib/mlxcelverse/overlays/rocm/mlx/backend/rocm/fence.cpp` so `Fence::wait(Stream, const array&, uint32_t value)` waits on `value` (`fence->event.wait(value)`) and `uint32_t Fence::update(Stream, const array&, bool)` returns `fence->count`, matching upstream `mlx/fence.h` after the MLX `243bdf09` pin advance in #2304 (`9bfabbab`). - Records `mlx/backend/rocm/fence.cpp` in `src/lib/mlxcelverse/overlays/rocm/LOCAL_FIXES.md` item 48 (`Retarget compatibility at 243bdf09`). ## Test plan - `make verify-rocm-overlay`: `records OK: 109 backend + 16 core files, 48 LOCAL_FIXES entries, 5 core files with a noted residual, pin 243bdf09`, `rocm_overlay_test: OK (29 checks)`. - `make verify-rocm-held` on `gfx1151` (`AMD Radeon 8060S Graphics`): 12,384 passed, 0 failed, 411 ignored, `rocm_smoke.sh` OK (`241.69 tok/s`, 32 tokens). - `unset LD_LIBRARY_PATH && ./target/release/mlxcel-server -m models/mlx/Qwen3-0.6B-4bit`: starts cleanly on `gfx1151` via `DT_RPATH = [/opt/rocm/lib]` and serves `/health` and `/v1/chat/completions`. Refs #1801, #2303
2 of 3 tasks
inureyes
added a commit
that referenced
this pull request
Oct 15, 2026
…2318) ## Summary Classify the #1712 failure as far as the evidence allows, make the next recurrence self-classifying, and fix a coverage gap found on the way: the "steel envelope" test has not exercised the steel kernel since May. ## Findings **The test never reached the steel kernel.** `update_and_turbo4_delegated_attention` tries the dequant-first SDPA route first, and that route has been default-on since f77e98b (2026-05-08, two days after the test landed in 09d9cc6). Under the default env the wrapper returns before `attention_turbo4_delegated_steel`, and nothing else calls it. So the test, and the launcher CONTRIBUTING.md names for post-pin-bump re-validation, compared dequant-first SDPA against the cold-only composition. The `4.6824e-1` came from that pairing, not from `turbo4_delegated_steel_sdpa`. **The wrong value is "cold V contribution dropped".** I rebuilt step 0 (cold_offset=8, hot_offset=1, same seeds) on CUDA and measured candidate wrong outputs against the graph reference. Attention with the cold V rows zeroed sits at RMS 4.6820e-1, 4e-5 from the reported 4.6824e-1 (the two correct paths differ by about 1.4e-4). The next closest candidates are far off: K read at full buffer capacity 4.03e-1, all-zero output 5.12e-1, hot V dropped 5.15e-2, missing inverse rotation 7.85e-1. So on one side the cold contribution was all zeros: either the bulk rotated-dequant output (dequant-SDPA route), the cold weighted-sum output (cold-only route), or a zero `v_rescale`. A never-written fresh Metal buffer reads as zeros, which explains why the value repeated to five digits while the failure itself was rare. The value alone cannot tell which side or which of these. **It has not recurred.** Nightly verify since 2026-09-08: 48 runs. 44 ran the test and all 44 passed (0accedd through 537ebe1, four Mac minis). One run failed before the test step (38430123889) and one was cancelled. Six of the passes ran on MLX 243bdf09 (PR #2304 branch and main after the merge). Every nightly passes `--test-threads=1`, while both failures came from multi-threaded workspace gates. That fits the shared-device hazard of #163/#1008 but does not prove it. In this window the delegated kernel source changed only by #2029 (dispatch through `select_kernel_port`, kernel bodies untouched), the #2305 move, and the #1796 header update. A code read of the three launchers found no unwritten output region, out-of-bounds index, or missing barrier. ## Changes - New test-only `cache/delegated_parity_diag.rs`. On failure it reports the route gates, a comparison of the two caches' K, packed V, rescale and hot V, each observed output against the graph reference (with the max-abs location), the dropped-contribution signatures, and an in-process re-run of every route on both caches. Thresholds are unchanged. - Both steel tests now also call `attention_turbo4_delegated_steel` directly, and still check the wrapper's output, so the pairing that failed keeps its coverage. ## Test plan - [x] CUDA (GB10): new `cache::delegated_parity_diag_tests` (2 tests) pass and run the full report path. The step-0 candidate table above comes from a throwaway test on the same binary. - [x] The two macOS-gated tests compile on Linux with their gate lifted (clippy `-D warnings`, `--lib --tests`). Gate restored before commit. - [ ] Metal: Nightly verify dispatched on this branch, run [38777513435](https://github.com/lablup/mlxcel/actions/runs/38777513435), pending. The steel kernel has not been tested since May, so a red result here is a real finding, not noise. Refs #1712
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Advance MLX from
59d600b5to captured upstream main243bdf09(111 commits, 312 files). Reconcile whole-file overlays to retain mlxcel's numerical behavior and runtime controls without overwriting upstream fixes.B * Mcorrection; use upstream #4594 for Xcode 27.75915908. Adapt move-only allocation ownership, borrowed-storage donation safety, compiled callbacks, SDPA sinks, gated-delta fallback, floor division, and odd inverse FFT. Add actual-header host ownership regressions; these do not prove HIP completion ordering.Validation
3aeccc2b: fmt/Clippy pass; 12,285 tests pass, 390 ignored; 51 FFmpeg video tests pass.53473340= production3aeccc2b+ independent fix: restore Kimi Linear CUDA inference #2296 mask fix1fec677b; that model fix is not included here. CLI SHA-256:b52d70388e0420582c4f60f3341d570c2afd736685b37e2c0c6e50ed9eaf5fae.ROCm compilation/runtime and M5-specific execution remain unavailable. Ordinary PR CI is tracked separately; queued checks are not passes. The requested whole-batch CUDA/Clippy and fresh main Metal integration gate remains deferred until the expanded follow-ups are ready. This PR does not independently close #1599, #2158, or #1986. English/Korean technical reports record provenance and remaining limits.
Latest main
59974283is integrated, including the mlxcelverse layout migration. Resolve three rename/delete or relocation conflicts, update ownership-test paths and ROCm record numbering, and preserve main's ROCm/MoE improvements. Post-merge CUDA compilation, contract/ownership/provenance gates, Python tests, and 112 focused test executions pass, including directly forced RHS kernels. CUDA library suites on the merged production source pass 11,247 tests (170 ignored); full CUDA workspace/all-targets Clippy and CPU-only linking pass on final4befdbc7. The CUDA block-float archive audit and six quantization tests pass. Upgrade rustls from 0.23.45 to 0.23.46 to resolve RUSTSEC-2026-0346 through 0349; local and hosted cargo-deny pass. Final Metal CI at4befdbc7passes fmt, Clippy, 12,315 workspace tests (390 ignored), and 51 FFmpeg video tests. Earlier actual Kimi results retain their original source attribution.Closes #2303