diff --git a/NOTICE b/NOTICE index b3d4b99c9..2b7cbead1 100644 --- a/NOTICE +++ b/NOTICE @@ -66,17 +66,41 @@ they are reproduced below: mlx (expm1f device routine) ---------------------------- +---------------------------- The fused xIELU Metal kernel in src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp reproduces the device-side -expm1f routine from mlx/backend/metal/kernels/expm1f.h in MLX -(https://github.com/ml-explore/mlx), Copyright (c) Apple Inc., licensed -under the MIT License. The JIT metal_kernel preamble does not pull that -header in and Metal's provides no expm1, so the routine is -inlined verbatim (renamed) to keep the fused activation byte-identical to -the elementwise path that calls mlx::core::expm1. The MIT License text in -the section above (Copyright © 2023 Apple Inc.) covers this reuse. +expm1f routine from MLX +(https://github.com/ml-explore/mlx/blob/main/mlx/backend/metal/kernels/expm1f.h), +Copyright (c) Apple Inc., licensed under the MIT License. The JIT metal_kernel +preamble does not pull that header in and Metal's provides no +expm1, so the routine is inlined verbatim (renamed) to keep the fused +activation byte-identical to the elementwise path that calls mlx::core::expm1. + +MLX is Copyright © 2023 Apple Inc. and licensed under the MIT License. The +copyright, permission, and warranty notice is reproduced below: + + MIT License + + Copyright © 2023 Apple Inc. + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. That routine is itself derived work carried in MLX under the following BSD 2-Clause license, reproduced verbatim as required: diff --git a/README.md b/README.md index a9bb1886e..b0ef72439 100644 --- a/README.md +++ b/README.md @@ -341,7 +341,7 @@ Apache License 2.0 unless otherwise noted. See [LICENSE](LICENSE). Third-party a ## Acknowledgments -- [MLX](https://github.com/ml-explore/mlx), Apple's machine learning framework. +- [MLX](https://github.com/ml-explore/mlx), Copyright © 2023 Apple Inc., licensed under the MIT License, provides the runtime and source adapted for mlxcel's fused xIELU `expm1f` routine. See [NOTICE](NOTICE). - [NripeshN/mlx](https://github.com/NripeshN/mlx), whose `rocm-support` branch is the AMD GPU (ROCm/HIP) MLX backend vendored as the ROCm part of mlxcelverse, under the MIT License. See [NOTICE](NOTICE). - [mlx-lm](https://github.com/ml-explore/mlx-lm), [mlx-vlm](https://github.com/Blaizzy/mlx-vlm), and [mlx-audio](https://github.com/Blaizzy/mlx-audio), whose model coverage and behavior mlxcel ports and mirrors. See [NOTICE](NOTICE). - [MLX Community](https://huggingface.co/mlx-community), pre-converted MLX checkpoints. diff --git a/TECHNICAL_REPORTS/pr-2138-metal-gather-qmm-rhs-rows.en.md b/TECHNICAL_REPORTS/pr-2138-metal-gather-qmm-rhs-rows.en.md new file mode 100644 index 000000000..9ccaecb3c --- /dev/null +++ b/TECHNICAL_REPORTS/pr-2138-metal-gather-qmm-rhs-rows.en.md @@ -0,0 +1,61 @@ +# PR #2138: Size Metal `gather_qmm_rhs` by the broadcast x rows + +**Date**: 2026-10-15 +**Status**: Integrated with main `9bfabbab` after #2304; the remaining delta is attribution and reports, with production source identical to validated main +**Risk**: Low (no-op for every production caller; changes behavior only for an x that `gather_qmm_rhs` has to broadcast) + +## Summary + +The Metal nightly on the M1 Ultra runner has failed since 2026-09-30 on `models::switch_layers::mxfp_tests::{mxfp4,mxfp8}_gather_qmm_matches_host_reference`, both in the `SortedSharedActivation` case. `GatherQMM::eval_gpu` in MLX's Metal backend passes `x.size() / K` to `gather_qmm_rhs` as the row count. That is the row count of x before `gather_qmm_rhs` broadcasts x against the indices. For one activation row gathered against 32 sorted slots it is 1, so the grid and the kernel's `M` bound cover one row and output rows 1..31 are never written. The overlay now passes `B * M`. The PR also makes two split-MTP tests portable across macOS path canonicalization and filesystems that reject invalid-byte names; the production resolver and its symlink protections are unchanged. + +Refs #1599. + +## 1. Nightly history + +| Dates | Failure | Status | +|---|---|---| +| 2026-09-28 | `verify-clippy`: unused `Result` in `fused_norm_parity_tests.rs:158` and `fused_rope_parity_tests.rs:133` | fixed by #2029 (d8d34e2b) | +| 2026-09-29 | `tests::family_order_is_exhaustive`: `FAMILY_ORDER` missing `Speech` | fixed by #2080 (5486e404) | +| 2026-09-30 to 10-11 | the two mxfp gather tests | fixed by this PR; focused Metal validation passed at `25202198` | +| 2026-10-12 (`1e8c7022`) | the same mxfp tests; split-MTP `/var` canonicalization and invalid-byte filename assumptions; three YAML `err_expect` lints | kernel and split-MTP tests fixed by this PR; YAML lint already fixed on current main | + +The mxfp tests arrived with #2071 (2cee9cf4) on 2026-09-30, so the last signature is a new test exposing a latent upstream edge case, not a regression. + +## 2. Root cause + +- The case uses x `[1, 1, K]`, indices `[32]` over 8 experts, `sorted = true`. With M == 1, B = 32, B / E = 4 and `right_sorted_`, `eval_gpu` takes `gather_qmm_rhs`. +- `gather_qmm_rhs` broadcasts x to 32 rows (`broadcast_with_indices`), but its `M` argument was computed by the caller from the original x: 1. `grid_dims.y = ceil(M / 16)` and the kernel's `tgp_bm = min(BM, M - y_row)` then write row 0 only. `out` comes from `allocator::malloc`, so the other rows hold whatever was there: 7.4e32 relative error (mxfp4) and non-finite values (mxfp8). +- The same runs pass the three earlier gather cases in bf16, f16 and f32, and `mxfp_matmuls_match_host_reference_on_cpu_device`, which runs `SortedSharedActivation` on MLX's CPU backend. The reference and the test are correct; "bf16" in the message is just the first dtype tried. +- The defect is mode-independent (affine would hit it too) and is still present on ml-explore/mlx main. + +## 3. Change + +`B * M` (the output row count) replaces `x.size() / K` at the one call site. It equals the old value whenever x is already expanded, which is every production caller (`SwitchLinear::forward(.., true)` always receives `gather_sort` output), and it also fixes the NAX variant, which receives the same argument. The overlay header and the CMake overlay notes list the new hunk. + +The split-MTP test for `.` and `..` outputs now compares the staging parent with the canonical temporary-directory path, matching the resolver's intentional behavior on macOS where `/var` resolves to `/private/var`. The non-UTF-8 test first probes the temporary filesystem: filesystems that accept the byte name retain the successful round trip, while rejecting filesystems must return the same I/O error and leave no output or staging entry. + +The normal merge of current main moved the Xcode 27 NAX compatibility file into `src/lib/mlxcelverse/overlays/metal/`. Because that overlay is a whole-file derivative of MLX, its preserved Apple header now has the required MLX source URL, MIT-license statement, and top-level `NOTICE` pointer. `NOTICE` covers the whole overlay and carries MLX's copyright and MIT terms, while the README acknowledgment identifies the licensed reuse. + +## 4. Validation + +- Historical exact-head Metal gate: Apple M1 Ultra (128 GB), macOS 27.0 (26A428), full Xcode 27 RC, PR head `25202198` on base `b34d517b`. All five `models::switch_layers::mxfp_tests` passed with the Metal overlay rebuilt, and the release build passed. +- Historical exact-head model smoke: Qwen3-30B-A3B-4bit, greedy 64 tokens, produced byte-identical text on base, repeated base, and PR; SHA-256 `d2f40b2d5831c3d7790d467fb51123f0d1f615fb7300aa2f92d86a9d8831a0de`. +- CUDA (GB10): the original PR branch built and all 32 `models::switch_layers` tests passed. This does not validate Metal. +- Current-main integration: main `59974283` was merged normally into the existing PR branch at `44045522`. The sole merge conflict was the NAX overlay's move from `src/lib/mlx-cpp` to `src/lib/mlxcelverse`; the quantized-kernel hunk, split-MTP tests, and overlay registration merged without semantic conflict. The five focused `models::switch_layers::mxfp_tests` pass under `--profile test-fast --features metal,accelerate`, and both focused split-MTP portability tests pass. Formatting, diff, license-header, MLX-pin parser, and cross-repository-reference checks pass. +- Fresh release real-model A/B: the compatibility baseline was built from `44045522` with only `quantized.cpp` restored from main `59974283`, excluding the `B * M` change while holding the Xcode 27 NAX compatibility overlay and all other source equal. It is explicitly not a pristine-main artifact. Its SHA-256 is `d76febfc9adb23ca6bc627310994c108cc4e721c26341ef7f49ab092a89db651`; the exact `44045522` candidate SHA-256 is `ee83df3db3cea7159579487718419d4329c7fbeb147655cb1847a6867b13a872`. +- Both release artifacts ran twice on Qwen3-30B-A3B-4bit with `MLXCEL_FUSED_MOE=0`, greedy decoding, no chat template, a 96-token repeated-word prompt, and 64 generated tokens. Qwen3 routes each token to 8 of 128 experts, so prefill covers about 768 expert-route slots per MoE layer, comfortably above the 64-slot boundary. All four generated texts are byte-identical: 962 bytes, SHA-256 `cbf1670d0b554a35c029ec4104ae3ecfc8a46e07296d7e33a5f3b2c214a5fa8f`. Timings varied and are not used for a performance claim. +- The exact-source candidate Metal nightly [38582194425](https://github.com/lablup/mlxcel/actions/runs/38582194425) passed at `44045522` on `lablup-macmini01`: formatting, Clippy, 12,316 general tests (390 ignored), and 51 video tests, with zero failures. +- Review found no remaining functional or security issue in the #1599 implementation. It identified a high-severity attribution gap in the whole-file MLX NAX overlay; the header, `NOTICE`, and README changes described above fix it, and the attribution patch passed follow-up review and the repository license-header check. +- PR CI's `cargo-deny` job exposed four rustls 0.23.45 advisories published on 2026-10-14. Exact main `59974283` locks the same version, so this is a base-wide prerequisite rather than a regression from #1599. It is not waived: issue #2312 and lock-only PR #2313 update rustls to 0.23.46 separately. + +## 5. Follow-up + +Report the call to ml-explore/mlx, and drop the hunk when a pin bump carries an upstream fix. Merge the separate rustls security prerequisite rather than suppressing its advisory gate. Keep #1599 open through merge and close it only after a green post-merge Metal nightly. + +## 6. Final integration after #2304 + +Main `9bfabbab` already carries this PR's `B * M` correction and portable split-MTP tests through #2304. Resolve the CMake and Metal quantized-file conflicts by retaining those main versions. Upstream MLX `243bdf09` includes the Xcode 27 correction from ml-explore/mlx#4594, so remove the obsolete whole-file NAX overlay and its registration instead of restoring the old `59d600b5e` source. Remove only the obsolete overlay-specific NOTICE wording; retain the expanded MIT terms and the expm1f attribution. + +The resulting production tree is identical to main. Its exact pre-squash source `4befdbc7` passed Metal nightly [38583586205](https://github.com/lablup/mlxcel/actions/runs/38583586205): formatting, Clippy, 12,315 workspace tests (390 ignored), and 51 FFmpeg video tests. The earlier release-model evidence above remains attributed to its original source and is not a fresh run on the new pin. #2304 also already upgraded rustls to 0.23.46, resolving the security prerequisite described above. + +Keep #1599 open until a post-merge main Metal nightly succeeds. M5-specific execution remains unverified. diff --git a/TECHNICAL_REPORTS/pr-2138-metal-gather-qmm-rhs-rows.ko.md b/TECHNICAL_REPORTS/pr-2138-metal-gather-qmm-rhs-rows.ko.md new file mode 100644 index 000000000..6bbd10c47 --- /dev/null +++ b/TECHNICAL_REPORTS/pr-2138-metal-gather-qmm-rhs-rows.ko.md @@ -0,0 +1,61 @@ +# PR #2138: Metal `gather_qmm_rhs`의 행 수를 broadcast된 x 기준으로 지정 + +**작성일**: 2026-10-15 +**상태**: #2304 이후 main `9bfabbab`과 통합. 남은 변경은 출처 표시와 보고서이며 프로덕션 소스는 검증된 main과 동일 +**위험도**: 낮음 (모든 프로덕션 호출에서 no-op이며, `gather_qmm_rhs`가 x를 broadcast해야 하는 경우에만 동작이 바뀜) + +## 요약 + +M1 Ultra 러너의 Metal nightly는 2026-09-30부터 `models::switch_layers::mxfp_tests::{mxfp4,mxfp8}_gather_qmm_matches_host_reference`의 `SortedSharedActivation` 케이스에서 실패했다. MLX Metal 백엔드의 `GatherQMM::eval_gpu`는 `gather_qmm_rhs`에 행 수로 `x.size() / K`를 넘긴다. 이 값은 `gather_qmm_rhs`가 x를 indices에 맞춰 broadcast하기 전의 행 수다. 활성 행 하나를 정렬된 슬롯 32개에 gather하면 이 값은 1이 되어, grid와 커널의 `M` 경계가 한 행만 덮고 출력의 1..31행은 기록되지 않는다. 이 PR은 overlay에서 `B * M`을 넘긴다. 또한 macOS 경로 정규화와 유효하지 않은 바이트 이름을 거부하는 파일시스템에서 split-MTP 테스트 두 개가 이식성 있게 동작하도록 고친다. 프로덕션 resolver와 심볼릭 링크 보호는 변경하지 않는다. + +Refs #1599. + +## 1. Nightly 이력 + +| 날짜 | 실패 | 상태 | +|---|---|---| +| 2026-09-28 | `verify-clippy`: `fused_norm_parity_tests.rs:158`, `fused_rope_parity_tests.rs:133`의 미사용 `Result` | #2029 (d8d34e2b)에서 수정 | +| 2026-09-29 | `tests::family_order_is_exhaustive`: `FAMILY_ORDER`에 `Speech` 누락 | #2080 (5486e404)에서 수정 | +| 2026-09-30 ~ 10-11 | mxfp gather 테스트 두 개 | 이 PR에서 수정; `25202198`에서 집중 Metal 검증 통과 | +| 2026-10-12 (`1e8c7022`) | 같은 mxfp 테스트, split-MTP의 `/var` 정규화 및 유효하지 않은 바이트 파일명 가정, YAML `err_expect` lint 세 개 | 커널과 split-MTP 테스트는 이 PR에서 수정; YAML lint는 현재 main에서 이미 수정됨 | + +mxfp 테스트는 2026-09-30 #2071 (2cee9cf4)과 함께 추가되었으므로, 마지막 실패는 회귀가 아니라 새 테스트가 upstream의 잠재된 경계 사례를 드러낸 것이다. + +## 2. 근본 원인 + +- 케이스 입력은 x `[1, 1, K]`, expert 8개에 대한 indices `[32]`, `sorted = true`다. M == 1, B = 32, B / E = 4, `right_sorted_`이므로 `eval_gpu`는 `gather_qmm_rhs`로 간다. +- `gather_qmm_rhs`는 x를 32행으로 broadcast하지만(`broadcast_with_indices`), `M` 인자는 호출자가 원래 x로 계산한 1이다. `grid_dims.y = ceil(M / 16)`과 커널의 `tgp_bm = min(BM, M - y_row)` 때문에 0행만 기록된다. `out`은 `allocator::malloc`에서 오므로 나머지 행에는 이전 메모리 값이 남는다: mxfp4는 상대 오차 7.4e32, mxfp8은 non-finite. +- 같은 실행에서 앞선 gather 케이스 세 개는 bf16, f16, f32 모두 통과했고, MLX CPU 백엔드에서 `SortedSharedActivation`을 돌리는 `mxfp_matmuls_match_host_reference_on_cpu_device`도 통과했다. reference와 테스트는 올바르며, 메시지의 "bf16"은 처음 시도한 dtype일 뿐이다. +- 결함은 모드와 무관하고(affine도 동일), ml-explore/mlx main에도 그대로 있다. + +## 3. 변경 + +유일한 호출 지점에서 `x.size() / K`를 `B * M`(출력 행 수)으로 바꿨다. x가 이미 확장된 경우 기존 값과 같으며, 모든 프로덕션 호출(`SwitchLinear::forward(.., true)`는 항상 `gather_sort` 출력을 받음)이 이에 해당한다. 같은 인자를 받는 NAX 변형도 함께 고쳐진다. overlay 헤더와 CMake overlay 주석에 새 hunk를 기록했다. + +`.` 및 `..` 출력의 split-MTP 테스트는 staging 부모를 정규화된 임시 디렉터리 경로와 비교한다. 이는 macOS에서 `/var`가 `/private/var`로 해석되는 production resolver의 의도와 일치한다. 비 UTF-8 테스트는 임시 파일시스템의 바이트 이름 지원을 먼저 검사한다. 지원하는 파일시스템에서는 성공 round trip을 그대로 검증하고, 거부하는 파일시스템에서는 같은 I/O 오류가 반환되며 출력이나 staging 항목이 남지 않아야 한다. + +현재 main을 일반 merge하면서 Xcode 27 NAX 호환 파일을 `src/lib/mlxcelverse/overlays/metal/`로 옮겼다. 이 overlay는 MLX 전체 파일의 파생물이므로 기존 Apple 헤더를 유지하면서 MLX 원본 URL, MIT 라이선스 문구, 최상위 `NOTICE` 포인터를 추가했다. `NOTICE`는 전체 overlay를 범위에 포함하고 MLX 저작권과 MIT 조건을 싣고 있으며, README acknowledgment도 라이선스에 따른 재사용임을 밝힌다. + +## 4. 검증 + +- 이전 정확한 head의 Metal gate: Apple M1 Ultra (128 GB), macOS 27.0 (26A428), full Xcode 27 RC, base `b34d517b` 위 PR head `25202198`. Metal overlay를 다시 빌드한 뒤 `models::switch_layers::mxfp_tests` 다섯 개가 모두 통과했고 release 빌드도 통과했다. +- 이전 정확한 head의 모델 스모크: Qwen3-30B-A3B-4bit, greedy 64 tokens에서 base, base 반복, PR이 byte-identical text를 생성했다. SHA-256은 `d2f40b2d5831c3d7790d467fb51123f0d1f615fb7300aa2f92d86a9d8831a0de`다. +- CUDA (GB10): 기존 PR 브랜치가 빌드됐고 `models::switch_layers` 테스트 32개가 모두 통과했다. 이는 Metal 검증이 아니다. +- 현재 main 통합: main `59974283`을 기존 PR 브랜치에 일반 merge하여 `44045522`를 만들었다. 유일한 충돌은 NAX overlay 위치가 `src/lib/mlx-cpp`에서 `src/lib/mlxcelverse`로 옮겨진 것이며, quantized-kernel hunk, split-MTP 테스트, overlay 등록은 의미 충돌 없이 병합됐다. `--profile test-fast --features metal,accelerate`에서 집중 `models::switch_layers::mxfp_tests` 5개가 통과했고 split-MTP 이식성 집중 테스트 2개도 통과했다. formatting, diff, license-header, MLX-pin parser, cross-repository-reference 검사도 통과했다. +- 새 release 실제 모델 A/B: 호환 baseline은 `44045522`에서 `quantized.cpp`만 main `59974283` 버전으로 되돌려 빌드했다. 따라서 `B * M` 변경은 제외하되 Xcode 27 NAX 호환 overlay와 나머지 소스는 동일하다. 이는 pristine main artifact가 아니다. SHA-256은 `d76febfc9adb23ca6bc627310994c108cc4e721c26341ef7f49ab092a89db651`이며, 정확한 `44045522` 후보의 SHA-256은 `ee83df3db3cea7159579487718419d4329c7fbeb147655cb1847a6867b13a872`다. +- 두 release artifact를 각각 두 번씩 Qwen3-30B-A3B-4bit에서 `MLXCEL_FUSED_MOE=0`, greedy decoding, chat template 비활성화, 96-token 반복 단어 prompt, 64개 생성 token으로 실행했다. Qwen3는 token마다 128개 expert 중 8개로 route하므로 prefill은 MoE layer마다 약 768 expert-route slot을 지나 64-slot 경계를 충분히 넘는다. 네 실행의 생성 text는 모두 byte-identical이며 962 bytes, SHA-256 `cbf1670d0b554a35c029ec4104ae3ecfc8a46e07296d7e33a5f3b2c214a5fa8f`다. Timing은 달랐으며 성능 주장에 사용하지 않는다. +- 정확한 소스 후보 Metal nightly [38582194425](https://github.com/lablup/mlxcel/actions/runs/38582194425)는 `lablup-macmini01`의 `44045522`에서 통과했다. Formatting, Clippy, 일반 테스트 12,316개(390개 ignored), video 테스트 51개가 실패 없이 통과했다. +- 리뷰에서 #1599 구현의 남은 기능·보안 문제는 발견되지 않았다. 다만 MLX NAX 전체 파일 overlay의 attribution 누락이 high-severity로 발견됐다. 위에서 설명한 헤더, `NOTICE`, README 변경으로 이를 수정했으며 attribution patch는 후속 리뷰와 저장소 license-header 검사를 통과했다. +- PR CI의 `cargo-deny` job은 2026-10-14 공개된 rustls 0.23.45 권고 네 개를 발견했다. 정확한 main `59974283`도 같은 버전을 잠그므로 #1599가 만든 회귀가 아니라 base-wide 선행 조건이다. 이를 waive하지 않으며 issue #2312와 lock-only PR #2313에서 rustls 0.23.46으로 별도 갱신한다. + +## 5. 후속 작업 + +ml-explore/mlx에 이 호출을 보고하고, upstream 수정이 포함된 pin bump 때 이 hunk를 제거한다. rustls 권고 gate를 억제하지 말고 별도 보안 선행 PR을 merge한다. #1599는 merge 후에도 열어 두고 post-merge Metal nightly가 통과한 뒤에만 닫는다. + +## 6. #2304 이후 최종 통합 + +Main `9bfabbab`에는 #2304를 통해 이 PR의 `B * M` 수정과 split-MTP 이식성 테스트가 이미 포함됐다. CMake와 Metal quantized 파일의 충돌은 해당 main 버전을 유지해 해결했다. Upstream MLX `243bdf09`에는 ml-explore/mlx#4594의 Xcode 27 수정이 있으므로, 오래된 `59d600b5e` 전체 파일을 복원하지 않고 중복 NAX overlay와 등록을 제거했다. NOTICE에서는 제거된 overlay 설명만 삭제하고 확장된 MIT 조건과 expm1f 출처 표시는 유지했다. + +결과 프로덕션 트리는 main과 동일하다. 정확한 squash 이전 소스 `4befdbc7`의 Metal nightly [38583586205](https://github.com/lablup/mlxcel/actions/runs/38583586205)는 formatting, Clippy, workspace 테스트 12,315개(390개 ignored), FFmpeg video 테스트 51개를 통과했다. 위 release 모델 검증은 원래 소스에 대한 증거이며 새 pin에서 재실행한 결과가 아니다. #2304가 rustls도 0.23.46으로 올렸으므로 위 보안 선행 조건은 해결됐다. + +#1599는 merge 후 main Metal nightly가 통과할 때까지 열어 둔다. M5 전용 실행은 아직 검증하지 못했다.