Build a CUDA bundle for Windows on ARM - #218
Conversation
Windows ARM64 hosts with an NVIDIA GPU are the only CUDA hosts we publish nothing for: the installer looks for a windows-arm64-cuda bundle, finds none, and falls back to ggml-org's single llama-bin-win-cuda-13.4-arm64.zip. Add the leg that produces it. It cross compiles ggml-cuda on an x64 runner with the amd64_arm64 MSVC toolset and CUDA 13.4 (the only Windows toolkit with ARM64 target libraries), then merges the backend into the arm64 CPU bundle the same run already builds. Same split upstream uses for its arm64 CUDA zip.
|
You have reached your Codex usage limits for security reviews. Please try again later. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ab879a8aa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - name: Upload bundle artifact | ||
| uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 | ||
| with: | ||
| name: app-${{ inputs.tag }}-windows-arm64-${{ env.PROFILE }} | ||
| path: dist/app-${{ inputs.tag }}-windows-arm64-${{ env.PROFILE }}.zip | ||
| if-no-files-found: error |
There was a problem hiding this comment.
Add app-${TAG}-windows-arm64-cuda13-portable.zip to the parent workflow’s “Verify full bundle coverage before publish” list. That gate explicitly handles cases where all child jobs and uploads are green but download-artifact omits an individual bundle; because this newly uploaded archive is not checked there, such an anomaly would publish a release and manifest without the Windows ARM64 CUDA bundle, silently restoring the fallback behavior this change is intended to eliminate.
Useful? React with 👍 / 👎.
| if re.fullmatch(r"cudart-llama-bin-win-cuda-\d+\.\d+-(?:x64|arm64)\.zip", name): | ||
| wanted.append((name, "windows-cuda-upstream")) |
There was a problem hiding this comment.
For normal scheduled mix releases, where the PR set is nonempty, is_upstream_release is false and the entire block containing this widened regex is skipped. The new local Windows ARM64 bundle is still advertised, but package_bundle.py deliberately excludes its CUDA runtime, so the release’s SHA index contains no approved ARM64 cudart hash and the installer cannot verify the runtime it must pair with the bundle. Fetch and index the base tag’s ARM64 cudart asset for mix builds as well.
Useful? React with 👍 / 👎.
| build-windows-cuda-arm64: | ||
| name: CUDA Windows ARM64 | ||
| needs: resolve | ||
| if: ${{ needs.resolve.outputs.exists != 'true' || github.event_name == 'workflow_dispatch' }} |
There was a problem hiding this comment.
On every manual dispatch this condition launches the fixed cuda13-portable ARM64 child regardless of only_profile. A dispatch intended to test another profile, such as cuda12-legacy, therefore also performs this unrelated Windows ARM64 compilation and can fail the whole subset run if that leg fails; the existing Linux and Windows CUDA matrices instead filter all rows by the requested profile. Gate this job to only_profile == 'all' or only_profile == 'cuda13-portable'.
Useful? React with 👍 / 👎.
1 participant
Windows on ARM is the only CUDA platform this repo publishes nothing for.
install_llama_prebuilt.pyalready looks for awindows-arm64-cudabundle first and only then falls back to ggml-org'sllama-bin-win-cuda-13.4-arm64.zip, so on a WoA machine with an NVIDIA GPU we currently hand the user upstream's build and give up three things we give every other CUDA host:assert-windows-bundle-signed.ps1; upstream's are unsigned.This adds the leg that produces it.
Design
There is no ARM64 Windows runner with a CUDA toolkit, so the leg cross compiles on an x64 runner with the
amd64_arm64MSVC toolset. That is the same approachunsloth-prebuilt-cpu.ymlalready uses for the Windows arm64 CPU bundle, and the same one ggml-org'srelease.ymluses for its arm64 CUDA zip.It builds only
ggml-cuda, againstcmake/arm64-windows-msvc-cuda.cmakewithGGML_CPU=OFF, then merges the resultingggml-cuda.dllinto the arm64 CPU bundle this same run already builds, and repacks throughpackage_bundle.py. That is upstream's arrangement too: their arm64 CUDA job packsggml-cuda.dllalone and the release merges it into the arm64 CPU zip.GGML_BACKEND_DL=ONon both sides is what makes the split legal, since ggml loads the backend through the registry, a C ABI, so the clang-built CPU tree and the MSVC-built backend never share a C++ interface.Toolkit comes from
packages.nvidia.com/bin-archive, not the redist CDN the x64 legs use: 13.4 is the first Windows toolkit shipping ARM64 target libraries and it is only in that pool. Host-side pieces (nvcc, crt, nvvm, cccl) stay x86_64 because the compiler runs on the x64 runner; onlycuda_cudartandlibcublasare the arm64 builds. Same component set ggml-org installs.Why a separate child workflow
Four things in this leg differ from every row of the x64 matrix: the toolkit source and version, the toolset, the single build target, and the packaging, which is a merge rather than a curate from a build tree. Folding those into
unsloth-prebuilt-cuda-windows.ymlwould mean conditionals on all four plus arch in every ccache key, with a one-time cache miss for the x64 legs. A separate child keeps the x64 path byte for byte unchanged.It also does not use the shared
unsloth-cuda-windows-setupcomposite. That action exists so the x64warmshards andbuildjob configure identically and share ccache entries; there is one job here, so there is nothing to keep in sync.The arm64 CPU bundle is a sibling job, not a
needs:. The parent's "Wait for the build matrix" step fails a publish if a child has no job record within 45 minutes of the run starting, and aneeds:would hold this job's record back until the whole CPU child finished. So the leg starts immediately, compiles, and only then polls for the artifact, which by that point has long since been uploaded.Why sm_120 and sm_121 only
Windows on ARM CUDA hosts are Blackwell only. NVIDIA's ARM64 Windows driver and its 13.4 toolkit target the GB10 class parts (sm_121) and discrete RTX 50 (sm_120). There is no Windows ARM64 machine with a Pascal, Turing, Ampere, Ada or Hopper GPU to cover, and building those archs would roughly double a cross-compiled leg's runtime to ship PTX nothing can load. Linux arm64 keeps 90 and 100 because Grace-Hopper and GB200 servers are real there.
Staging evidence
Proved the toolchain on a staging run before proposing it: https://github.com/danielhanchen/unsloth-staging-2/actions/runs/35100180000
Same toolkit install, same toolchain file, same configure line, same arch list. Results:
Configure 48 seconds,
ggml-cudabuild 26.7 minutes for both archs on a 4 vCPU runner with a cold cache, 77 MB artifact. The workflow re-checks the PE machine field itself and fails the job ifggml-cuda.dllis not an ARM64 image, so an x64 backend can never reach a bundle.Other changes
assemble_metadata.py:("windows", "arm64")added toKIND_BY_CUDAaswindows-arm64-cuda, which is exactly theinstall_kindthe installer asks for. The bundle filename regex already accepted windows plus arm64.-x64to(?:x64|arm64).package_bundle.pydeliberately leaves the CUDA runtime out of every bundle, so without this the arm64 bundle has no approved hash for the runtime it needs to pair with.package_bundle.py:libompadded to the Windows local-DLL prefixes. The arm64 CPU bundle is a clang build withGGML_OPENMP=ONand carrieslibomp140.aarch64.dll;ggml-cpu.dllimports it by name, so curating a bundle from that tree without it produces an archive where nothing loads. No effect on the x64 legs, which are MSVC builds with no libomp.build-windows-cuda-arm64job, added to the waiter'sPREFIXESand to thereclaimandalertneeds lists. It requestsactions: readfor the artifact poll, which a called workflow cannot grant itself.What this does not prove
The staging run built the backend and checked its machine type; it did not run it, because no CI runner has a Windows ARM64 machine with an NVIDIA GPU. The first real bundle needs to be exercised on actual hardware before anyone should trust the selector's choice of it. The merge and packaging path is also unexercised until this runs for the first time in a full release, since the staging probe stopped after the build.