Skip to content

feat(deps): CUDA 13 torch on Linux aarch64, CPU elsewhere; deterministic CUDA kernels - #85

Merged
ajbarea merged 5 commits into
mainfrom
feat/gpu-on-linux-aarch64
Sep 26, 2026
Merged

ajbarea merged 5 commits into
mainfrom
feat/gpu-on-linux-aarch64

Conversation

@ajbarea

@ajbarea ajbarea commented Sep 12, 2026 •

Copy link
Copy Markdown
Owner

[tool.uv.sources] now routes torch and torchvision by platform marker: Linux aarch64 to the pytorch-cu130 index, every other platform (CI included) to pytorch-cpu. The commented cu128 opt-in it replaces had no aarch64 build. The lock resolves one version everywhere, torch 2.14.0 / torchvision 0.29.0.

set_seed calls torch.use_deterministic_algorithms(True) when CUDA is present, covered by a CUDA-only test. ROADMAP.md now scopes the replay claim to client training and tracks run-level replay: identical smoke runs ended at 0.979 and 0.721 because Flower's client sampling is unseeded over per-run random node IDs.

Verification:

  • x86 CPU: make lint, make test (18 passed, CUDA test skipped), make audit, make smoke exit 0.
  • TIGRIS GH200 (driver 610.43.02): CUDA 13 torch runs on the device (102 GB, bf16); deterministic mode gives bitwise-equal embedding + linear gradients across runs, with or without CUBLAS_WORKSPACE_CONFIG.
  • TIGRIS GH200, torch 2.14.0+cu130, at cfaf0d197 (this branch with main merged): make lint clean; pytest 43 passed, 0 skipped, including the CUDA-only deterministic test; GPU make smoke (client-resources-num-gpus=0.2) 2 rounds, 0 failures. GPU memory rose from 0 to 3,467 MiB during the rounds and utilization peaked at 49%; nvidia-smi lists no processes on these nodes, so memory is the evidence.
  • TIGRIS GH200 at ec1f87136: the CUDA test compares a float32 index_add_ over duplicate indices. With deterministic mode off, 5 runs gave 5 distinct results (max abs diff 3.7e-3); with it on, 5 identical. Full suite 43 passed, 0 skipped.

@ajbarea ajbarea added the enhancement New feature or request label Sep 12, 2026
@ajbarea ajbarea self-assigned this Sep 12, 2026
@ajbarea ajbarea added the enhancement New feature or request label Sep 12, 2026
@ajbarea ajbarea closed this Sep 16, 2026
@ajbarea ajbarea reopened this Sep 16, 2026
…tic CUDA kernels

Torch and torchvision resolve from the cu130 index on Linux aarch64 and the cpu index everywhere else, replacing the commented cu128 opt-in that had no aarch64 build. Both forks move to torch 2.14.0 / torchvision 0.29.0 so every platform runs the same version. set_seed disables nondeterministic CUDA kernels when a GPU is present.
…n-level replay

Identical smoke runs ended at 0.979 and 0.721: Flower's client sampling is unseeded over random node IDs, so set_seed only makes each client's local training replay.
@ajbarea
ajbarea force-pushed the feat/gpu-on-linux-aarch64 branch from 251097c to b8b3993 Compare September 22, 2026 00:12
@ajbarea

ajbarea commented Sep 22, 2026

Copy link
Copy Markdown
Owner Author

Rebased onto main to clear a uv.lock conflict. The branch was force-pushed, so a local copy needs git fetch && git reset --hard origin/feat/gpu-on-linux-aarch64 before any further work on it.

The conflict came from chore: refresh the dev toolchain (#97), which rewrote the lock for ruff/ty/pytest while this branch rewrites it for the torch index routing. pyproject.toml merged cleanly — the [tool.uv.sources] block and the dev-dependency floors do not overlap — so only the lock needed resolving, and a lock is regenerated rather than hand-merged.

Taking main's lock and re-resolving held torch at 2.13.0, because uv lock keeps an existing pin that still satisfies the constraint and torch>=2.12 does. An explicit --upgrade-package torch --upgrade-package torchvision restores what this branch intends. Both are also current: 2.14.0 is the latest on PyPI and is published on the cu130 aarch64 index and the cpu index, so the routing resolves on both legs.

Lock now carries torch 2.14.0 / torchvision 0.29.0 from here, and ruff 0.16.8 / ty 0.0.82 / pytest 9.1.1 / anyio 4.14.2 from main, with 11 cu130 and 20 cpu wheel URLs recorded.

Checked locally on x86 CPU: ruff format --check, ruff check, ty check, uv lock --check all clean, and pytest --cov=phalanx passes at 18 passed, 1 skipped — the skip being the CUDA-only deterministic test. CI is green on all required checks.

Not merging: the description names a GPU make smoke on a GH200, queued as a Slurm batch job, as the gate.

… the aarch64 CUDA wheels

The gradient comparison would pass without deterministic mode, so the CUDA
test now compares a float32 index_add_ over duplicate indices, which
accumulates with atomics, and restores the global flag. Docs state that
every Linux aarch64 machine, Docker on Apple Silicon included, gets the
CUDA build. set_seed says why deterministic mode is strict.
@ajbarea
ajbarea merged commit fbc665c into main Sep 26, 2026
12 checks passed
@ajbarea
ajbarea deleted the feat/gpu-on-linux-aarch64 branch September 26, 2026 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant