feat(deps): CUDA 13 torch on Linux aarch64, CPU elsewhere; deterministic CUDA kernels - #85
Conversation
…tic CUDA kernels Torch and torchvision resolve from the cu130 index on Linux aarch64 and the cpu index everywhere else, replacing the commented cu128 opt-in that had no aarch64 build. Both forks move to torch 2.14.0 / torchvision 0.29.0 so every platform runs the same version. set_seed disables nondeterministic CUDA kernels when a GPU is present.
…n-level replay Identical smoke runs ended at 0.979 and 0.721: Flower's client sampling is unseeded over random node IDs, so set_seed only makes each client's local training replay.
251097c to
b8b3993
Compare
|
Rebased onto The conflict came from Taking Lock now carries torch 2.14.0 / torchvision 0.29.0 from here, and ruff 0.16.8 / ty 0.0.82 / pytest 9.1.1 / anyio 4.14.2 from Checked locally on x86 CPU: Not merging: the description names a GPU |
… the aarch64 CUDA wheels The gradient comparison would pass without deterministic mode, so the CUDA test now compares a float32 index_add_ over duplicate indices, which accumulates with atomics, and restores the global flag. Docs state that every Linux aarch64 machine, Docker on Apple Silicon included, gets the CUDA build. set_seed says why deterministic mode is strict.
[tool.uv.sources]now routes torch and torchvision by platform marker: Linux aarch64 to thepytorch-cu130index, every other platform (CI included) topytorch-cpu. The commented cu128 opt-in it replaces had no aarch64 build. The lock resolves one version everywhere, torch 2.14.0 / torchvision 0.29.0.set_seedcallstorch.use_deterministic_algorithms(True)when CUDA is present, covered by a CUDA-only test.ROADMAP.mdnow scopes the replay claim to client training and tracks run-level replay: identical smoke runs ended at 0.979 and 0.721 because Flower's client sampling is unseeded over per-run random node IDs.Verification:
make lint,make test(18 passed, CUDA test skipped),make audit,make smokeexit 0.CUBLAS_WORKSPACE_CONFIG.cfaf0d197(this branch withmainmerged):make lintclean;pytest43 passed, 0 skipped, including the CUDA-only deterministic test; GPUmake smoke(client-resources-num-gpus=0.2) 2 rounds, 0 failures. GPU memory rose from 0 to 3,467 MiB during the rounds and utilization peaked at 49%;nvidia-smilists no processes on these nodes, so memory is the evidence.ec1f87136: the CUDA test compares a float32index_add_over duplicate indices. With deterministic mode off, 5 runs gave 5 distinct results (max abs diff 3.7e-3); with it on, 5 identical. Full suite 43 passed, 0 skipped.