Add Vulkan GPU backend - #96
Closed
Janzert wants to merge 12 commits into
Closed
Conversation
The CUDA path is NVIDIA-only. This adds a second backend that runs on any
Vulkan 1.2 device with shaderInt64 -- AMD, Intel, NVIDIA, and llvmpipe --
selected at runtime with --gpu-backend {auto,cuda,vulkan}.
Both cudarc and ash dlopen their driver, so one binary can carry both
backends and require neither at build time. `auto` tries CUDA first, so
behaviour on an NVIDIA machine is unchanged; an explicitly named backend
that fails to initialize is fatal rather than silently falling back to a
much slower path.
Detailed mode only for now; niceonly falls back to the CPU. The detailed
kernel takes no input buffers (each thread derives its own n), so it needs
neither the MSD prefix pipeline nor the stride residue table on-device.
Kernels are generated as WGSL per base and compiled to SPIR-V by naga at
runtime -- the same JIT-per-base design NVRTC gives the CUDA path, which
matters because every divisor in the hot loop must be a compile-time
constant.
The one place the port could not be literal is the scan_digits chunk
split. nvcc strength-reduces `u64 / CHUNK_DIV` to a multiply-high; ACO
does that only for 32-bit divisors (NIR's nir_opt_idiv_const is width
limited), and expands the 64-bit form into a ~220-instruction restoring
shift-subtract loop. So CHUNK_DIV is kept under 2^16 and the split is done
as two 32-bit constant divisions over 16-bit halves, which is exact.
Measured on base 40 (Radeon 860M): 3.88e8 vs 1.36e8 candidates/s, i.e.
2.85x, despite needing ~1.7x more chunk iterations; shader code size
halves and the division expansion leaves the ISA entirely.
Per-base constants move to a new backend-neutral gpu_config module, since
client_process_gpu is cfg(feature = "gpu") and unreachable from a Vulkan
build. The CUDA path keeps every one of its existing tests.
Measured end to end, 1e9 numbers at base 40: 4.33e8/s on the integrated
Radeon 860M against 6.50e7/s for the 16-thread CPU on the same machine.
CPU/GPU parity is exact (distribution and near-miss list) at bases 10, 40,
62 and 80 -- and unlike the CUDA parity tests, it also passes against
lavapipe, so correctness can be checked with no GPU at all.
Assisted-by: Claude Code:claude-opus-5
`--gpu-backend auto` is meant to try CUDA and fall through to Vulkan, but cudarc panics rather than returning an error when libcuda.so is absent (`panic_no_lib_found` returns `!`). That is the ordinary case on a machine with no NVIDIA driver -- precisely the case the fallback exists for -- so `nice_client --gpu` died with cudarc's 20-line message instead of using the Vulkan backend sitting right there in the same binary. Wrap CUDA init in catch_unwind with the panic hook silenced for the duration, so a routine fallback is quiet and the reason is still available at debug level. Upstream catches the same panic for the same reason in nvrtc_compiles_kernels_for_all_supported_bases. An explicit --gpu-backend cuda still fails hard, now with the underlying message and a pointer at --gpu-backend vulkan, rather than a panic. This relies on unwinding; the workspace does not set panic = "abort". Assisted-by: Claude Code:claude-opus-5
Bases with b % 4 == 3 have an empty residue set R_b. That means there are provably no solutions -- but it also means StrideTable's first_valid_at_or_after indexes valid_residues[idx] on an empty vec and panics, so the answer has to be decided before any table is built. process_range_niceonly_vulkan built one unconditionally. The CUDA path has the same guard inside process_range_niceonly_gpu; this puts it ahead of the CPU fallback rather than after it, so it also covers bases the GPU itself cannot take. The server never hands out such bases, so this was not reachable in production -- but --benchmark and any direct call are. Also document --gpu-backend in the README, note that the two backend features are additive and need neither library at build time, and build all three GPU feature combinations in `just test`. Assisted-by: Claude Code:claude-opus-5
Ports niceonly_ranges_kernel to WGSL and lifts the host side of the niceonly path out of client_process_gpu.rs into a new backend-neutral module, so CUDA and Vulkan drive one pipeline instead of two copies. This is the gpu_config.rs split applied to behaviour rather than constants: backends supply a RangeSink (launch + sync), everything above that -- the adaptive MSD floor, the worker/channel/batching loop, the per-field stats -- is written once. The CUDA path was moved onto it; its kernel source is unchanged and it keeps every one of its tests. Correctness is exact against client_process::process_range_niceonly at bases 10, 12, 25, 40, 45 and 62, on RADV and on lavapipe, and base 10 returns exactly [69] through the whole pipeline. That is the real claim: the GPU checks a superset of the CPU's candidates (its MSD floor is coarser), so the two only agree if the on-device candidate reconstruction and the digit scan are both right. One construct did not port, and it is the same one as last time. CUDA computes n mod M with two 64-bit divisions by a constant, which RADV/ACO expands into a ~220-instruction shift-subtract loop rather than strength-reducing. The shader now takes field_start mod M as a push constant and reduces only the 64-bit offset, byte by byte: acc = (acc << 8 | byte) % M, eight 32-bit constant divisions. Exact because acc < M <= 2^24, and real moduli (b-1)*b^2 top out near 2^21. RADV_DEBUG=asm confirms it: zero s_brev_b64, one v_cmp_le_u64 (the loop bound), no scratch, no spills. Two things this turned up that are worth calling out. AdaptiveFloor's metric was wrong on any backend whose launch blocks. It inferred GPU time by subtracting MSD time from the total, which assumes the two phases are disjoint -- true for an async CUDA launch, false for a Vulkan dispatch that waits on a fence inside launch. The same 1e13 base-50 field read "msd 122.6s, gpu 0.15s" before and "msd 5.3s, gpu 133.0s" after: not slightly off, inverted. It would have driven the floor to MSD_FLOOR_MAX every field, which measures >2000s against 129s at the optimum. The pipeline now times its own calls into the sink and subtracts them; CUDA's numbers barely move. Bounding the descriptor channel (needed because a blocking consumer lets the MSD workers queue over a gigabyte) introduced a deadlock on the launch-error path: the consumer breaks, workers parked in send never wake, and thread::scope waits on them forever. Fixed by dropping the receiver inside the scope and breaking on send failure, and pinned by a timeout-guarded test that hangs for 120s without the fix. Not included, deliberately: the modular prefilter (base <= 40 only) and async dispatch. See vulkan-nice-spec.md for both, and for the floor sweep -- end to end this buys ~1.16x over the 16-thread CPU on an iGPU, not detailed mode's 6.7x, because the floor that keeps the GPU fed also multiplies the work it is fed. Assisted-by: Claude:claude-opus-5 [RADV_DEBUG] [clippy]
The niceonly kernel's uniform pre-check — are the lowest p base-b digits of n^2 and n^3 all distinct? — rejects ~99% of candidates in fixed, branch-free work, so whole lanes skip the full check. It was the last piece of niceonly_ranges_kernel not ported. __umul64hi, the difficulty the port expected here, is not needed at all, because the construct it appears in does not survive for a different reason. reduce_pre ends in `lo % PRE_MOD`, and the digit peel is `sq % BASE` / `sq /= BASE` on a u64: three more 64-bit divisions by a compile-time constant, which is exactly the expansion NIR's width-limited nir_opt_idiv_const leaves behind (~220 instructions), in the one place that runs on every candidate. So carry the residue as three chunks of base^chunk_digits < 2^16 — the same chunk the digit scan already uses — instead of in a u64. `n mod b^p` is three rounds of the existing split16 step, the multiply is truncated schoolbook over those chunks (a*b + acc + carry <= chunk_div^2 - 1 < 2^32, so every intermediate stays a u32), and the digits fall out of each chunk with 32-bit divisions. Nothing 64-bit is ever divided. Three chunks also lands on exactly CUDA's digit count at every base that enables either filter. Measured on an AMD Radeon 860M, 1e12 numbers at base 40, MSD floor pinned at 4000: the device phase drops from 31.8-34.7s to 19.0-20.7s (1.68x on the kernel, 1.30e9 vs 7.7e8 candidates/s) and the field from 38.3-41.3s to 25.4-27.2s, against 67.7-71.0s for the same field on the 16-thread CPU. RADV_DEBUG=asm,shaderstats confirms the expansion never appeared: 0 s_brev_b64, scratch 0, 56 VGPRs and 18 subgroups/SIMD with the filter and without, for 362 extra instructions. Correctness needed its own test. The niceonly parity tests cannot see this filter — the only nice number any of them finds is 69, in base 10, where the prefilter is disabled — so a filter that rejected every candidate would pass all of them, which is the bug the CUDA kernel shipped in v3.2.14. A probe build of the shader reports the prefilter's survivors directly and they must match a Rust mirror of the chunk arithmetic candidate for candidate, on RADV and on lavapipe. It agrees at bases 30/34/40, and the 1.12% survival it measures at b40 matches the CUDA crossover study's 1.1% on other hardware. Also adds the massive-low-base benchmark (1e12 @ base 40): every range large enough to measure niceonly was base 50, where the prefilter is compiled out on profitability grounds, so nothing exercised it. NICE_VULKAN_PREFILTER=0 compiles it out, which is how the A/B above was taken. Assisted-by: Claude:claude-opus-5
…patch Phase 3, both changes identified and measured for by earlier phases. The lane tiling was CUDA's warp carried over as a constant 32. Nothing here is a warp — the lanes stride a range's candidates by index and never communicate — but they do all repeat the range's setup, the residue reduction and a ~12-iteration binary search. At MSD floor 250 a base-40 range holds ~39 candidates, so 32 lanes bought 32 copies of that setup to share out 1.2 candidates each. The shader now takes log2(lanes) as a push constant and the host sizes it per dispatch from the batch's mean range length. Device time at floor 250 by pinned width: 30.0s at 32 lanes, 24.6 at 16, 22.8 at 8, 21.6 at 4, 21.1 at 2 and 1 — monotone where ranges are short, flat within noise everywhere else, which is the whole shape of the result. Dispatch is now asynchronous over a ring of command buffers and fences, with per-slot range descriptor buffers so the host fills batch i+1 while the device reads batch i. The reservation is a guard that holds the submitter lock across the caller's fill: without it two threads could be handed the same slot, since nothing marks a slot taken until its submit. Holding it costs no device time — it blocks other hosts, not the queue. Measured on an AMD Radeon 860M, totals at the floor each configuration is best at. Base 50, 1e13: 129.4s -> 86.5s, against 149.7-180.5s for the 16-thread CPU, so this base goes from 1.16x the CPU to 1.7-2.1x. Base 40, 1e12: 25.4s -> 22.5s pinned, ~23.6s adaptive. The prefilter does not apply at base 50, so that base's whole gain is this phase. It also repairs the control loop rather than the constant. AdaptiveFloor aims at msd == gpu, which minimizes a maximum — right only when the phases overlap. They now do: the same base-40 field that used to walk the floor from 24.6s down to 57s settles into a limit cycle at 23.5-25.0s against 22.3s for the best fixed floor, ~5% off where it was 2.3x off. The residue is the hard +/-1.5x step, not the target. Two constants flagged as never swept are now swept and both are flat: HIST_COPIES (1/2/4/8: 2.66/2.58/2.62/2.65s) and MAX_WORKGROUPS (1024..16384: 22.6-22.8s). Left at 4 and 4096. One caveat the async path introduces: device_secs is time the host spent blocked in launch/sync, not the device's work, and an asynchronous backend drives it to nearly zero on a field the device keeps up with. Compare totals. Assisted-by: Claude:claude-opus-5
The bound was quoted as LAUNCH_BATCH_RANGES * PIPELINE_DEPTH descriptors, which names the wrong quantity. A channel item is the whole output of one PROCESSING_CHUNK_SIZE chunk; LAUNCH_BATCH_RANGES is the consumer's flush threshold and never bounds what is queued. The real cap is PIPELINE_DEPTH times the most ranges one chunk can yield. get_valid_ranges_recursive returns a range whole once it is at or below the floor, so that is about PROCESSING_CHUNK_SIZE / floor -- ~4000 at MSD_FLOOR_MIN, fewer at any coarser floor, and the depth-22 subdivision limit never binds first. Roughly 3 MB rather than the 50 MB the old arithmetic implied; the stated bound was conservative, so nothing was unsafe, but this constant is one of the two things that changed for CUDA users (the channel used to be unbounded) and it should say what it does. Comment only; no behaviour change. Assisted-by: Claude:claude-opus-5
…nt LSD bitmap Brings wasabipesto/nice 0ed8652 onto vulkan-backend. Two upstream filter fixes, both of which the Vulkan backend inherits without shader changes: - has_duplicate_msd_prefix loses its cross MSD-LSD collision check, which was unsound (it treated the range start's low square/cube digits as fixed across a range whose n mod b^k varies). Ranges containing nice numbers were being skipped, e.g. [68, 70) in base 10 around 69. The Vulkan niceonly path calls get_valid_ranges_recursive, so it was skipping the same ranges the CPU was. - get_valid_multi_lsd_bitmap now checks all 2k fixed low digits of n^2 and n^3 for pairwise distinctness at fixed width. The GPU uploads this host-built stride table verbatim, so the tighter bitmap reaches the device with no kernel or codegen change. Expect niceonly to slow down where the old filter was wrongly aggressive (upstream measured ~1.6x at b40, ~2.6x at b50); that cost is the fix.
…2 storage Brings wasabipesto/nice up to a066ab3 (origin/main). wasabipesto#87 came in on the previous merge; this one is wasabipesto#88 plus the version/changelog commits. Conflict: wasabipesto#88 edited GPU_LSD_K and the MSD-floor machinery in place in client_process_gpu.rs, but this branch had already moved both into gpu_niceonly.rs so the Vulkan backend could share them. Resolved by keeping the shared module and carrying wasabipesto#88's semantic change to it — GPU_LSD_K 2 -> 3. The duplicated AdaptiveFloor block upstream re-added was dropped; it is identical to the one already in gpu_niceonly.rs. The k=3 table needed one real fix on the Vulkan side. The niceonly shader reduces a range offset with a Horner loop, acc = (acc << c | chunk) % M, which requires M <= 2^(32-c); c was a fixed 8, sound only because M = (b-1)*b^k stayed under 2^24 at k=2. At k=3 every modulus gains a factor of b: base 65 hits 17 576 000 and base 128 hits 266 338 304, so NiceonlyConfig::new would have refused every base >= 65 — base 80, which is in production here, among them. stride_chunk_bits now picks c per base (8 while M <= 2^24, else 4, bound 2^28), and the emitted shader and the host mirror derive it from the same modulus. It costs nothing measurable: this reduction runs once per range descriptor, not per candidate. Also adapted this branch's stride-table test mirrors to the u32 residue and gap types, the same way wasabipesto#88 did for the CUDA ones, and added base 80 to the device parity list — every base below 65 keeps the byte chunk, so nothing else here executes the 4-bit path on hardware. Verified: 120 unit tests plus all 6 device tests pass on the AMD iGPU; niceonly and detailed both match the CPU at bases 10/12/25/40/45/62/80. Assisted-by: Claude:claude-opus-5
…ark sweep, telemetry Upstream replaced the fixed named benchmark fields (common/src/benchmark.rs) with a structured adaptive sweep in client/src/bench.rs (wasabipesto#90), added benchmark uploads and opt-in submission telemetry (wasabipesto#91), a POST /estimate endpoint with GPU model canonicalization (wasabipesto#93, wasabipesto#95), and swapped the MSD prefix checks for an interval digit-domain (Hall) analysis (wasabipesto#89). One conflict and two integration fixes: * common/src/benchmark.rs was deleted upstream and carried our MassiveLowBase mode, which existed only because every other niceonly benchmark was base 50, where the modular prefilter is compiled out (GPU_PREFILTER_MAX_BASE). Took the deletion: the new sweep's niceonly scenarios include two base-40 windows (b40_msd_strong, b40_msd_weak), so the prefilter is exercised again. * DataToServer gained a telemetry field; client_process_vulkan.rs builds two of them and needed the same `telemetry: None` wasabipesto#91 gave the CUDA path. * gpu_name() was CUDA-only, so a Vulkan run reported gpu:true with a null gpu_model, and on a box with both backends a --gpu-backend vulkan run would have reported the CUDA device's name. Both feed the new estimator's matching. It now asks the live GpuHandle, which is the only thing that knows which backend actually initialized; Vulkan already recorded device_name at init. wasabipesto#89 needed no change on our side: the Vulkan pipeline calls get_valid_ranges_recursive rather than reimplementing the filter, so it inherits the stronger prune. Device parity re-verified — all 6 device tests and 131 unit tests pass, including vulkan_matches_cpu_niceonly at base 80. No GPU performance number was re-measured. wasabipesto#89 trades -2 to -12% must-process work for +0 to +8% CPU wall clock, which shifts the host/device balance in the niceonly pipeline and invalidates the pre-merge Vulkan speedups again. Assisted-by: Claude:claude-opus-5
…eardown `DetailedRun::drop` and `NiceonlyRun::drop` called `vkDeviceWaitIdle`, which is specified as a `vkQueueWaitIdle` on *every* queue of the device and so requires external synchronization against all of them. That is a precondition neither `Drop` is in a position to guarantee, and violating it is undefined behaviour whose observed cost in nice-count was not a crash but whole dispatches silently doing nothing -- the host fenced, read zeroed counters, and folded the work away. No such violation is reachable today: one field is processed at a time, and only `run_range_pipeline`'s consumer thread submits -- the MSD workers just produce descriptors down a channel. So this is not a bug fix, it is the removal of an unstated invariant. Nothing in either `Drop` records that the call is safe only because no second thread submits, and a future overlapping field or second context would break it silently rather than loudly. `wait_all` needs no such invariant: it takes the submitter lock and waits the fences of the slots actually in flight, which is a superset of the dropping run's own dispatches and the only work that can still be reading the descriptor pools and buffers freed immediately below. `VulkanContext::drop` keeps `device_wait_idle`. It owns the queue, and the `'a` borrow both run types take makes "every run is already dropped" a compile-time guarantee rather than a convention. Teardown-only, so there is nothing to measure. All 131 unit tests and all 6 device tests pass on the AMD iGPU, including `vulkan_matches_cpu_detailed` and `vulkan_matches_cpu_niceonly` through base 80, both of which exercise each changed `Drop`. Assisted-by: Claude:claude-opus-5
Owner
|
Wow! I'll have to dig into this later but I think Vulkan would be a great target here. Tests in CI without a GPU is a nice plus. I also agree with feature option C, there's no reason to have the two GPU backends broken out unless it's like double the size or build time. |
…ella The `gpu` feature named one backend out of two. It read as "GPU support" but meant CUDA specifically, and the file layout already showed the strain: `gpu_config.rs` and `gpu_niceonly.rs` are backend-neutral while `client_process_gpu.rs` is CUDA-only, all under one feature name. `cuda` and `vulkan` now name the two backends and `gpu` is an umbrella over both, in `nice_common` and `nice_client` alike. The client's direct `cudarc` dependency moves onto `cuda`. `Cargo.lock` is unchanged, so the CI build's `--locked` still resolves. BREAKING: `--features gpu` keeps working but produces a different binary, with no diagnostic from cargo. That is intended — `docker-build-push.yml` is deliberately left on `--features gpu`, so the published image now carries both backends. Its `nvidia/cuda` base ships no Vulkan ICD, so the second backend is inert there, and `auto` still prefers CUDA, so default behaviour does not change. Cargo does not propagate feature cfgs upward: a `--features cuda` build has `cfg(feature = "cuda")` and *no* `cfg(feature = "gpu")`. The umbrella is therefore a build-command convenience only, and no `#[cfg]` names it — the backend-neutral sites stay `any(feature = "cuda", feature = "vulkan")`. Collapsing one of those to the umbrella would compile and silently drop the code from every single-backend build. Also hide `--gpu`, `--gpu-device` and `--gpu-backend` from `--help` when no backend is compiled in. Hidden rather than removed, so the flags still parse and a CPU-only binary answers `--gpu` with the explanatory error instead of clap's "unexpected argument" — which matters for the docker images, where that flag is the only difference between the two tags. Verified: all five feature combinations build and are clippy-clean; each reports its own backends correctly at runtime; `auto` falls through CUDA to Vulkan in an umbrella build; 131 unit tests and all 6 Vulkan device parity tests pass on an AMD 860M. Assisted-by: Claude:claude-opus-5
wasabipesto-bot
pushed a commit
to wasabipesto-bot/nice
that referenced
this pull request
Aug 16, 2026
CUDA gets the wide flavor (one u64 constant division per limb, 5 digits per chunk at base 40, matching the hand kernel); wgpu keeps split16 for drivers that cannot strength-reduce 64-bit constant division. One kernel source, one comptime bool — this is the per-device choice PR wasabipesto#96 lists as future work for the WGSL generator. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
15 tasks
Contributor
Author
|
PR #99 builds on and supersedes this so I think this one can now be closed. |
wasabipesto
pushed a commit
that referenced
this pull request
Aug 19, 2026
* feat(gpu): add a Vulkan compute backend alongside CUDA
The CUDA path is NVIDIA-only. This adds a second backend that runs on any
Vulkan 1.2 device with shaderInt64 -- AMD, Intel, NVIDIA, and llvmpipe --
selected at runtime with --gpu-backend {auto,cuda,vulkan}.
Both cudarc and ash dlopen their driver, so one binary can carry both
backends and require neither at build time. `auto` tries CUDA first, so
behaviour on an NVIDIA machine is unchanged; an explicitly named backend
that fails to initialize is fatal rather than silently falling back to a
much slower path.
Detailed mode only for now; niceonly falls back to the CPU. The detailed
kernel takes no input buffers (each thread derives its own n), so it needs
neither the MSD prefix pipeline nor the stride residue table on-device.
Kernels are generated as WGSL per base and compiled to SPIR-V by naga at
runtime -- the same JIT-per-base design NVRTC gives the CUDA path, which
matters because every divisor in the hot loop must be a compile-time
constant.
The one place the port could not be literal is the scan_digits chunk
split. nvcc strength-reduces `u64 / CHUNK_DIV` to a multiply-high; ACO
does that only for 32-bit divisors (NIR's nir_opt_idiv_const is width
limited), and expands the 64-bit form into a ~220-instruction restoring
shift-subtract loop. So CHUNK_DIV is kept under 2^16 and the split is done
as two 32-bit constant divisions over 16-bit halves, which is exact.
Measured on base 40 (Radeon 860M): 3.88e8 vs 1.36e8 candidates/s, i.e.
2.85x, despite needing ~1.7x more chunk iterations; shader code size
halves and the division expansion leaves the ISA entirely.
Per-base constants move to a new backend-neutral gpu_config module, since
client_process_gpu is cfg(feature = "gpu") and unreachable from a Vulkan
build. The CUDA path keeps every one of its existing tests.
Measured end to end, 1e9 numbers at base 40: 4.33e8/s on the integrated
Radeon 860M against 6.50e7/s for the 16-thread CPU on the same machine.
CPU/GPU parity is exact (distribution and near-miss list) at bases 10, 40,
62 and 80 -- and unlike the CUDA parity tests, it also passes against
lavapipe, so correctness can be checked with no GPU at all.
Assisted-by: Claude Code:claude-opus-5
* fix(gpu): fall back to Vulkan when the CUDA library is missing
`--gpu-backend auto` is meant to try CUDA and fall through to Vulkan, but
cudarc panics rather than returning an error when libcuda.so is absent
(`panic_no_lib_found` returns `!`). That is the ordinary case on a machine
with no NVIDIA driver -- precisely the case the fallback exists for -- so
`nice_client --gpu` died with cudarc's 20-line message instead of using
the Vulkan backend sitting right there in the same binary.
Wrap CUDA init in catch_unwind with the panic hook silenced for the
duration, so a routine fallback is quiet and the reason is still available
at debug level. Upstream catches the same panic for the same reason in
nvrtc_compiles_kernels_for_all_supported_bases.
An explicit --gpu-backend cuda still fails hard, now with the underlying
message and a pointer at --gpu-backend vulkan, rather than a panic.
This relies on unwinding; the workspace does not set panic = "abort".
Assisted-by: Claude Code:claude-opus-5
* fix(gpu): guard residue-empty bases before building a stride table
Bases with b % 4 == 3 have an empty residue set R_b. That means there are
provably no solutions -- but it also means StrideTable's
first_valid_at_or_after indexes valid_residues[idx] on an empty vec and
panics, so the answer has to be decided before any table is built.
process_range_niceonly_vulkan built one unconditionally. The CUDA path has
the same guard inside process_range_niceonly_gpu; this puts it ahead of the
CPU fallback rather than after it, so it also covers bases the GPU itself
cannot take.
The server never hands out such bases, so this was not reachable in
production -- but --benchmark and any direct call are.
Also document --gpu-backend in the README, note that the two backend
features are additive and need neither library at build time, and build
all three GPU feature combinations in `just test`.
Assisted-by: Claude Code:claude-opus-5
* feat(vulkan): run niceonly on the GPU, on a pipeline both backends share
Ports niceonly_ranges_kernel to WGSL and lifts the host side of the
niceonly path out of client_process_gpu.rs into a new backend-neutral
module, so CUDA and Vulkan drive one pipeline instead of two copies.
This is the gpu_config.rs split applied to behaviour rather than
constants: backends supply a RangeSink (launch + sync), everything above
that -- the adaptive MSD floor, the worker/channel/batching loop, the
per-field stats -- is written once. The CUDA path was moved onto it; its
kernel source is unchanged and it keeps every one of its tests.
Correctness is exact against client_process::process_range_niceonly at
bases 10, 12, 25, 40, 45 and 62, on RADV and on lavapipe, and base 10
returns exactly [69] through the whole pipeline. That is the real claim:
the GPU checks a superset of the CPU's candidates (its MSD floor is
coarser), so the two only agree if the on-device candidate
reconstruction and the digit scan are both right.
One construct did not port, and it is the same one as last time. CUDA
computes n mod M with two 64-bit divisions by a constant, which RADV/ACO
expands into a ~220-instruction shift-subtract loop rather than
strength-reducing. The shader now takes field_start mod M as a push
constant and reduces only the 64-bit offset, byte by byte:
acc = (acc << 8 | byte) % M, eight 32-bit constant divisions. Exact
because acc < M <= 2^24, and real moduli (b-1)*b^2 top out near 2^21.
RADV_DEBUG=asm confirms it: zero s_brev_b64, one v_cmp_le_u64 (the loop
bound), no scratch, no spills.
Two things this turned up that are worth calling out.
AdaptiveFloor's metric was wrong on any backend whose launch blocks. It
inferred GPU time by subtracting MSD time from the total, which assumes
the two phases are disjoint -- true for an async CUDA launch, false for
a Vulkan dispatch that waits on a fence inside launch. The same 1e13
base-50 field read "msd 122.6s, gpu 0.15s" before and "msd 5.3s, gpu
133.0s" after: not slightly off, inverted. It would have driven the
floor to MSD_FLOOR_MAX every field, which measures >2000s against 129s
at the optimum. The pipeline now times its own calls into the sink and
subtracts them; CUDA's numbers barely move.
Bounding the descriptor channel (needed because a blocking consumer lets
the MSD workers queue over a gigabyte) introduced a deadlock on the
launch-error path: the consumer breaks, workers parked in send never
wake, and thread::scope waits on them forever. Fixed by dropping the
receiver inside the scope and breaking on send failure, and pinned by a
timeout-guarded test that hangs for 120s without the fix.
Not included, deliberately: the modular prefilter (base <= 40 only) and
async dispatch. See vulkan-nice-spec.md for both, and for the floor
sweep -- end to end this buys ~1.16x over the 16-thread CPU on an iGPU,
not detailed mode's 6.7x, because the floor that keeps the GPU fed also
multiplies the work it is fed.
Assisted-by: Claude:claude-opus-5 [RADV_DEBUG] [clippy]
* feat(vulkan): port the niceonly modular prefilter, in chunks not u64
The niceonly kernel's uniform pre-check — are the lowest p base-b digits of
n^2 and n^3 all distinct? — rejects ~99% of candidates in fixed, branch-free
work, so whole lanes skip the full check. It was the last piece of
niceonly_ranges_kernel not ported.
__umul64hi, the difficulty the port expected here, is not needed at all,
because the construct it appears in does not survive for a different reason.
reduce_pre ends in `lo % PRE_MOD`, and the digit peel is `sq % BASE` /
`sq /= BASE` on a u64: three more 64-bit divisions by a compile-time
constant, which is exactly the expansion NIR's width-limited
nir_opt_idiv_const leaves behind (~220 instructions), in the one place that
runs on every candidate.
So carry the residue as three chunks of base^chunk_digits < 2^16 — the same
chunk the digit scan already uses — instead of in a u64. `n mod b^p` is three
rounds of the existing split16 step, the multiply is truncated schoolbook over
those chunks (a*b + acc + carry <= chunk_div^2 - 1 < 2^32, so every
intermediate stays a u32), and the digits fall out of each chunk with 32-bit
divisions. Nothing 64-bit is ever divided. Three chunks also lands on exactly
CUDA's digit count at every base that enables either filter.
Measured on an AMD Radeon 860M, 1e12 numbers at base 40, MSD floor pinned at
4000: the device phase drops from 31.8-34.7s to 19.0-20.7s (1.68x on the
kernel, 1.30e9 vs 7.7e8 candidates/s) and the field from 38.3-41.3s to
25.4-27.2s, against 67.7-71.0s for the same field on the 16-thread CPU.
RADV_DEBUG=asm,shaderstats confirms the expansion never appeared: 0
s_brev_b64, scratch 0, 56 VGPRs and 18 subgroups/SIMD with the filter and
without, for 362 extra instructions.
Correctness needed its own test. The niceonly parity tests cannot see this
filter — the only nice number any of them finds is 69, in base 10, where the
prefilter is disabled — so a filter that rejected every candidate would pass
all of them, which is the bug the CUDA kernel shipped in v3.2.14. A probe
build of the shader reports the prefilter's survivors directly and they must
match a Rust mirror of the chunk arithmetic candidate for candidate, on RADV
and on lavapipe. It agrees at bases 30/34/40, and the 1.12% survival it
measures at b40 matches the CUDA crossover study's 1.1% on other hardware.
Also adds the massive-low-base benchmark (1e12 @ base 40): every range large
enough to measure niceonly was base 50, where the prefilter is compiled out
on profitability grounds, so nothing exercised it. NICE_VULKAN_PREFILTER=0
compiles it out, which is how the A/B above was taken.
Assisted-by: Claude:claude-opus-5
* perf(vulkan): tile lanes to the ranges, and stop waiting on every dispatch
Phase 3, both changes identified and measured for by earlier phases.
The lane tiling was CUDA's warp carried over as a constant 32. Nothing here is
a warp — the lanes stride a range's candidates by index and never communicate
— but they do all repeat the range's setup, the residue reduction and a
~12-iteration binary search. At MSD floor 250 a base-40 range holds ~39
candidates, so 32 lanes bought 32 copies of that setup to share out 1.2
candidates each. The shader now takes log2(lanes) as a push constant and the
host sizes it per dispatch from the batch's mean range length. Device time at
floor 250 by pinned width: 30.0s at 32 lanes, 24.6 at 16, 22.8 at 8, 21.6 at
4, 21.1 at 2 and 1 — monotone where ranges are short, flat within noise
everywhere else, which is the whole shape of the result.
Dispatch is now asynchronous over a ring of command buffers and fences, with
per-slot range descriptor buffers so the host fills batch i+1 while the device
reads batch i. The reservation is a guard that holds the submitter lock across
the caller's fill: without it two threads could be handed the same slot, since
nothing marks a slot taken until its submit. Holding it costs no device time —
it blocks other hosts, not the queue.
Measured on an AMD Radeon 860M, totals at the floor each configuration is best
at. Base 50, 1e13: 129.4s -> 86.5s, against 149.7-180.5s for the 16-thread
CPU, so this base goes from 1.16x the CPU to 1.7-2.1x. Base 40, 1e12: 25.4s ->
22.5s pinned, ~23.6s adaptive. The prefilter does not apply at base 50, so
that base's whole gain is this phase.
It also repairs the control loop rather than the constant. AdaptiveFloor aims
at msd == gpu, which minimizes a maximum — right only when the phases overlap.
They now do: the same base-40 field that used to walk the floor from 24.6s
down to 57s settles into a limit cycle at 23.5-25.0s against 22.3s for the
best fixed floor, ~5% off where it was 2.3x off. The residue is the hard
+/-1.5x step, not the target.
Two constants flagged as never swept are now swept and both are flat:
HIST_COPIES (1/2/4/8: 2.66/2.58/2.62/2.65s) and MAX_WORKGROUPS (1024..16384:
22.6-22.8s). Left at 4 and 4096.
One caveat the async path introduces: device_secs is time the host spent
blocked in launch/sync, not the device's work, and an asynchronous backend
drives it to nearly zero on a field the device keeps up with. Compare totals.
Assisted-by: Claude:claude-opus-5
* docs(gpu): state PIPELINE_DEPTH's memory bound in the right unit
The bound was quoted as LAUNCH_BATCH_RANGES * PIPELINE_DEPTH descriptors,
which names the wrong quantity. A channel item is the whole output of one
PROCESSING_CHUNK_SIZE chunk; LAUNCH_BATCH_RANGES is the consumer's flush
threshold and never bounds what is queued.
The real cap is PIPELINE_DEPTH times the most ranges one chunk can yield.
get_valid_ranges_recursive returns a range whole once it is at or below
the floor, so that is about PROCESSING_CHUNK_SIZE / floor -- ~4000 at
MSD_FLOOR_MIN, fewer at any coarser floor, and the depth-22 subdivision
limit never binds first. Roughly 3 MB rather than the 50 MB the old
arithmetic implied; the stated bound was conservative, so nothing was
unsafe, but this constant is one of the two things that changed for CUDA
users (the channel used to be unbounded) and it should say what it does.
Comment only; no behaviour change.
Assisted-by: Claude:claude-opus-5
* fix(vulkan): wait on the submission slots, not the whole device, at teardown
`DetailedRun::drop` and `NiceonlyRun::drop` called `vkDeviceWaitIdle`, which
is specified as a `vkQueueWaitIdle` on *every* queue of the device and so
requires external synchronization against all of them. That is a precondition
neither `Drop` is in a position to guarantee, and violating it is undefined
behaviour whose observed cost in nice-count was not a crash but whole
dispatches silently doing nothing -- the host fenced, read zeroed counters,
and folded the work away.
No such violation is reachable today: one field is processed at a time, and
only `run_range_pipeline`'s consumer thread submits -- the MSD workers just
produce descriptors down a channel. So this is not a bug fix, it is the
removal of an unstated invariant. Nothing in either `Drop` records that the
call is safe only because no second thread submits, and a future overlapping
field or second context would break it silently rather than loudly.
`wait_all` needs no such invariant: it takes the submitter lock and waits the
fences of the slots actually in flight, which is a superset of the dropping
run's own dispatches and the only work that can still be reading the
descriptor pools and buffers freed immediately below.
`VulkanContext::drop` keeps `device_wait_idle`. It owns the queue, and the
`'a` borrow both run types take makes "every run is already dropped" a
compile-time guarantee rather than a convention.
Teardown-only, so there is nothing to measure. All 131 unit tests and all 6
device tests pass on the AMD iGPU, including `vulkan_matches_cpu_detailed`
and `vulkan_matches_cpu_niceonly` through base 80, both of which exercise
each changed `Drop`.
Assisted-by: Claude:claude-opus-5
* Add a CubeCL detailed-mode backend for evaluation against the hand-written kernels
The vulkan/codegen.rs detailed kernel expressed as a #[cube] Rust function:
same per-base constants, same split16 chunk scan (every division 32-bit by a
comptime-constant divisor), same 4-copy workgroup histogram, same MISS_STRIDE
near-miss records. Per-base WGSL string generation becomes #[comptime]
parameters; CubeCL JIT-specializes one kernel per base at first launch.
Selected with --gpu-backend cubecl (feature "cubecl"; detailed mode only).
The benchmark sweep runs through it unchanged, so rates are directly
comparable with the cuda and vulkan backends on the same scenarios.
Parity: cubecl_matches_cpu_detailed checks exact distribution + near-miss
agreement with the CPU at bases 10/40/62/80, passing on lavapipe (same
harness as vulkan_matches_cpu_detailed).
This is a benchmark-grade evaluation port, not a merge candidate: niceonly
is not ported, and device selection takes wgpu's default adapter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add the CubeCL native CUDA runtime behind --gpu-backend cubecl-cuda
The kernel was already runtime-generic; this makes the host body generic
too and adds a CUDA context variant (feature cubecl-cuda), so the NVIDIA
comparison can race CubeCL's CUDA codegen against the hand-written kernels
directly, without depending on container Vulkan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Raise cudarc API surface to cuda-12080 for cubecl-cuda 0.10 compatibility
cubecl-cuda references CUDA 12.8 tensor-map symbols unconditionally; with
the workspace's cuda-12000 floor unioned in, cudarc compiles the 12.0
surface and the build fails. The hand-CUDA path calls none of the new
APIs and symbol lookup is lazy under dynamic loading. Evaluation branch
only; revisit the floor before any upstreaming.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add a CubeCL-CUDA parity test for real NVIDIA silicon
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Select the chunk-scan width per runtime at comptime
CUDA gets the wide flavor (one u64 constant division per limb, 5 digits
per chunk at base 40, matching the hand kernel); wgpu keeps split16 for
drivers that cannot strength-reduce 64-bit constant division. One kernel
source, one comptime bool — this is the per-device choice PR #96 lists as
future work for the WGSL generator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Enable Vulkan portability enumeration so MoltenVK devices are visible
Loaders >= 1.3.207 hide portability (non-conformant) drivers unless the
instance enables VK_KHR_portability_enumeration, and the spec requires
VK_KHR_portability_subset on devices that advertise it. Both are detected
at runtime, so this is a no-op on conformant Linux/Windows stacks.
Verified on an Apple M4 mini through MoltenVK 1.4.2: all six device
parity tests pass (detailed, niceonly, prefilter mirror), and the
detailed benchmark runs at 6.7e8 n/s (b40) / 3.9e8 n/s (b50) --
CubeCL-on-Metal measures 0.86x of this hand-WGSL backend on the same
device.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Restructure GPU features: per-backend flags under one gpu umbrella
The CUDA backend's feature and symbols lose their generic names: feature
gpu -> cuda, client_process_gpu -> client_process_cuda, GpuContext ->
CudaContext, GPU_BATCH_SIZE -> CUDA_BATCH_SIZE, process_*_gpu ->
process_*_cuda. (cudarc's own CudaContext is imported as DriverContext
where the two would collide.) Names that are genuinely backend-neutral —
gpu_config, gpu_niceonly, --gpu-backend, the gpu CLI flag — keep them.
A new umbrella feature takes the gpu name: it enables every backend
(cuda, vulkan, cubecl, cubecl-cuda), so the release recipe is two
builds — default features for a lightweight CPU-only binary, --features
gpu for one binary that runs on the CPU or any supported GPU. No backend
needs its GPU libraries at build time; verified by building the full
umbrella on a machine with no CUDA toolkit.
--gpu-backend auto now ranks backends by measured speed: CUDA, then
CubeCL, then Vulkan (CubeCL measured 1.4x the hand-WGSL backend on
RDNA4/RADV and 2.5x on NVIDIA). In niceonly mode auto skips CubeCL,
which has no niceonly kernel yet, and falls through to Vulkan; an
explicitly named backend that fails to init stays fatal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Run device parity tests in CI on every backend a hosted runner can drive
Two jobs: lavapipe on Linux (vulkan + cubecl through software Vulkan,
the same device the suites were developed against) and macOS arm64
(cubecl through wgpu->Metal on the runner's paravirtualized Apple GPU,
and the hand-WGSL backend through MoltenVK, exercising the new
portability-enumeration path). The workflow header notes the backends
CI cannot cover: cuda and cubecl-cuda need NVIDIA silicon, so their
result parity stays a manual step on real hardware — NVRTC
compile-tests of every base's kernel still run in the normal suite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Port niceonly mode to the CubeCL backend
The niceonly kernel joins detailed as a #[cube] Rust function: the same
on-device candidate reconstruction from the residue table, offset
reduction (chunked u32 Horner, every divisor comptime), lane tiling, and
low-digit modular prefilter as the generated WGSL, reviewable side by
side with it. The stride modulus and residue count ride along as
comptime parameters, so CubeCL specializes one kernel per base exactly
as the string generators bake literals.
The host end is a RangeSink like the other backends', driving the shared
gpu_niceonly pipeline; the tiling helpers (lane_shift_for,
stride_chunk_bits, MAX_STRIDE_MODULUS, MAX_LANES_PER_RANGE) and the
residue-empty guard move from vulkan/codegen.rs and
client_process_vulkan.rs into gpu_niceonly, where the Vulkan backend
re-imports them — they were never Vulkan-specific, and the CubeCL
backend must build without the vulkan feature.
Device tests mirror the Vulkan suite: CPU parity at bases
10/12/25/40/45/62/80 (the GPU checks a superset of the CPU's candidates;
the sets must still be identical), 69-in-base-10 through the whole
pipeline, a CUDA-runtime variant, and a prefilter probe that reports
survivors at every lane width against an independent u128 host mirror —
an over-rejecting prefilter passes every parity test (the v3.2.14 bug
class), so the filter's own output has to be observable. All pass on
lavapipe; survivor counts match the hand-WGSL filter's exactly
(396/159058 b30, 2099/350205 b34, 4472/288591 b40).
With both modes served, --gpu-backend auto drops its niceonly carve-out:
the order is CUDA, CubeCL, Vulkan in every mode.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Pin cubecl exactly and document the cudarc surface rationale
=0.10.0: CubeCL is pre-1.0 and its host API drifts between releases;
bumps should be deliberate, gated on the device parity suite.
cuda-12080 stays, now with the argument written down: it selects which
bindings are compiled, not which driver is required — cudarc's dynamic
loading resolves symbols lazily, the hand-CUDA path calls nothing newer
than 12.0, and cubecl-cuda's 12.8 declarations (tensor-map) are never
launched by our kernels. Parity-verified on real silicon.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Fix the lavapipe ICD glob for Ubuntu's file name
Ubuntu ships the lavapipe manifest as lvp_icd.json; upstream Mesa
builds name it lvp_icd.<arch>.json. The job's glob only matched the
latter, so VK_ICD_FILENAMES exported empty and the loader found no
drivers. Glob both, and fail the step loudly when no ICD is found
instead of exporting an empty loader list. Verified in an ubuntu:24.04
container.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Report the active GPU backend in benchmark and telemetry hardware info
Three backends at very different throughputs can share one device name
(the 9070 XT differs 1.4x between vulkan and cubecl), so hardware rows
without the backend cannot be compared. hardware.gpu_backend carries the
--gpu-backend value of the backend that actually won init — read from
the live handle, since auto decides at init among the compiled backends.
Additive JSON; absent on CPU runs; no schema bump.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Cache the niceonly plan per base instead of rebuilding it per field
CubeclNiceonlyRun rebuilt the stride table and re-uploaded the residue
buffer on every field. StrideTable::new walks the whole modulus on one
CPU thread — 8.6e6 steps at base 52 — and at benchmark-window field
sizes that walk dominated the pipeline: on an RTX 4060 the per-field
logs showed the device phase at 0.65-1.0x of hand-CUDA while the
scenario rates cratered to 0.24-0.57x, the difference being wall time
spent between fields on table rebuilds the hand backends never do (both
cache their per-base plans in the context; now this backend does too).
Explains the base-dependence (modulus grows with base) and the device
spread (the slower the box's CPU, the worse the hit).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Add NICE_CUBECL_LANES to pin the niceonly lane tiling for measurement
Same shape of knob as the hand backend's NICE_VULKAN_LANES: a power of
two up to 32 pins every dispatch's lane width; unset keeps the adaptive
per-dispatch sizing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Straight-line the niceonly full check like the hand kernels
The pass-loop structure copied sq into a zero-padded cu_limbs window,
built cu in its own array and copied it into the scratch, and branched
on a runtime pass selector — per candidate. At bases with the low-digit
prefilter (<= 40) almost nothing reaches the full check and none of
that mattered; above it every candidate pays, which is where the
remaining device-phase gap against hand-CUDA lived. Now the sq scan
reads a sq_limbs window and the cube is built directly in the scratch
and destroyed by its scan — no cu array, no copies, no pass branch,
matching check_is_nice in nice_kernels.cu shape for shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Drain the detailed histogram once per 64 batches, not per batch
Batches launch asynchronously; the blocking histogram read is the only
stream sync, and doing it per 50M batch put a GPU-idle bubble after
every ~12ms of kernel work — measured as most of the hand-CUDA
backend's remaining detailed-mode lead on an RTX 4060. Each candidate
increments exactly one u32 bin, so a drain per 64 batches keeps every
bin under 64 * 50e6 < u32::MAX (const-asserted). The hand-CUDA kernel
sidesteps the bound with u64 bins; WGSL has no u64 atomics, and this
host loop serves both runtimes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Compute remainders by mul-sub from the quotient, never by %
The kernels paired every constant division with a matching %. nvcc
lowers each u64 constant division independently — a full multiply-high
correction sequence per operator — so every div+mod pair cost double,
and the digit peel pays the pair per digit. SASS for the b50 detailed
kernel showed +42% instructions and ~100 extra ISETP correction
compares vs the hand kernel, whose source computes every remainder as
cur - q * DIV for exactly this reason. All sixteen div+mod sites in
the detailed scan, niceonly scans, and prefilter now do the same.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Keep the digit-scan scratch in shared memory, not a promoted array
ptxas register-promotes the generated kernels' scratch array, turning
every runtime-indexed access in the split loop into a compare/select
chain over all words — ~100 extra SASS compares per kernel at b50,
where the hand kernels' local-memory array pays one LDL/STL. Shared
memory gives the same one-instruction indexed access without the local
round trip: one word per thread per limb, padded to an odd stride so a
warp's accesses spread across banks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Fail cubecl-cuda init loudly when the runtime cannot run kernels
Review found the silent-wrong-results case: with an NVIDIA driver but
no CUDA toolkit, CubeCL compiles lazily at first launch, a missing
NVRTC panics its server thread, and launches keep reporting success —
the client completes and reports numbers measured from zeroed buffers.
new_cuda now proves the runtime end-to-end with a one-thread smoke
kernel and a verified readback before reporting success (the hand
backends' init-time compile checks, ported), and detailed_impl checks
histogram conservation — every candidate lands in exactly one bin — so
any future silently-dropped work refuses to produce results on any
runtime.
Also from review: the client's cubecl-cuda feature now enables its own
cubecl gate (feature cfgs do not propagate upward, so the standalone
feature did not compile); the wgpu adapter log and telemetry name now
include the resolved graphics API (AutoGraphicsApi may pick DX12 on
Windows, and backend comparisons need to know); Auto's docs state that
cubecl-cuda is deliberately explicit-only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Revert the prefilter's remainder rewrites: mul-sub miscompiles on MSL
The M4's prefilter probe caught survivors diverging from the host
mirror after the mul-sub commit — Metal only; the CUDA and SPIR-V
paths agree with the mirror, and the digit scan's mul-sub sites pass
full parity on Metal, so the miscompile is specific to these four
prefilter sites under naga's MSL backend (or Apple's MSL compiler).
The prefilter runs once per candidate, so pairing / with % here costs
nothing measurable; the scan keeps its mul-sub form and its measured
gains. One more entry for the naga-on-Metal ledger next to the
select(ulong, uint, bool) division-guard bug.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Flush each detailed batch as its own submission (watchdog fix, from review)
Applied from Janzert's diagnosis on #99, with the regression test:
CubeCL aggregates dispatches into one command buffer and submits at
CUBECL_WGPU_MAX_TASKS (32) or the next read — so one submission held a
whole field, and GPU watchdogs (amdgpu ~2s, Windows TDR 2s) reset the
device on any GPU slow enough that a field outruns them: every detailed
field past base ~46 wedged an AMD 860M on its first claim. flush()
submits without waiting, so the drain pipelining is unchanged; measured
free on the reporting machine. Niceonly is safe incidentally (its
per-dispatch buffer writes flush the stream) — now documented at that
site so pooling those buffers doesn't quietly inherit the bug.
Also from the same review pass, two defects in b825b2e's conservation
check: a stray duplicate of the ensure inside the drain branch compared
a partially-drained histogram against the whole field (fired at 65+
batches; a replace-all editing accident), and the failure hint said
'install the CUDA toolkit' regardless of runtime — it now names the
watchdog on wgpu.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Janzert <janzert@janzert.com>
* Auto picks the measured-fastest backend order per mode
Detailed: cubecl-cuda -> cubecl -> cuda -> vulkan. The CubeCL family
wins detailed on every vendor tested (1.03-1.07x over hand-CUDA on
NVIDIA, 1.2-2.6x over hand-Vulkan elsewhere), and since b825b2e a
toolkit-less cubecl-cuda init fails cleanly, so NVIDIA-without-toolkit
falls through to wgpu speed instead of the hand-WGSL fallback.
Niceonly: cuda -> cubecl -> vulkan. Hand-CUDA keeps a slim edge at the
b50+ bases where long-run wall time concentrates (0.91-0.97 for
cubecl-cuda); everywhere without a toolkit CubeCL is the best
available: it wins RADV b50+ (1.28-1.38), wins NVIDIA-over-wgpu
outright, and runs out of the box on Apple. The known concession is
Apple niceonly via MoltenVK (0.55-0.79), which needs optional brew
packages and stays reachable with --gpu-backend vulkan.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Unify wasm32 dependency features so the cubecl graph builds on wasm again
The evaluation's wasm build fixes lived only in the throwaway probe
crate; at head, cargo check -p nice_common --features cubecl --target
wasm32-unknown-unknown failed in getrandom (no entropy backend named).
Target-gated dependencies activate getrandom's wasm_js backend and
cubecl-common's serde feature on wasm builds, tied to the cubecl
feature; native builds are unaffected. First step of readying the core
for the browser/WebGPU client.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Ready the core for the browser client: async init/reads, wasm-safe timing
Three changes so the coming wasm/WebGPU PR does not have to touch this
backend again (its remaining work is the u32-only kernel variant, which
can live in its own module, plus wasm-client plumbing):
- CubeclContext::new_default_async over init_setup_async — the only
form a browser permits (the sync init_setup panics on wasm by
design). Both constructors share one process-wide client.
- process_range_detailed_cubecl_async: the detailed path is now an
async body using read_async (a wasm future cannot block); the sync
entry is that future under block_on, so native device tests exercise
the same code. Niceonly stays sync — its MSD thread pipeline is not
browser scope.
- Instant now comes from web-time, which re-exports std on native and
works on wasm32, where std::time::Instant panics at runtime.
Also --gpu-wgpu-device (env NICE_GPU_WGPU_DEVICE), from review item 3:
--gpu-device indexes per-backend namespaces (CUDA ordinals != Vulkan
ordinals != wgpu adapters), so on a multi-GPU box no single number is
right for every backend. The flag takes CubeCL's own device spelling
(DiscreteGpu(0), IntegratedGpu(1), Cpu, ...) and is translated to the
CUBECL_WGPU_DEFAULT_DEVICE env var before init, keeping one parser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* README: GPU acceleration section and a current --help dump
One place covering the GPU path: the four backends and what each needs
at runtime per platform, auto's per-mode selection order and its
explicit-is-fatal rule, multi-GPU selection (--gpu-device's per-backend
namespaces and the typed --gpu-wgpu-device), and the troubleshooting
answers for the failure modes we have actually seen — missing NVRTC,
watchdog resets, MoltenVK setup, wrong laptop adapter, CPU fallback
bases. The embedded --help output was three backends out of date;
regenerated from the umbrella build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Demote the hand-Vulkan backend to an experimental opt-in feature
The gpu umbrella no longer includes vulkan: every platform it serves
is covered by cubecl, which beats it in detailed mode on all vendors
measured, so the standard GPU binary and the -gpu docker image drop the
hand-WGSL backend and its ash/naga dependencies. The feature stays in
the tree, marked experimental, for --features gpu,vulkan builds — to be
promoted or removed once its remaining niceonly niches (Metal via
MoltenVK, RADV at prefilter-heavy bases) stop mattering or get closed.
No user-facing change otherwise: auto's order already ends with vulkan
only in builds that include it, and CI's device-parity jobs name their
features explicitly and still cover it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Brian <janzert@janzert.com>
Co-authored-by: Claude <claude@example.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This isn't ready to merge, but I wanted to get it here to see if there is interest in adding a Vulkan backend at all. The Vulkan backend doesn't match the performance of the CUDA implementation, but it does open up the use of a much broader range of hardware. I believe most modern consumer hardware should work with it.
This was written pretty much wholly by Claude and I have done practically zero GPU programming so I'm not going to be able to do much useful review of the code here.
If there is interest in adding this, it still needs a few things before merging. Probably most importantly, broader testing and optimization passes on more hardware. Also, as mentioned by Claude below, the feature flags should probably be reworked in some way. Unlike Claude though, I lean towards what it calls option C (rename the current
gpuflag tocudaand makegpuan umbrella flag for both backends). It doesn't seem like there are that many uses or problems caused by simply reassigning the name, and I would almost be tempted to just keep a singlegpufeature flag and always build both backends.What follows is Claude's summary for the PR (with some editing by myself).
Add a Vulkan compute backend alongside CUDA
Summary
The CUDA path is NVIDIA-only and needs the CUDA toolkit present at runtime for
NVRTC. This branch adds a second GPU backend that runs on any Vulkan 1.2 device
with
shaderInt64— AMD, Intel, NVIDIA, and llvmpipe — selected at runtime with--gpu-backend {auto,cuda,vulkan}.Both
cudarcandashdlopentheir driver, so one binary can carry bothbackends and require neither library at build time.
autotries CUDA first, sobehaviour on an NVIDIA machine is unchanged. An explicitly named backend that
fails to initialize is fatal rather than silently falling back — for a
distributed compute client, quietly dropping to a much slower path is worse than
stopping and saying so.
Both modes are ported: detailed and niceonly. CPU/GPU parity is exact, and
because the Vulkan tests also pass against lavapipe, correctness is now
checkable in CI with no GPU at all — which the CUDA parity tests cannot do.
What moved on the CUDA side, and why
Two shared modules were extracted rather than duplicated.
client_process_gpu.rsis
#![cfg(feature = "gpu")]and therefore unreachable from a Vulkan build, so:common/src/gpu_config.rs— the backend-neutral per-base constants.common/src/gpu_niceonly.rs— the niceonly host pipeline: the adaptiveMSD floor, the worker/channel/batching loop, the per-field stats. Backends plug
in a
RangeSink(launch+sync); everything above that is written once.The CUDA path was moved onto this pipeline, not copied from it. The kernel and
detailed mode are byte-identical, but CUDA niceonly now runs the shared host loop.
Two consequences worth flagging for review:
PIPELINE_DEPTH(56fba6ecorrects the comment that stated its memory boundin the wrong unit — the real cap is ~3 MB, not the 50 MB the old arithmetic
implied).
msd_secsis redefined, and it feedsAdaptiveFloor.The CUDA path keeps every one of its existing tests, and it has been verified
live against the production API on an RTX 4060. Telemetry reported
NVIDIA GeForce RTX 4060under both backends on a machine that also has an iGPU.Detailed mode has separately been run live against the production API on every
device in the table below, at the current head of this branch.
Short runs rather than a soak test, but real claims and real submissions on all
five GPUs.
I'd still take a second pair of eyes on the
PIPELINE_DEPTHbound and themsd_secsredefinition, since those are the two behaviour changes CUDA usersinherit and neither is the sort of thing a short run surfaces.
Measurements
Vulkan vs CPU on an integrated GPU
An integrated AMD Radeon 860M against the same machine's 16-thread CPU, via
nice_client --benchmark— median of 3 interleaved sweeps per arm at a 30 sbudget, CPU arm pinned to
--threads 16, measured at branch head, i.e. afterall three upstream merges.
b40_detailedb50_detailedb40_msd_weak(niceonly)b50_msd_weak(niceonly)b50_residue_dense(niceonly)b52_msd_weak(niceonly)Two things to read carefully here.
b40_msd_strongis omitted, because it is not a device comparison. It readsCPU 1.90e12 against GPU 6.33e10 — the CPU apparently 30× ahead — and upstream's
own
SCORE_REFERENCESshow the same inversion (1.0e12 CPU against 2.3e11 GPU).The cause looks structural rather than real: that scenario takes
start: None,so both arms begin at the band start, but
window_gpuis 8e9 againstwindow_cpu's 1e8. In a region defined by the MSD filter being strong, an 80×longer window runs 80× further down the band and out of the strongly-filtered
head, so the two arms are characterising different stretches. The other niceonly
scenarios pin an explicit
startin uniformly msd-weak territory, and detailedmode does no MSD filtering at all, so neither is affected.
These are per-region rates, not a whole-field speedup. A production field
mixes strongly- and weakly-filtered stretches, so its end-to-end ratio sits
below the msd-weak rows above. Where the GPU wins is where there is work to do;
where the MSD filter has already thrown the work away, it has nothing to win.
Vulkan vs CUDA on the same GPU
Both backends on one RTX 4060, via
nice_client --benchmarkso the measurementwindows are identical work. Median of 3 interleaved sweeps at a 120 s budget.
Run-to-run spread was under 1% on both arms, and a 30 s budget gave the same
ratios to within 2%.
b40_msd_strong(niceonly)b40_msd_weak(niceonly)b50_residue_dense(niceonly)b50_msd_weak(niceonly)b52_msd_weak(niceonly)b40_detailedb50_detailedCUDA remains the right default on NVIDIA, and
autopicks it. The point ofthis backend is the hardware CUDA cannot reach at all; that it lands within 12-14%
on NVIDIA's own turf at base 40 is a bonus rather than the goal.
The spread across scenarios is not noise — it is the
split16workaround, andit is quantified. The Vulkan codegen calls
chunk_constants_u16unconditionally, capping the digit-scan chunk at 2^16 because RADV
strength-reduces only 32-bit constant division; CUDA uses
chunk_constantsat2^31. That is 3 digits per chunk against 5 at base 40, and 2 against 5 at bases
50 and 52. The base-40 niceonly rows stay near parity for the other half of
the reason: the modular prefilter is enabled at
base <= GPU_PREFILTER_MAX_BASE(40), so ~99% of candidates never reach the scan. Detailed mode scans every
candidate and is uniformly worst.
So the gap is one term, in one loop, with a known fix: choose the chunk width
per device rather than hardcoding the RADV-safe one.
chunk_constants_belowalready takes the limit as a parameter and shaders are generated at runtime, so
this is a small change — but it rests on NVIDIA's SPIR-V compiler
strength-reducing the 64-bit form the way ptxas does. That has yet to be
measured and verified though.
Tested on
AMD Radeon(TM) Graphics1Intel(R) Graphics1Names are as the device reports them — i.e. what the client sends as
gpu_model.Three vendors, integrated and discrete, plus a software rasterizer. The two
NVIDIA rows matter most for review: they are the check that adding a second
backend did not disturb the first, and the RTX 4060 ran both backends on the
same machine.
Every device in the table has processed and submitted real fields against the
production API, not just passed the parity tests — short live runs in detailed
mode at the current head of this branch. Niceonly was additionally exercised
live on the RTX 4060 through both backends.
Testing
vulkan_matches_cpu_detailedandvulkan_matches_cpu_niceonlycheck exactparity (distribution and near-miss list) at bases 10, 40, 62 and 80.
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json cargo test -p nice_common --features vulkan --release --lib -- --ignored --nocapturejust testnow builds all three GPU feature combinations (gpu,vulkan,gpu,vulkan).Three things worth knowing, in case they save someone time
cudarcpanics rather than returning whenlibcuda.sois absent(
panic_no_lib_foundreturns!). That is the ordinary case on any machine withno NVIDIA driver — precisely the case
autoexists to fall through — so theclient died before the fallback could run.
try_init_cudawraps init incatch_unwindwith the panic hook silenced. You already catch the same panic forthe same reason in
nvrtc_compiles_kernels_for_all_supported_bases. This relieson unwinding, so
panic = "abort"in the workspace profile would break it.RADV does not strength-reduce 64-bit constant division, only 32-bit. nvcc
turns
u64 / CHUNK_DIVinto a multiply-high; ACO expands it into a~220-instruction restoring shift-subtract loop, because NIR's
nir_opt_idiv_constis width-limited. So the CUDA kernel's "no division in thehot path" premise does not port for free. Keeping
CHUNK_DIVunder 2^16 andsplitting into two 32-bit divisions over 16-bit halves is exact and worth 2.85×
on base 40, despite needing ~1.7× more chunk iterations. This bit three times —
the digit scan,
n mod M, and the whole prefilter, where the fix was to stopholding the residue in a u64 at all.
CUDA's warp width was carried over as a constant 32 lanes and was wrong here.
Nothing in this kernel is a warp — lanes stride a range's candidates by index and
never communicate — but they do all repeat the range setup, the residue reduction
and a ~12-iteration binary search. At MSD floor 250 a base-40 range holds ~39
candidates, so 32 lanes bought 32 copies of that setup to share out 1.2 candidates
each. Device time by pinned width: 30.0 s at 32 lanes, 21.1 s at 1–2, monotone
where ranges are short and flat within noise elsewhere.
Proposal: rename
gpu→cuda, add an umbrella featureNot done on this branch — this PR adds
vulkanalongsidegpuand changes noexisting name. Raising it separately because it touches your release path and is
your call.
The problem is that
gpunow names one backend out of two. It reads as"GPU support" but means CUDA specifically, and the file layout already shows the
strain:
gpu_config.rsandgpu_niceonly.rsare genuinely backend-neutral, whileclient_process_gpu.rsis CUDA-only, and one feature name covers both.Three ways to fix it, in increasing order of blast radius:
A. Umbrella only — one added line, nothing changes meaning.
Gets
--features gpu-allfor the both-backends build. Does not fix the naming.B. Rename with an alias — the one I'd suggest.
cudais the honest name;gpukeeps its current meaning byte-for-byte, so.github/workflows/docker-build-push.ymland the threecargo test --features gpulines in the README are unaffected. Costs one line over (C) and nothingsilently changes.
C. Rename with
gpuas the umbrella —cuda,vulkan,gpu = ["cuda", "vulkan"]. Cleanest end state, but--features gpukeeps working whileproducing a different binary, with no diagnostic from cargo. The sharpest case
is
docker-build-push.yml:110, which builds the published CUDA client image with--features gpu: it would start linkingash+naga, andautoinside acontainer with no NVIDIA driver would fall through to Vulkan instead of failing.
Same for the README's CUDA test instructions, which would then also compile the
Vulkan tree.
Either rename also creates pressure to rename
client_process_gpu.rs→client_process_cuda.rs(to pair withclient_process_vulkan.rs) whilegpu_config.rs/gpu_niceonly.rscorrectly keep thegpuname. Right thing todo; also the kind of file rename that conflicts with everything.
One thing to note whichever way this goes: cargo features are additive, so an
umbrella means any crate in the graph enabling it forces
vulkanon for the wholebuild. Harmless in this workspace today, but it makes accidental over-enabling
reachable where it currently is not.
The runtime surface —
--gpu,NICE_GPU,--gpu-backend,NICE_GPU_BACKEND—is already backend-neutral and would not change under any of these.
Any of these can be done as a follow-up commit on this branch, or the names can
be left alone entirely.
Not included
gpu_modelis less identifying on Vulkan than on CUDA, and the estimatorshould probably know that. Vulkan's
VkPhysicalDeviceProperties::deviceNameis whatever the driver feels like: RADV on discrete parts gives
AMD Radeon 860M Graphics (RADV GFX1152), but the Windows AMD and Inteldrivers give
AMD Radeon(TM) GraphicsandIntel(R) Graphicsfor any recentintegrated part. Those normalize to one bucket across unrelated silicon, so an
estimate built from them would average across SKUs. I have not touched
estimator.rs— the options I can see are to record the backend alongside themodel and refuse to match generic Vulkan names, or to enrich the reported name
with
deviceID/vendorID, which are available and unambiguous. Either couldbe added here, but it is a schema question and so yours to call.
Dispatch is asynchronous, which was the last big win; what is left after it is
small. "Asynchronous" here means the host no longer submits a batch of range
descriptors and then blocks until the device drains it.
VulkanContextholds aring of
SUBMISSION_SLOTS = 2command buffers, each with its own fence and itsown descriptor buffer, handed out round-robin; the host fills slot i+1 while
the device is still reading slot i, and blocks only when it comes back to a
slot whose fence has not signalled. Two slots is the whole win — one batch in
flight against one being filled is all the overlap available, and a third only
adds latency to
sync. This is also what madeAdaptiveFloorconverge: itsmsd == gpuobjective minimizes a maximum, which is the right target only oncethe two phases actually overlap.
Deliberately left, all measured-small or measured-flat:
VULKAN_PREFILTER_LIMBSis 3, which alreadymatches CUDA's digit count at every base that enables the prefilter. A fourth
would reject more candidates at ten mul-adds instead of six. Unswept.
split16item above.PIPELINE_DEPTHandAdaptiveFloor's ±1.5× step, which is the ~5%residual described below.
very short and very long ranges is tiled for neither. Bucketing a batch by
length would fix it; not tried. The MSD recursion bounds range length by the
floor, which is why the mean is a decent summary in practice.
Nothing has been tuned for a discrete card. Every constant here was chosen on an
iGPU that shares DDR5 with the CPU and pays nothing for host→device transfer;
a discrete card inverts all three of those, and I'd expect the lane width, the
batch size and the MSD floor all to want re-measuring there.
AdaptiveFloor's512000/coresseed is a long way from optimal on both machinesmeasured — it starts at 32000 where the iGPU wants ~2000 — but the seed is
deliberately left alone, because it is a startup transient rather than a
standing cost. The loop walks down to a limit cycle in roughly ten fields and
settles within ~5% of the best fixed floor; the residual is the hard ±1.5× step
size overshooting, not the target. Against a client that runs for hours or days
that is not worth tuning, and a seed dialed in for one machine would be wrong on the
next one anyway. (before async dispatch the loop was optimizing a maximum across
phases that did not overlap, and it settled 2.3× off rather than 5%.)
Footnotes
These two are verbatim, and that is a problem. Some Vulkan drivers
report a generic marketing string where CUDA reports a specific one.
AMD Radeon(TM) Graphicsis what every recent AMD integrated part callsitself, and
Intel(R) Graphicslikewise — neither identifies the silicon.Since
normalize_gpu_modelonly strips vendor tokens andgpu_models_matchis an exact token match, these collapse into a single estimator bucket
spanning many distinct SKUs. That is the merging Canonicalize GPU model names across Vast and CUDA naming #95 exists to prevent,
reached from the device side rather than the listing side. See "Not included"
below. ↩ ↩2