Asynchronous dispatch / op batching on Metal and CUDA: per-op launch cost dominates small workloads

[vyncint]

Measured (oxmega device-bench, M4 Pro, 100 epochs, 1,356 rows × 45)

workload	CPU	Metal
mlp-h16 BCE, 1 seed	0.30 s	1.77 s
mlp-h16 BCE, 3 stacked seeds	0.54 s	2.16 s
mlp-h16 set-NLL, 3 seeds	0.82 s	6.5 s
linear-lag5 set-NLL, 3 seeds	0.86 s	6.9 s

Batching three seeds into one graph cut Metal's per-op count 3× and it is still 4–8× slower than the CPU: every op is a synchronous command buffer (wait_until_completed per dispatch) and every AdamW parameter update is several tiny ops. CUDA has the same shape (one stream, synchronous downloads).

Proposal

• Encode many ops into one command buffer / stream and synchronise only at to_device(Cpu)/to_vec (lazy completion), keeping the current per-op semantics as the observable contract.
• Fused optimizer step kernels (AdamW over a parameter list in one launch).
• Measure with the table above; the device-bench in vyncint/oxmega is a ready-made benchmark.

[vyncint]

Shipped in oxmera 0.3.0 (merge 0683ca0, PR #31; all twelve crates on crates.io): asynchronous Metal dispatch (encode into an open command buffer, commit every 64 ops or at a host read, wait on every outstanding buffer) and a fused adam_step kernel on Metal and CUDA. Measured on your device-bench (M4 Pro, back-to-back with 0.2.0, three rounds): Metal 19.4–20.1 s → 7.4–7.6 s; CPU 6.0–6.4 s — within 1.2× of the CPU, not faster. ADR-0008 records the two races found on the way. Details and measurements in the 0.3.0 changelog.