Fix CUDA resume and optimize training throughput

[ferdinandobons]

What changed

• Fix CUDA checkpoint resume by moving mapped RNG states back to CPU ByteTensors before restoration.
• Add an opt-in fused multi-head attention path with one combined QKV projection and one batched SDPA call per Transformer block.
• Vectorize memmapped batch extraction and use pinned, non-blocking CUDA transfers.
• Add lightweight --log-interval progress output independent of validation and checkpoint cadence.
• Preserve legacy checkpoint behavior by defaulting missing fused-attention configuration to the original separate-head implementation.
• Keep the Lesson 42 snapshot synchronized with the final project.
• Add CUDA_TRAINING_OPTIMIZATIONS.md with concepts, measurements, tradeoffs, and production commands.

Why

CUDA resume failed because torch.load(..., map_location="cuda") moved RNG-state tensors to CUDA while torch.cuda.set_rng_state_all requires CPU ByteTensors.

The educational attention implementation also launched Q, K, V, and SDPA separately for every head. On the measured 12-block, 12-head model this caused 432 QKV projections and 144 attention calls per micro-batch. Fusing heads preserves the model computation while substantially reducing Python and CUDA launch overhead.

Separating logging from evaluation gives frequent progress visibility without running validation, context diagnostics, and multi-gigabyte checkpoint writes every 100 steps.

Impact

• Existing checkpoints continue to load with the original attention layout.
• New runs can opt into --fused-attention.
• CUDA resume preserves model, optimizer, GradScaler, and RNG state.
• On the local RTX 4060 benchmark, fused batch 8 reached 20,312 tokens/s versus 3,740 tokens/s for the best separate-head batch 8 run.
• The effective batch remains 8,192 token positions per optimizer step and model parameter count is unchanged.

Validation

• python -B tools/validate_learngpt.py
• python -B -m unittest discover -s tests -v
• 37 tests passed.
• Fused and separate attention output parity verified after weight mapping.
• Equal parameter counts verified.
• Fresh CUDA and resume smoke runs completed with zero AMP overflows.
• Batch 8 and batch 16 were benchmarked; batch 8 was faster and retained safer VRAM headroom.