nnue_train: FT混合精度のsampled数値診断を追加

[SH11235]

FT moment checkpoints already contain rounded values, so they cannot distinguish storage zeroing, caps, and rounded-away updates. This adds default-OFF native CUDA LayerStack diagnostics at explicit absolute optimizer steps, using a fixed-seed uniform FT element sample split into base and virtual rows.

The JSON records report pre-cap versus stored m/v values, finite/nonzero denominators, zeroing, cap, round-to-same and unscaled errors. Master changes are separated into norm loss, optimizer, and complete-step updates; forward-copy changes use the actual buffer after Lookahead and factorizer folding. Ordinary optimizer expressions, precision defaults, training order, and existing dFT clamp metrics are preserved. cuda-oxide, Simple, dFT distribution, QAT and provenance are explicitly outside this diagnostic's support.

On selected steps an observing specialization shares the native update helper; ordinary steps retain their existing entry points. No extra full-FT scan or ordinary-step readback is added. Selected optimizer steps perform a sample-index binary search during the existing FT traversal. Setup has one context synchronization; per-step readback is 52 bytes per sampled element. See docs/precision-diagnostics.md for exact sampling, scopes, denominators, and limitations.

Validation completed on sh11235-ws (RTX 5090), Windows native, native CUDA, driver 596.36, CUDA Toolkit 12.9, pinned nightly-2026-04-03:

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets -- -D warnings
• cargo test --workspace --release -- --test-threads=1
• CPU-only precision CLI rejection test
• Synthetic CUDA zero/cap/round-to-same and exact ON/OFF optimizer bits across FP32/FP16 moments, mirror ON/OFF, denominator ON/OFF, decay, full/sparse samples
• Multi-step trainer ON/OFF and OFF/OFF comparisons with existing native equivalence tolerances, including norm loss, Lookahead and factorizer

Kernel-level bit equality uses identical gradient buffers. Independent full trainer runs also show bit differences in the OFF/OFF control, so full trainer comparisons use the existing numerical tolerances. No throughput improvement or overhead bound is claimed.

Integration: cli.rs, training.rs, trainer_layerstack.rs and native_kernels.cu may overlap concurrent QAT/provenance branches. Preserve both the normal update helper and the diagnostic record layout (13 floats), pre-norm-loss snapshot, and post-Lookahead/fold snapshot. The new JSON log is independent of checkpoint/experiment schemas.

Full gate passed: bash scripts/local-ci.sh (exit 0 / PASS) on sh11235-ws, WSL2 Ubuntu 24.04, RTX 5090, CUDA Toolkit 12.9.86, driver 596.36. This includes native / oxide / oxide-parity compile and lint, rebuilt cuda-oxide kernels, backend parity, cuda-oxide GPU/CPU equivalence tests, and release tests for the whole workspace. A shared GPU validation lock serialized this run after the QAT session. Independent Codex and Claude reviews both approved the final implementation. No merge, full training campaign, or matches were started.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-13T08:42:13.165475Z	f589a14	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.