nnue_train: accelerate dense QAT integer forward - #415
Merged
Merged
Conversation
RTX 5090 Windows native CUDA, fixed real-data updates excluding loader/save: 3-run mean 1970266 -> 3355736 pos/s; medians 1970577 -> 3352014 pos/s. Forward raw i32/final f32 bits match on 262144 positions, three runs per variant. Loss trajectories differ slightly; reference repeats also differ. No full-state bit claim. Reproduce with recorded real_training_fixture.rs in the test module: QAT_SOURCE_RAW=/path/to/1000sb.ckpt QAT_PROGRESS=/path/to/progress.bin QAT_TEACHER=/path/to/teacher.psv QAT_BENCH_OUTPUT=/path/to/result release test: trainer_layerstack::qat_impl::tests::qat_real_training_measurement --ignored --exact --test-threads=1 CUDA source replaces llround conversions with float2int_rn and a warp shuffle reduction; PTX instruction counts were not measured. Native tests 171+2, clippy and fmt pass; independent review APPROVE. Full CI blocked on existing Windows cuda-oxide binding type mismatches; this local commit must not be pushed until full CI passes on Linux.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dense QAT の整数 forward が各出力を1 threadで逐次計算していたため、1536幅の内積が学習時間を占めていた。256幅以上を1出力1warpで計算し、量子化済み入力・重みの整数復元を f32 intrinsic に置き換える。bias の丸め、整数積算結果、STE・backward・optimizer・学習レシピは維持する。
起動 thread 数は checked multiplication と u32 検査を行い、対応範囲を超える場合はエラーにする。全格子値の整数復元と起動数 overflow の回帰テストを追加した。
RTX 5090 / Windows native CUDA / 1536×16×32 / batch 65536、同じ1000sb raw・実教師4 batchを使った固定60更新の3組比較で、中央値は 1,970,577 → 3,352,014 pos/s（1.70倍）。loader・checkpoint保存・exportを除く固定更新区間であり、本格学習全体の速度を保証する数値ではない。
更新前の同一状態・実教師262,144局面を各実装3回比較し、dense 3層の raw i32 と最終 f32 出力は bit 一致。合成入力のカーネル比較でも一致。学習後の raw/export 全体は基準実装の反復同士でも一致しないため、学習の bit 再現性は保証しない。
再現条件: 実験ノートの
real_training_fixture.rsを test module に追加した release test binary に、QAT_SOURCE_RAW=/path/to/1000sb.ckpt QAT_PROGRESS=/path/to/progress.bin QAT_TEACHER=/path/to/teacher.psv QAT_BENCH_OUTPUT=/path/to/resultを渡し、trainer_layerstack::qat_impl::tests::qat_real_training_measurement --ignored --exact --test-threads=1を実行する。run_real_fixture.pyはこのビルドと3組の実行をまとめたローカル用スクリプト。fixture と全測定記録は実験ノートに保存している。検証: native nnue-trainer 171 unit + 2 integration tests、native全target clippy、fmtはPASS。独立エージェントによる最終コードレビューはAPPROVE。
フルlocal-ciはWindowsの依存cuda-coreで生成bindingsのi32/u32型不一致10件によりexit101。Linux GPUホストは別学習中のため投入していない。ユーザーの明示承認により、この変更ではフルlocal-ci待ちを解除してPRを作成し、GitHub CIを確認する。GitHub CIはGPU依存crateを除外しており、native実機検証と分けて扱う。