perf: reuse rollout behavior logprobs

[0z5a]

Summary

• feed sampler-aligned logprobs_old from rollout results directly into online GRPO
• avoid allocating, synchronizing, and forwarding a duplicate old-policy model in the online path
• preserve the synchronized old_model fallback for offline GRPO batches
• validate behavior-logprob shape and finiteness before loss computation
• document the online and offline behavior-policy paths

L20 benchmark

Exact PR commit 2b3a5f0, NVIDIA L20 (SM89), BF16, PyTorch 2.11.0+cu128, CUDA 12.8. The checkpoint is the real AstrAI 1.2B configuration (24 layers, hidden 1536, FFN 6912, vocabulary 100k). The microbenchmark uses four 128-token sequences with 64 response tokens; model timings use 3 warmups and 10 synchronized samples, while the reuse path uses 20 warmups and 100 samples.

online-GRPO behavior source	resident CUDA cost	median step work	p90	peak transient
duplicate old-policy model + logprob forward	2.246 GiB	14.540 ms	14.650 ms	0.475 GiB
rollout logprobs_old reuse, including shape/finite validation	0 GiB	0.0655 ms	0.0714 ms	0.000002 GiB

For this batch, reuse removes one 2.246 GiB model replica and makes the behavior-logprob part of the training step about 222x faster. This is intentionally scoped to online GRPO: offline batches without captured rollout logprobs still use the old-policy model compatibility path.

Validation

• PYTHONPATH=$PWD CUDA_VISIBLE_DEVICES=5 bash scripts/pre_commit.sh --skip-deps
• 654 tests passed
• ruff format and import checks passed
• online GRPO/DPO end-to-end tests verify only one frozen reference model is created

The full GPU suite used the existing compiled CUDA extension modules from the L20 checkout, linked into the isolated test worktree.