feat: add Dr.GRPO loss normalization

[0z5a]

Summary

• add an opt-in dr_grpo loss variant
• remove group reward standard-deviation normalization in that variant
• normalize policy loss by the fixed batch_size * group_size * max_completion_length budget
• derive the online fixed budget from rollout_max_tokens when it is not set explicitly
• keep the existing grpo objective and KL regularizer behavior unchanged

Why

Dr.GRPO identifies two normalization choices that can bias R1-Zero-style training: reward scaling by the within-group standard deviation and response-dependent policy-loss normalization. This change exposes both corrections as one explicit objective variant while preserving the current default exactly.

For a paper-faithful no-KL recipe, use loss_variant="dr_grpo", a fixed max_completion_length, and kl_coef=0. Offline GRPO requires the fixed completion budget explicitly; online GRPO defaults it to rollout_max_tokens.

Primary reference: Understanding R1-Zero-Like Training: A Critical Perspective.

L20 microbenchmark

Exact commit: ffa8520de2c6ac0494702e5eb6d7c9bb33d30536

NVIDIA L20, FP32 formula path, B=256, G=8, response tensor length 512, fixed budget 1024, 1,048,576 response slots and 785,863 valid tokens. Twenty alternating samples, 200 iterations per sample, CUDA-event timing after warmup.

objective	median	p90
standard GRPO	0.128371 ms	0.133325 ms
Dr.GRPO	0.082757 ms	0.090342 ms

Dr.GRPO was 35.53% faster in this isolated formula microbenchmark because it removes the reward-std and dynamic token-count reductions. This is not an end-to-end training throughput claim; model forward/backward still dominate real training.

Validation

• python -m pytest -q: 658 passed on NVIDIA L20 GPU5
• focused GRPO, CLI, and online rollout suite: 19 passed
• Ruff format and lint checks passed
• tests cover default formula compatibility, exact Dr.GRPO numerics with variable masks, fixed-budget validation, invalid configuration, and online budget derivation

Compatibility

The default remains loss_variant="grpo"; existing configs and loss numerics are unchanged.