feat(experimental): add DAPO training support

[CharlesXu-HQ]

What does this PR do?

Adds opt-in, experimental DAPO training support, following the DAPO paper and the maintained verl DAPO recipe.

The implementation includes:

• asymmetric Clip-Higher policy clipping
• dynamic sampling that collects complete informative prompt groups
• response-token-mean policy-gradient normalization across uneven microbatches, gradient accumulation, and data parallelism
• soft overlong reward shaping
• rollout-policy logprobs as the old-policy reference, so importance ratios can evolve across optimizer updates
• additive experimental registry, CLI, and documentation surfaces; existing stable algorithms remain unchanged
• data-integrity checks for rollout token/logprob alignment, finite old logprobs, exact n_samples, and reward/group cardinality

Latest-main integration

The branch is rebased onto upstream main at 01b2321 and adapted to the new backend architecture introduced by #493:

• DAPO now consumes the canonical packed CUDA loss layout
• response-token normalization hooks live in areno.api.backend.cuda.training
• DAPO metrics use the shared backend metric-reduction contract
• all current common trainer options are propagated into DAPOTrainerConfig
• the CLI rejects DAPO with MLX before initialization because the experimental loss currently has a CUDA/Torch implementation only

Related issue

Fixes #485

A follow-up numerical clarification is documented in this issue comment: AReno's current on-policy surrogate has forward value exp(new - new.detach()) = 1, which leaves Clip-Higher numerically inactive. This PR therefore keeps the real rollout-policy ratio scoped to experimental DAPO while leaving stable GRPO/GSPO behavior unchanged.

Type of change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change (public API / CLI behavior changes in a non-backward-compatible way)
• 📝 Documentation update
• ♻️ Refactoring
• ⚡ Performance improvement
• ✅ Test coverage improvement

How was it tested?

Fresh validation for rebased head 790995e:

• python -m pytest tests/test_dapo_cpu.py -q — 27 passed
• CUDA_VISIBLE_DEVICES= python -m pytest tests/test_train_cli_config_cpu.py -q -k dapo — 11 passed, 99 deselected
• ruff check areno/api/algorithms.py areno/api/backend/cuda/backend.py areno/api/backend/cuda/training.py areno/cli/train.py areno/experimental/dapo tests/test_dapo_cpu.py tests/test_train_cli_config_cpu.py — passed
• ruff format --check ... on the same changed Python paths — passed
• python -m compileall -q areno/experimental/dapo areno/api/backend/cuda areno/cli/train.py — passed
• python -m sphinx -W --keep-going -b html -D html_copy_source=0 docs <output> — 46 documents built without warnings
• TORCH_CUDA_ARCH_LIST=12.0 MAX_JOBS=2 python setup.py build_ext --inplace — all 12 extension units compiled and linked for sm_120
• git diff --check — passed

Algorithm and data correctness coverage includes:

• hand-derived Clip-Higher loss, clip fractions, and analytical gradients across all clipped/unclipped sign branches
• central finite-difference gradient agreement
• exact rollout → reward record → overlong shaping → group advantage → TrainSequence → canonical packed action tensor field alignment
• inactive packed-token masking equivalence
• real split_data_pack_by_dp loss/gradient equivalence for an uneven 3-to-2 split
• combined gradient-accumulation and DP equivalence, including a microbatch smaller than the DP world that is replicated by the runtime
• real multi-prompt scoring order through filtering, accepted-group selection, and whole-group overflow truncation
• exact n_samples enforcement before prompt-group scoring
• mutation checks confirming the DP scale, prompt index, hand-derived Clip-Higher, and packed-data alignment tests fail when their respective production paths are intentionally corrupted

The full CPU suite is designed for the CPU-only GitHub matrix. In the available CUDA PyTorch environment, installing Triton makes three untouched upstream MoE CPU tests run instead of skip; they fail because their SimpleNamespace fixtures do not contain the newly required routing_layer_slot. With CUDA hidden, the run reached 586 passed and 8 skipped before those 3 unrelated failures, then was stopped at a blocked Hugging Face download. None of the reported failing files are changed by this PR.

Real-model validation

Environment: Python 3.12.13, PyTorch 2.11.0+cu128, CUDA 12.8, one NVIDIA GeForce RTX 5090. Exact source archive SHA-256: 529f91bb4c6b5d7b3ee7e26ef45f2d4f2ca8fd27c7d97779cf2efc25529ff15e (790995e).

The following bounded Qwen3-0.6B run deliberately makes the first prompt group uninformative and the second informative, so it exercises filter → retry → train rather than only a direct training smoke:

{"prompt":"Continue this short sequence: 1, 2,"}
{"prompt":"Name one primary color:"}

def reward_fn(record):
    prompt = record.source_record.get("prompt", "")
    if prompt.startswith("Continue this short sequence"):
        return 1.0
    return float(record.metadata["sample_index"])

ARENO_LOG_COMPLETIONS=1 python -m areno.cli.main train \
  --algo dapo \
  --ckpt Qwen/Qwen3-0.6B \
  --model-hub modelscope \
  --dataset-path <validation.jsonl> \
  --reward-fn-path <reward.py> \
  --epochs 1 \
  --max-steps 1 \
  --world-size 1 \
  --tp-size 1 \
  --train-devices 0 \
  --batch-size 1 \
  --dapo-gen-batch-size 1 \
  --dapo-max-num-gen-batches 2 \
  --n-samples 2 \
  --mini-bs 1 \
  --gradient-accumulation-steps 1 \
  --max-prompt-tokens 64 \
  --max-new-tokens 4 \
  --temperature 1.0 \
  --attn-backend native \
  --eager-decode \
  --no-activation-checkpointing

Observed result (exit code 0):

• after candidate 1: gen_batches=1 generated_groups=1 qualified_groups=0 filtered_groups=1
• after candidate 2: gen_batches=2 generated_groups=2 qualified_groups=1 filtered_groups=1
• final sampling metrics: dapo_sampling_efficiency=0.5, dapo_discarded_qualified_groups=0
• finite training metrics: loss=1.4305115e-06, grad_norm=0.6675834
• real ratio metrics: ratio_mean=1.0000181, ratio_std=2.9320003e-05
• logged policy values: rollout_logprobs_mean=-0.00108719, train_logprobs_mean=-0.00107050

The real-model run is a bounded integration test, not a convergence or benchmark reproduction. I did not run a paper-scale AIME experiment or a multi-GPU end-to-end training job; distributed response-token normalization is covered by hand-derived CPU references and tests that invoke the runtime's real DP split function.

Checklist

• The PR title summarizes the contribution.
• Linked the related issue in the description (if any).
• Existing tests pass (pytest tests/ -k cpu); the clean CPU-only GitHub matrix is the authoritative full-suite run.
• New behavior is covered by tests.
• Described the test commands run and hardware scope.
• Public API / CLI changes are additive and backward-compatible (see CONTRIBUTING.md).

[xsuler]

@CharlesXu-HQ Please resolve merge conflicts

[CharlesXu-HQ]

Resolved — I rebased the branch onto current main (01b2321) and force-pushed 790995e with --force-with-lease.

The conflict resolution adapts DAPO to the modular CUDA backend and canonical packed loss layout introduced by #493, propagates the current common trainer options, and rejects MLX at CLI preflight because DAPO is currently CUDA/Torch-only.

Fresh validation on the rebased head:

• DAPO CPU tests: 27 passed
• DAPO CLI/config tests: 11 passed
• Ruff lint + format checks: passed
• strict Sphinx build: 46 documents, no warnings
• native CUDA extension rebuild and bounded Qwen3-0.6B dynamic-sampling → train integration: passed

GitHub now reports the PR as mergeable.