Skip to content

feat: add configurable DAPO objective controls - #58

Draft
0z5a wants to merge 4 commits into
ViperEkura:mainfrom
0z5a:codex/dapo-loss
Draft

feat: add configurable DAPO objective controls#58
0z5a wants to merge 4 commits into
ViperEkura:mainfrom
0z5a:codex/dapo-loss

Conversation

@0z5a

@0z5a 0z5a commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add DAPO Clip-Higher through independent lower and upper GRPO clipping bounds
  • add configurable token-level versus sequence-level policy-loss aggregation
  • add soft overlong reward shaping with buffer and scale controls
  • preserve the existing objective exactly when all new options remain at defaults
  • include a reproducible L20 benchmark and frozen InfraSWE comparison evidence

Dynamic sampling is intentionally not included in this PR: correct refill semantics require a rollout-buffer lifecycle change rather than silently discarding zero-variance groups.

L20 benchmark

Environment: NVIDIA L20 (SM89), PyTorch 2.11.0+cu128, CUDA 12.8, seed 3407, 20 warmups and 100 timed repetitions.

Shape (batch × group × response) Baseline median Default candidate Delta Full DAPO median Absolute DAPO cost
4 × 8 × 256 0.256 ms 0.254 ms -0.80% 0.312 ms +0.056 ms
4 × 8 × 1024 0.257 ms 0.257 ms 0.00% 0.316 ms +0.059 ms
8 × 8 × 2048 0.269 ms 0.269 ms 0.00% 0.330 ms +0.060 ms
4 × 8 × 4096 0.269 ms 0.268 ms -0.38% 0.329 ms +0.059 ms

Default-path maximum absolute output difference is 0 for all four cases. Timings isolate the objective calculation and exclude model forward/backward work.

Validation

  • local suite: 635 passed, 103 skipped
  • L20 focused suite: 26 passed
  • Ruff format/import checks: passed
  • InfraSWE diagnostic ProjectFit: 92.15
  • InfraSWE BenchmarkTrust: 97.40
  • official InfraSWE score: unresolved because signed draft-seal, hidden probes, fresh-process replay, and verified evidence-manifest inputs are unavailable

The implementation follows the objective components described in the DAPO paper. Raw benchmark JSON, the runnable benchmark, and InfraSWE scoring inputs are committed under benchmarks/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant