Add GSM8K examples for RLOO, REINFORCE++, GSPO and SAPO - #622
Open
KunyangZhang wants to merge 1 commit into
Open
KunyangZhang wants to merge 1 commit into
KunyangZhang wants to merge 1 commit into
Conversation
Four algorithms are implemented and listed in the supported-algorithm table but have no example: RLOO, REINFORCE++, GSPO and SAPO (their `Doc / Example` column reads `-`). This follows the "Improve Examples" item in agentscope-ai#470. Each example mirrors examples/grpo_gsm8k/gsm8k.yaml and differs only in the experiment name and `algorithm.algorithm_type`; every other field (dataset, model, cluster, buffer, explorer, trainer) is unchanged, so the algorithm's own `default_config()` supplies the advantage/policy-loss wiring. The support tables in README.md, README_zh.md, docs/.../main.md and the dataset-perspective tables (EN and ZH) now link the new examples; the CISPO example link, which existed in the repository but was still shown as `-`, is restored as well. Verified by running the body of tests/common/config_test.py::test_all_examples_are_valid against both pre-existing and new configs (load_config + check_and_update), and by checking the resolved wiring: rloo -> advantage_fn=rloo, reinforceplusplus -> advantage_fn=reinforceplusplus/gamma=1.0, gspo -> policy_loss_fn=gspo, sapo -> policy_loss_fn=sapo (tau_pos=1.0, tau_neg=1.05). Refs agentscope-ai#470
KunyangZhang
force-pushed
the
feat/missing-gsm8k-examples
branch
from
September 15, 2026 16:35
8c6e7dd to
0c34dfa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Four supported algorithms have no example config: RLOO, REINFORCE++, GSPO and SAPO
(their
Doc / Examplecell indocs/sphinx_doc/source/main.mdreads-). This follows the"Improve Examples" item in #470 ("Add examples for workflows or algorithms implemented but not
yet documented in the examples directory").
Background. The supported-algorithm table lists these four with
algorithm_typevalues but noexample, so the only way to find a starting config for them is to read
trinity/algorithm/algorithm.py.Every other comparable algorithm (GRPO, CISPO, TOPR, sPPO, AsymRE, REC, OPD, JSD, ...) ships a
GSM8K example.
Changes.
examples/rloo_gsm8k/,examples/reinforceplusplus_gsm8k/,examples/gspo_gsm8k/,examples/sapo_gsm8k/— each with agsm8k.yamland a shortREADME.md.README.md,README_zh.md,docs/sphinx_doc/source/main.mdandsource_zh/main.md— fills in the example links for these four algorithms, and also restores themissing CISPO link (its example already exists in the repository but the table still showed
-).docs/sphinx_doc/source/tutorial/example_dataset_perspective.mdand thesource_zhtwin — addsthe four GSM8K rows next to the existing CISPO/AsymRE rows.
Each YAML is
examples/grpo_gsm8k/gsm8k.yamlwith exactly two lines changed: the experimentnameand
algorithm.algorithm_type. Dataset, model, cluster, buffer, explorer and trainer sections areuntouched, so the algorithm's own
default_config()supplies the component wiring and theresources stay at the smallest existing GSM8K footprint (2 GPUs).
How to test. The new configs are covered by the existing
tests/common/config_test.py::test_all_examples_are_valid, which walksexamples/and runsload_config+check_and_update()on every YAML. I ran that exact body locally against the twopre-existing GSM8K configs and the four new ones — all six pass identically. I also printed the
resolved wiring to confirm each example selects the intended components:
AlgorithmConfigValidator().validate(cfg)passes for all four, so the registry lookups and theper-component
default_args()resolution are exercised. I could not runpytest tests/common/config_test.pydirectly here because the module imports
vllmat collection time (vllm.config.ModelConfig) and thismachine has no GPU/vLLM install; the local run above reproduces the test body instead.
Checklist
test_all_examples_are_valid)Refs #470
CI here runs on
/unittestfrom a maintainer; I will watch for the result and fix anything it reports.