Skip to content

Add GSM8K examples for RLOO, REINFORCE++, GSPO and SAPO - #622

Open
KunyangZhang wants to merge 1 commit into
agentscope-ai:mainfrom
KunyangZhang:feat/missing-gsm8k-examples
Open

KunyangZhang wants to merge 1 commit into
agentscope-ai:mainfrom
KunyangZhang:feat/missing-gsm8k-examples

Conversation

@KunyangZhang

@KunyangZhang KunyangZhang commented Sep 15, 2026

Copy link
Copy Markdown

Description

Four supported algorithms have no example config: RLOO, REINFORCE++, GSPO and SAPO
(their Doc / Example cell in docs/sphinx_doc/source/main.md reads -). This follows the
"Improve Examples" item in #470 ("Add examples for workflows or algorithms implemented but not
yet documented in the examples directory").

Background. The supported-algorithm table lists these four with algorithm_type values but no
example, so the only way to find a starting config for them is to read trinity/algorithm/algorithm.py.
Every other comparable algorithm (GRPO, CISPO, TOPR, sPPO, AsymRE, REC, OPD, JSD, ...) ships a
GSM8K example.

Changes.

  • examples/rloo_gsm8k/, examples/reinforceplusplus_gsm8k/, examples/gspo_gsm8k/,
    examples/sapo_gsm8k/ — each with a gsm8k.yaml and a short README.md.
  • The support tables in README.md, README_zh.md, docs/sphinx_doc/source/main.md and
    source_zh/main.md — fills in the example links for these four algorithms, and also restores the
    missing CISPO link (its example already exists in the repository but the table still showed -).
  • docs/sphinx_doc/source/tutorial/example_dataset_perspective.md and the source_zh twin — adds
    the four GSM8K rows next to the existing CISPO/AsymRE rows.

Each YAML is examples/grpo_gsm8k/gsm8k.yaml with exactly two lines changed: the experiment name
and algorithm.algorithm_type. Dataset, model, cluster, buffer, explorer and trainer sections are
untouched, so the algorithm's own default_config() supplies the component wiring and the
resources stay at the smallest existing GSM8K footprint (2 GPUs).

How to test. The new configs are covered by the existing
tests/common/config_test.py::test_all_examples_are_valid, which walks examples/ and runs
load_config + check_and_update() on every YAML. I ran that exact body locally against the two
pre-existing GSM8K configs and the four new ones — all six pass identically. I also printed the
resolved wiring to confirm each example selects the intended components:

examples/rloo_gsm8k/gsm8k.yaml               algorithm_type=rloo                advantage_fn=rloo                 policy_loss_fn=ppo
examples/reinforceplusplus_gsm8k/gsm8k.yaml  algorithm_type=reinforceplusplus   advantage_fn=reinforceplusplus    policy_loss_fn=ppo   (gamma=1.0)
examples/gspo_gsm8k/gsm8k.yaml               algorithm_type=gspo                advantage_fn=grpo                 policy_loss_fn=gspo  (clip_range_low=3e-4, clip_range_high=4e-4)
examples/sapo_gsm8k/gsm8k.yaml               algorithm_type=sapo                advantage_fn=grpo                 policy_loss_fn=sapo  (tau_pos=1.0, tau_neg=1.05)

AlgorithmConfigValidator().validate(cfg) passes for all four, so the registry lookups and the
per-component default_args() resolution are exercised. I could not run pytest tests/common/config_test.py
directly here because the module imports vllm at collection time (vllm.config.ModelConfig) and this
machine has no GPU/vLLM install; the local run above reproduces the test body instead.

Checklist

  • Code has passed all tests (example configs validated with the body of test_all_examples_are_valid)
  • Docstrings have been added/updated in Google Style (no Python code changed)
  • Documentation has been updated (algorithm support tables + dataset-perspective tables, EN and ZH)
  • Code is ready for review

Refs #470

CI here runs on /unittest from a maintainer; I will watch for the result and fix anything it reports.

Four algorithms are implemented and listed in the supported-algorithm table
but have no example: RLOO, REINFORCE++, GSPO and SAPO (their `Doc / Example`
column reads `-`). This follows the "Improve Examples" item in agentscope-ai#470.

Each example mirrors examples/grpo_gsm8k/gsm8k.yaml and differs only in the
experiment name and `algorithm.algorithm_type`; every other field (dataset,
model, cluster, buffer, explorer, trainer) is unchanged, so the algorithm's own
`default_config()` supplies the advantage/policy-loss wiring.

The support tables in README.md, README_zh.md, docs/.../main.md and the
dataset-perspective tables (EN and ZH) now link the new examples; the CISPO
example link, which existed in the repository but was still shown as `-`, is
restored as well.

Verified by running the body of tests/common/config_test.py::test_all_examples_are_valid
against both pre-existing and new configs (load_config + check_and_update), and by
checking the resolved wiring: rloo -> advantage_fn=rloo, reinforceplusplus ->
advantage_fn=reinforceplusplus/gamma=1.0, gspo -> policy_loss_fn=gspo,
sapo -> policy_loss_fn=sapo (tau_pos=1.0, tau_neg=1.05).

Refs agentscope-ai#470
@KunyangZhang
KunyangZhang force-pushed the feat/missing-gsm8k-examples branch from 8c6e7dd to 0c34dfa Compare September 15, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant