feat(tutorial): add a CPU-first RL walkthrough

examples/tutorial/README.md

@@ -0,0 +1,114 @@
+# CPU-First RL Learning Path: Design and Scope
+
+[中文](./README_CN.md) | [Detailed walkthrough](./WALKTHROUGH.md)
+
+## Summary
+
+This directory adds a single-process, CPU-only path through Molt's core RL
+logic. It is designed for contributors who want to understand or make small
+changes to the reward-to-update path on a laptop before moving the same change
+to a GPU server.
+
+The tutorial reuses Molt's production dataset, agent, environment, trajectory,
+experience, advantage, replay-buffer, and loss code. It replaces only the
+boundaries that require distributed scheduling or GPU model execution.
+
+This is a learning and debugging harness, not a second production training
+backend.
+
+## Motivation
+
+Molt makes RL experiments convenient, but its production path necessarily
+combines the learning algorithm with Ray, vLLM, AutoModel, FSDP, CUDA, and the
+communication between them. That is the right architecture for real training,
+but it makes the underlying control flow difficult to inspect without a GPU
+cluster.
+
+A CPU path gives contributors a fast way to:
+
+- follow one example from dataset row to prompt, rollout, reward, experience,
+  loss, gradient, and optimizer update;
+- use ordinary breakpoints in one process;
+- validate lightweight algorithm or data-flow changes locally;
+- identify the exact remaining distributed and hardware-specific behavior that
+  must be tested on a server.
+
+The same explicit path also helps coding agents trace a change from its input
+field or flag to the executed branch, tensor, metric, and test.
+
+## Design principles
+
+1. **Keep Molt's source authoritative.** Import and execute the existing
+   implementation instead of copying it into tutorial code.
+2. **Cut only infrastructure boundaries.** Replace Ray scheduling, vLLM serving,
+   distributed model execution, and weight broadcast; keep the RL records and
+   calculations real.
+3. **Keep the path linear.** One script exposes the complete sequence without a
+   parallel orchestration layer, so both people and tools can trace it directly.
+4. **Make substitutions visible.** Local shims fail fast if the walkthrough
+   accidentally enters Ray queue scheduling or vLLM sampling configuration.
+5. **Leave production code unchanged.** All CPU-specific adaptation stays under
+   `examples/tutorial/`.
+
+## What is included
+
+- `cpu_rl_walkthrough.py`: the executable CPU learning path.
+- `requirements-cpu.txt`: a small, pinned environment without Ray, vLLM, or GPU
+  runtimes.
+- `WALKTHROUGH.md` and `WALKTHROUGH_CN.md`: setup, source correspondence,
+  execution flow, deliberate small-scale settings, and suggested breakpoints.
+
+The walkthrough uses the real Qwen3 tokenizer and chat template, but it never
+downloads model weights. Its local model is a real CPU
+`Embedding -> Linear` policy whose sampled tokens, behavior log probabilities,
+backward pass, and optimizer update remain internally consistent.
+
+## Reused and replaced boundaries
+
+| Area | Behavior in this tutorial |
+| --- | --- |
+| Dataset and loader | Real Hugging Face `Dataset`, `PromptDataset`, and `StatefulDataLoader` |
+| Agent, environment, reward | Real math agent, `StepEnvRunner`, environment, grader, and `Result` |
+| RL records | Real `Trajectory` and `Experience` |
+| Sampling pipeline | Real group checks, dynamic filtering, and trajectory conversion |
+| Experience building | Real `RemoteExperienceMaker` logic with synchronous local result unwrapping |
+| Algorithm | Real `reinforce_baseline`, replay collation, PPO policy loss, importance correction, and KL-as-loss |
+| Optimization | Real PyTorch autograd, gradient clipping, AdamW, and scheduler |
+| Ray | Scheduling, actors, queues, placement, and waiting are removed |
+| vLLM | Replaced by a small local generation transport |
+| GPU model stack | AutoModel, FSDP, CUDA/NCCL, and weight broadcast are replaced by a CPU actor shared by rollout/training plus a frozen reference copy |
+
+## Non-goals
+
+- Providing a supported CPU backend for production training.
+- Simulating model quality, throughput, memory pressure, or distributed timing.
+- Validating multi-rank, vLLM, FSDP, CUDA, or communication correctness.
+- Replacing Molt's full development environment or test dependencies.
+
+Those behaviors still belong on the normal GPU/distributed path. The tutorial
+only makes the algorithmic handoff to that path smaller and easier to inspect.
+
+## Verification
+
+The walkthrough has been exercised end to end on macOS arm64 with Python 3.12
+in a fresh environment containing no installed `molt`, Ray, or vLLM package.
+The online tokenizer bootstrap, cached offline run, `pip check`, and internal
+shape/data-flow assertions pass without downloading model weights.
+
+Repository `compileall` and the CPU-safe unit tests also pass. Collecting the
+complete production test suite still requires vLLM; the tutorial environment
+does not replace those development dependencies. Linux dependency resolution
+selects the official PyTorch CPU wheel; a Linux runtime smoke test is a useful
+CI follow-up.
+
+## Follow-up
+
+The planned next step is a companion skill for users and coding agents:
+
+1. make and inspect a lightweight change against the CPU path;
+2. trace the affected records, tensors, branches, and metrics locally;
+3. map the change to the corresponding GPU/distributed path;
+4. use the server only for the remaining hardware and distributed validation.
+
+See the [detailed walkthrough](./WALKTHROUGH.md) for installation, the exact
+executed path, source mapping, and breakpoint order.

examples/tutorial/README_CN.md

@@ -0,0 +1,81 @@
+# CPU 优先的 RL 学习路径：设计与范围
+
+[English](./README.md) | [详细教程](./WALKTHROUGH_CN.md)
+
+## 摘要
+
+这个目录为 Molt 的核心 RL 逻辑增加了一条单进程、纯 CPU 的执行路径。它面向希望先在个人电脑上理解奖励到参数更新的完整过程，或做轻量修改，再把同一项修改迁移到 GPU 服务器的贡献者。
+
+教程直接复用 Molt 原有的数据集、agent、环境、trajectory、experience、优势估计、replay buffer 和 loss 实现；只替换必须依赖分布式调度或 GPU 模型执行的边界。
+
+它是学习和调试工具，不是第二套生产训练后端。
+
+## 动机
+
+Molt 的目标是让 RL 实验更方便，但生产执行链必然会把学习算法与 Ray、vLLM、AutoModel、FSDP、CUDA 及其通信机制组合在一起。这很适合真实训练，却使贡献者很难在没有 GPU 集群的情况下观察最核心的控制流。
+
+纯 CPU 路径让贡献者可以：
+
+- 从一条数据开始，依次观察 prompt、rollout、reward、experience、loss、梯度和优化器更新；
+- 在单进程中使用普通断点逐行调试；
+- 先在本地验证轻量的算法或数据流修改；
+- 明确迁移到服务器之后，真正还需要验证哪些分布式和硬件行为。
+
+这条显式执行链也方便 coding agent 把一项修改从输入字段或参数一路追踪到实际分支、张量、指标和测试。
+
+## 设计原则
+
+1. **以 Molt 原始实现为准。** 直接导入并执行仓库代码，不把实现复制进教程。
+2. **只切断基础设施边界。** 替换 Ray 调度、vLLM 服务、分布式模型执行和权重广播，保留真实的 RL 数据记录与计算。
+3. **保持线性执行。** 用一个脚本展示完整流程，不引入并行编排层，便于人和工具直接追踪。
+4. **让替换点清晰可见。** 如果教程意外进入 Ray 队列调度或 vLLM 采样配置，本地 shim 会立即报错。
+5. **不改生产代码。** 所有 CPU 适配都放在 `examples/tutorial/` 下。
+
+## 本次新增
+
+- `cpu_rl_walkthrough.py`：可直接运行的 CPU 学习路径。
+- `requirements-cpu.txt`：不包含 Ray、vLLM 和 GPU runtime 的小型固定依赖环境。
+- `WALKTHROUGH.md` 和 `WALKTHROUGH_CN.md`：安装方式、源码对应关系、执行链、小规模设置与建议断点。
+
+教程使用真实的 Qwen3 tokenizer 和 chat template，但不会下载模型权重。本地模型是一个真正运行在 CPU 上的 `Embedding -> Linear` policy；采样 token、行为策略 log probability、反向传播和优化器更新在数学上保持一致。
+
+## 复用范围与替换边界
+
+| 模块 | 本教程中的行为 |
+| --- | --- |
+| 数据与加载 | 真实的 Hugging Face `Dataset`、`PromptDataset` 和 `StatefulDataLoader` |
+| Agent、环境与奖励 | 真实的 math agent、`StepEnvRunner`、环境、grader 和 `Result` |
+| RL 数据记录 | 真实的 `Trajectory` 和 `Experience` |
+| 采样处理 | 真实的分组完整性检查、动态过滤和 trajectory 转换 |
+| Experience 构建 | 真实的 `RemoteExperienceMaker` 逻辑，本地同步解包返回值 |
+| 算法 | 真实的 `reinforce_baseline`、replay 拼批、PPO policy loss、重要性修正与 KL-as-loss |
+| 参数更新 | 真实的 PyTorch autograd、梯度裁剪、AdamW 和 scheduler |
+| Ray | 移除 actor、队列、placement、等待和调度 |
+| vLLM | 替换为小型本地生成 transport |
+| GPU 模型栈 | 用 rollout/training 共享的 CPU actor 加一份冻结 reference 副本，替换 AutoModel、FSDP、CUDA/NCCL 和权重广播 |
+
+## 非目标
+
+- 提供可用于生产训练的正式 CPU 后端。
+- 模拟模型质量、吞吐量、显存压力或分布式时序。
+- 验证多 rank、vLLM、FSDP、CUDA 或通信正确性。
+- 替代 Molt 完整的开发环境或测试依赖。
+
+这些行为仍应在正常的 GPU/分布式路径中验证。本教程只负责让迁移前的算法逻辑更容易观察，并缩小需要交给服务器验证的范围。
+
+## 验证情况
+
+教程已在 macOS arm64、Python 3.12 的全新环境中完成端到端验证；该环境没有安装 `molt`、Ray 或 vLLM。在线初始化 tokenizer、缓存后的离线运行、`pip check` 以及脚本内部的形状和数据流断言均通过，且没有下载模型权重。
+
+仓库的 `compileall` 和 CPU-safe 单元测试也已通过。收集完整生产测试仍然需要 vLLM；教程环境并不替代这些开发依赖。Linux 依赖解析会选择 PyTorch 官方 CPU wheel，后续适合在 CI 中补充 Linux 运行时 smoke test。
+
+## 后续计划
+
+下一步计划围绕这条 CPU 路径提供一套配套 skill，供用户和 coding agent 使用：
+
+1. 在 CPU 路径上实现并观察轻量修改；
+2. 在本地追踪受影响的数据记录、张量、分支和指标；
+3. 把修改映射到对应的 GPU/分布式路径；
+4. 只把剩余的硬件与分布式验证放到服务器完成。
+
+安装方法、完整执行链、源码映射和建议断点请见[详细教程](./WALKTHROUGH_CN.md)。

examples/tutorial/WALKTHROUGH.md

@@ -0,0 +1,127 @@
+# Molt CPU RL Walkthrough
+
+[Overview](./README.md) | [中文教程](./WALKTHROUGH_CN.md)
+
+## Motivation
+
+Molt is designed to make RL experimentation easy, but the production quick starts necessarily bring up Ray, vLLM, AutoModel, FSDP, and a GPU cluster before a new contributor can step through the reward-to-update logic.
+
+This tutorial provides a small CPU learning harness for that logic. It reuses Molt's original dataset, agent, environment, trajectory, experience, advantage, replay-buffer, and loss implementations, while replacing only the distributed scheduler and GPU model-execution boundaries. The result is intentionally not a second training backend: it is a readable, single-process path for learning, debugging, and validating lightweight changes before moving them to a production recipe.
+
+The same structure also helps coding agents trace a change from input row to prompt, rollout, reward, experience, loss, gradient, and updated policy without having to emulate a cluster.
+
+## What stays real
+
+- The Qwen3 tokenizer and chat template loaded through Molt's `get_tokenizer()`.
+- Hugging Face `Dataset`, `PromptDataset`, and `StatefulDataLoader`.
+- `StepEnvRunner`, the math environment and grader, `Result`, and `Trajectory`.
+- Group completeness checks, dynamic filtering, and `Trajectory -> Experience` conversion.
+- Old-policy and frozen-reference forwards through `RemoteExperienceMaker`.
+- `reinforce_baseline`, replay-buffer collation, `PolicyLoss`, KL-as-loss, autograd, gradient clipping, AdamW, and the learning-rate scheduler.
+
+The local policy is a real CPU `Embedding -> Linear` model rather than a random shape stub. The fake generation transport samples tokens from its softmax, records the corresponding behavior-policy log probabilities, and shares the same model object with training, so an optimizer update is immediately visible to the next forward.
+
+## What is replaced
+
+- Ray actors, queues, placement, and `remote/wait` scheduling.
+- vLLM serving and transport.
+- AutoModel/FSDP/CUDA/NCCL model execution.
+- The `PolicyTrainer`/`FsdpStrategy` shell that is coupled to those distributed runtimes. The walkthrough spells out its corresponding CPU forward, loss, backward, clipping, optimizer, and scheduler operations.
+- FSDP-to-vLLM weight broadcast. In one process, rollout and training share one model object.
+
+Every replacement is visible near the top of the single walkthrough file. Ray queue scheduling and vLLM `SamplingParams` fail immediately if the tutorial accidentally enters those production paths.
+
+## Files and source correspondence
+
+- `cpu_rl_walkthrough.py` is the only executable entry point. It is deliberately linear so it can be debugged from the first line.
+- `requirements-cpu.txt` contains the pinned CPU dependencies and omits Ray and GPU runtimes.
+
+No Molt implementation is copied. The import boundary follows the eager imports in `molt/models/__init__.py`, `samples_generator.py`, and `experience_maker.py`. `TinyPolicy` and the fake engine implement the interfaces consumed by `Actor.forward` and `StepEnvRunner`. The explicit update block corresponds to `PolicyTrainer.training_step` and the CPU-meaningful parts of `FsdpStrategy.backward/optimizer_step`.
+
+The existing `molt/`, `tests/`, and production examples are unchanged; all adaptation code lives in this directory.
+
+## Run
+
+The tutorial requirements are intentionally separate from the full repository requirements:
+
+```bash
+python3.12 -m venv .venv
+.venv/bin/python -m pip install -r examples/tutorial/requirements-cpu.txt
+.venv/bin/python examples/tutorial/cpu_rl_walkthrough.py
+```
+
+The script adds the repository root to `sys.path`, so it does not need `pip install -e . --no-deps` and does not create incomplete `molt` package metadata. Direct dependencies are pinned; Linux also queries the official PyTorch CPU wheel index.
+
+The tokenizer is pinned to revision `c1899de289a04d12100db370d81485cdf75e47ca` of `Qwen/Qwen3-0.6B`. `snapshot_download()` allows only config, tokenizer, vocabulary, and merge files. The first run downloads about 15 MB and no model weights.
+
+After the cache is warm, the tutorial can run explicitly offline:
+
+```bash
+HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
+  .venv/bin/python examples/tutorial/cpu_rl_walkthrough.py
+```
+
+## Executed path
+
+```text
+Hugging Face Dataset.from_list (one local row)
+  -> PromptDataset.__getitem__ / collate_fn
+  -> StatefulDataLoader(batch_size=1)
+  -> _collect_prompt_batch
+  -X Ray dispatch / vLLM transport (direct runner + TinyVllmEngine call)
+  -> examples/python/agents/math.py
+  -> StepEnvRunner.execute -> MathEnv.step -> Result -> Trajectory
+  -> SamplesGenerator._filter_group (complete-group check + dynamic filtering)
+  -> SamplesGenerator._process_response_into_experience
+  -> RemoteExperienceMaker.build_experiences
+  -> TinyPolicy old-policy forward + frozen-reference forward
+  -> reinforce_baseline advantage
+  -> balance_experiences
+  -> NaiveReplayBuffer -> torch DataLoader
+  -X PolicyTrainer / AutoModel / FSDP Actor (explicit CPU update + TinyPolicy)
+  -> log_probs_from_logits / PolicyLoss / compute_approx_kl / agg_loss
+  -> loss.backward -> clip_grad_norm_ -> AdamW -> scheduler
+  -X FSDP-to-vLLM broadcast (one shared model object)
+```
+
+`-X` marks every cut boundary. Local `ray.get` only unwraps synchronous local results; real `wait`/queue scheduling and vLLM `SamplingParams` remain disabled. `Result`, `Trajectory`, and `Experience` are the original Molt records, not tutorial dictionaries.
+
+## Deliberate small-scale changes
+
+- The dataset is one in-memory Hugging Face row, with the same `prompt` and `reward_model` fields as the Qwen3 math quick start.
+- The production recipe's eight samples per prompt are reduced to two. Per-token inverse-CDF draws deterministically produce one `\boxed{4}` and one `\boxed{5}` while preserving the autoregressive policy/log-probability contract.
+- The two rollouts share a group ID, so the real `reinforce_baseline` estimator produces positive and negative advantages.
+- The quick-start settings for dynamic filtering, old-policy/reference forwards, KL-as-loss (`k2`, coefficient `0.001`), PPO clipping, and importance-sampling correction remain active. Actor and reference are identical on the first step, so the KL value is zero even though the complete branch executes.
+- DP, TP, and CP are one; dynamic batching is disabled. These settings remove throughput and memory scheduling without changing the single-rank tensor semantics being studied.
+- Pinned memory is disabled on CPU. The prompt loader is still `StatefulDataLoader`, and training still uses `NaiveReplayBuffer + DataLoader`.
+- AdamW uses the quick-start betas, epsilon, zero weight decay, constant scheduler, and `max_norm=1.0`. Only the learning rate is increased from `1e-6` to `0.01` so one tutorial step produces a visible update.
+
+## Suggested breakpoint order
+
+1. `PromptDataset.__getitem__`: prompt rendering before tokenization.
+2. `_collect_prompt_batch`: the five DataLoader columns.
+3. `StepEnvRunner.execute`: tokenizer, fake generation transport, and real environment step.
+4. `SamplesGenerator._filter_group`: rollout completeness and the `0.5` group score passing dynamic filtering.
+5. `SamplesGenerator._process_response_into_experience`: token axis `T` becoming next-token axis `T-1`.
+6. `RemoteExperienceMaker.make_experience`: local old-policy/reference forwards and log-probability fields.
+7. `RemoteExperienceMaker.compute_advantages_and_returns`: rewards becoming positive and negative advantages.
+8. `NaiveReplayBuffer.append/collate_fn`: rollout batches becoming training microbatches.
+9. `PolicyLoss.forward`, then `compute_approx_kl/agg_loss`: PPO, importance correction, and KL-as-loss.
+10. `total_loss.backward()`: gradients followed by clipping, AdamW, scheduler, and zero-grad.
+
+## Validation
+
+The walkthrough has been verified on macOS arm64 with Python 3.12 in a fresh environment containing no installed `molt`, Ray, or vLLM package. `pip check`, online tokenizer bootstrap, cached offline execution, and the end-to-end assertions all pass. Linux resolution selects the PyTorch `2.11.0+cpu` wheel; a Linux runtime smoke test remains desirable in CI.
+
+The full Molt test environment still requires the production vLLM dependency. This tutorial requirements file is only for the CPU walkthrough, not a replacement development environment for the entire repository.
+
+## Roadmap
+
+A planned follow-up is a coding-agent skill built around this CPU path. The intended workflow is:
+
+1. make and inspect a lightweight change against the CPU walkthrough;
+2. trace the affected Molt records, tensors, branches, and metrics locally;
+3. transfer the change to the corresponding GPU/distributed path;
+4. validate only the remaining distributed and hardware-specific behavior on the server.
+
+Keeping the CPU path structurally close to the production code should make that transfer easier for both users and coding agents.