From 3f51afb1090646ac65bf85cd5c54b2fb1d977e15 Mon Sep 17 00:00:00 2001 From: Yann Ge Date: Thu, 16 Jul 2026 00:31:20 +0800 Subject: [PATCH] feat(tutorial): add a CPU-first RL walkthrough Signed-off-by: Yann Ge --- examples/tutorial/README.md | 114 ++++++ examples/tutorial/README_CN.md | 81 +++++ examples/tutorial/WALKTHROUGH.md | 127 +++++++ examples/tutorial/WALKTHROUGH_CN.md | 127 +++++++ examples/tutorial/cpu_rl_walkthrough.py | 458 ++++++++++++++++++++++++ examples/tutorial/requirements-cpu.txt | 19 + 6 files changed, 926 insertions(+) create mode 100644 examples/tutorial/README.md create mode 100644 examples/tutorial/README_CN.md create mode 100644 examples/tutorial/WALKTHROUGH.md create mode 100644 examples/tutorial/WALKTHROUGH_CN.md create mode 100644 examples/tutorial/cpu_rl_walkthrough.py create mode 100644 examples/tutorial/requirements-cpu.txt diff --git a/examples/tutorial/README.md b/examples/tutorial/README.md new file mode 100644 index 0000000..8091d6d --- /dev/null +++ b/examples/tutorial/README.md @@ -0,0 +1,114 @@ +# CPU-First RL Learning Path: Design and Scope + +[中文](./README_CN.md) | [Detailed walkthrough](./WALKTHROUGH.md) + +## Summary + +This directory adds a single-process, CPU-only path through Molt's core RL +logic. It is designed for contributors who want to understand or make small +changes to the reward-to-update path on a laptop before moving the same change +to a GPU server. + +The tutorial reuses Molt's production dataset, agent, environment, trajectory, +experience, advantage, replay-buffer, and loss code. It replaces only the +boundaries that require distributed scheduling or GPU model execution. + +This is a learning and debugging harness, not a second production training +backend. + +## Motivation + +Molt makes RL experiments convenient, but its production path necessarily +combines the learning algorithm with Ray, vLLM, AutoModel, FSDP, CUDA, and the +communication between them. That is the right architecture for real training, +but it makes the underlying control flow difficult to inspect without a GPU +cluster. + +A CPU path gives contributors a fast way to: + +- follow one example from dataset row to prompt, rollout, reward, experience, + loss, gradient, and optimizer update; +- use ordinary breakpoints in one process; +- validate lightweight algorithm or data-flow changes locally; +- identify the exact remaining distributed and hardware-specific behavior that + must be tested on a server. + +The same explicit path also helps coding agents trace a change from its input +field or flag to the executed branch, tensor, metric, and test. + +## Design principles + +1. **Keep Molt's source authoritative.** Import and execute the existing + implementation instead of copying it into tutorial code. +2. **Cut only infrastructure boundaries.** Replace Ray scheduling, vLLM serving, + distributed model execution, and weight broadcast; keep the RL records and + calculations real. +3. **Keep the path linear.** One script exposes the complete sequence without a + parallel orchestration layer, so both people and tools can trace it directly. +4. **Make substitutions visible.** Local shims fail fast if the walkthrough + accidentally enters Ray queue scheduling or vLLM sampling configuration. +5. **Leave production code unchanged.** All CPU-specific adaptation stays under + `examples/tutorial/`. + +## What is included + +- `cpu_rl_walkthrough.py`: the executable CPU learning path. +- `requirements-cpu.txt`: a small, pinned environment without Ray, vLLM, or GPU + runtimes. +- `WALKTHROUGH.md` and `WALKTHROUGH_CN.md`: setup, source correspondence, + execution flow, deliberate small-scale settings, and suggested breakpoints. + +The walkthrough uses the real Qwen3 tokenizer and chat template, but it never +downloads model weights. Its local model is a real CPU +`Embedding -> Linear` policy whose sampled tokens, behavior log probabilities, +backward pass, and optimizer update remain internally consistent. + +## Reused and replaced boundaries + +| Area | Behavior in this tutorial | +| --- | --- | +| Dataset and loader | Real Hugging Face `Dataset`, `PromptDataset`, and `StatefulDataLoader` | +| Agent, environment, reward | Real math agent, `StepEnvRunner`, environment, grader, and `Result` | +| RL records | Real `Trajectory` and `Experience` | +| Sampling pipeline | Real group checks, dynamic filtering, and trajectory conversion | +| Experience building | Real `RemoteExperienceMaker` logic with synchronous local result unwrapping | +| Algorithm | Real `reinforce_baseline`, replay collation, PPO policy loss, importance correction, and KL-as-loss | +| Optimization | Real PyTorch autograd, gradient clipping, AdamW, and scheduler | +| Ray | Scheduling, actors, queues, placement, and waiting are removed | +| vLLM | Replaced by a small local generation transport | +| GPU model stack | AutoModel, FSDP, CUDA/NCCL, and weight broadcast are replaced by a CPU actor shared by rollout/training plus a frozen reference copy | + +## Non-goals + +- Providing a supported CPU backend for production training. +- Simulating model quality, throughput, memory pressure, or distributed timing. +- Validating multi-rank, vLLM, FSDP, CUDA, or communication correctness. +- Replacing Molt's full development environment or test dependencies. + +Those behaviors still belong on the normal GPU/distributed path. The tutorial +only makes the algorithmic handoff to that path smaller and easier to inspect. + +## Verification + +The walkthrough has been exercised end to end on macOS arm64 with Python 3.12 +in a fresh environment containing no installed `molt`, Ray, or vLLM package. +The online tokenizer bootstrap, cached offline run, `pip check`, and internal +shape/data-flow assertions pass without downloading model weights. + +Repository `compileall` and the CPU-safe unit tests also pass. Collecting the +complete production test suite still requires vLLM; the tutorial environment +does not replace those development dependencies. Linux dependency resolution +selects the official PyTorch CPU wheel; a Linux runtime smoke test is a useful +CI follow-up. + +## Follow-up + +The planned next step is a companion skill for users and coding agents: + +1. make and inspect a lightweight change against the CPU path; +2. trace the affected records, tensors, branches, and metrics locally; +3. map the change to the corresponding GPU/distributed path; +4. use the server only for the remaining hardware and distributed validation. + +See the [detailed walkthrough](./WALKTHROUGH.md) for installation, the exact +executed path, source mapping, and breakpoint order. diff --git a/examples/tutorial/README_CN.md b/examples/tutorial/README_CN.md new file mode 100644 index 0000000..86852e6 --- /dev/null +++ b/examples/tutorial/README_CN.md @@ -0,0 +1,81 @@ +# CPU 优先的 RL 学习路径：设计与范围 + +[English](./README.md) | [详细教程](./WALKTHROUGH_CN.md) + +## 摘要 + +这个目录为 Molt 的核心 RL 逻辑增加了一条单进程、纯 CPU 的执行路径。它面向希望先在个人电脑上理解奖励到参数更新的完整过程，或做轻量修改，再把同一项修改迁移到 GPU 服务器的贡献者。 + +教程直接复用 Molt 原有的数据集、agent、环境、trajectory、experience、优势估计、replay buffer 和 loss 实现；只替换必须依赖分布式调度或 GPU 模型执行的边界。 + +它是学习和调试工具，不是第二套生产训练后端。 + +## 动机 + +Molt 的目标是让 RL 实验更方便，但生产执行链必然会把学习算法与 Ray、vLLM、AutoModel、FSDP、CUDA 及其通信机制组合在一起。这很适合真实训练，却使贡献者很难在没有 GPU 集群的情况下观察最核心的控制流。 + +纯 CPU 路径让贡献者可以： + +- 从一条数据开始，依次观察 prompt、rollout、reward、experience、loss、梯度和优化器更新； +- 在单进程中使用普通断点逐行调试； +- 先在本地验证轻量的算法或数据流修改； +- 明确迁移到服务器之后，真正还需要验证哪些分布式和硬件行为。 + +这条显式执行链也方便 coding agent 把一项修改从输入字段或参数一路追踪到实际分支、张量、指标和测试。 + +## 设计原则 + +1. **以 Molt 原始实现为准。** 直接导入并执行仓库代码，不把实现复制进教程。 +2. **只切断基础设施边界。** 替换 Ray 调度、vLLM 服务、分布式模型执行和权重广播，保留真实的 RL 数据记录与计算。 +3. **保持线性执行。** 用一个脚本展示完整流程，不引入并行编排层，便于人和工具直接追踪。 +4. **让替换点清晰可见。** 如果教程意外进入 Ray 队列调度或 vLLM 采样配置，本地 shim 会立即报错。 +5. **不改生产代码。** 所有 CPU 适配都放在 `examples/tutorial/` 下。 + +## 本次新增 + +- `cpu_rl_walkthrough.py`：可直接运行的 CPU 学习路径。 +- `requirements-cpu.txt`：不包含 Ray、vLLM 和 GPU runtime 的小型固定依赖环境。 +- `WALKTHROUGH.md` 和 `WALKTHROUGH_CN.md`：安装方式、源码对应关系、执行链、小规模设置与建议断点。 + +教程使用真实的 Qwen3 tokenizer 和 chat template，但不会下载模型权重。本地模型是一个真正运行在 CPU 上的 `Embedding -> Linear` policy；采样 token、行为策略 log probability、反向传播和优化器更新在数学上保持一致。 + +## 复用范围与替换边界 + +| 模块 | 本教程中的行为 | +| --- | --- | +| 数据与加载 | 真实的 Hugging Face `Dataset`、`PromptDataset` 和 `StatefulDataLoader` | +| Agent、环境与奖励 | 真实的 math agent、`StepEnvRunner`、环境、grader 和 `Result` | +| RL 数据记录 | 真实的 `Trajectory` 和 `Experience` | +| 采样处理 | 真实的分组完整性检查、动态过滤和 trajectory 转换 | +| Experience 构建 | 真实的 `RemoteExperienceMaker` 逻辑，本地同步解包返回值 | +| 算法 | 真实的 `reinforce_baseline`、replay 拼批、PPO policy loss、重要性修正与 KL-as-loss | +| 参数更新 | 真实的 PyTorch autograd、梯度裁剪、AdamW 和 scheduler | +| Ray | 移除 actor、队列、placement、等待和调度 | +| vLLM | 替换为小型本地生成 transport | +| GPU 模型栈 | 用 rollout/training 共享的 CPU actor 加一份冻结 reference 副本，替换 AutoModel、FSDP、CUDA/NCCL 和权重广播 | + +## 非目标 + +- 提供可用于生产训练的正式 CPU 后端。 +- 模拟模型质量、吞吐量、显存压力或分布式时序。 +- 验证多 rank、vLLM、FSDP、CUDA 或通信正确性。 +- 替代 Molt 完整的开发环境或测试依赖。 + +这些行为仍应在正常的 GPU/分布式路径中验证。本教程只负责让迁移前的算法逻辑更容易观察，并缩小需要交给服务器验证的范围。 + +## 验证情况 + +教程已在 macOS arm64、Python 3.12 的全新环境中完成端到端验证；该环境没有安装 `molt`、Ray 或 vLLM。在线初始化 tokenizer、缓存后的离线运行、`pip check` 以及脚本内部的形状和数据流断言均通过，且没有下载模型权重。 + +仓库的 `compileall` 和 CPU-safe 单元测试也已通过。收集完整生产测试仍然需要 vLLM；教程环境并不替代这些开发依赖。Linux 依赖解析会选择 PyTorch 官方 CPU wheel，后续适合在 CI 中补充 Linux 运行时 smoke test。 + +## 后续计划 + +下一步计划围绕这条 CPU 路径提供一套配套 skill，供用户和 coding agent 使用： + +1. 在 CPU 路径上实现并观察轻量修改； +2. 在本地追踪受影响的数据记录、张量、分支和指标； +3. 把修改映射到对应的 GPU/分布式路径； +4. 只把剩余的硬件与分布式验证放到服务器完成。 + +安装方法、完整执行链、源码映射和建议断点请见[详细教程](./WALKTHROUGH_CN.md)。 diff --git a/examples/tutorial/WALKTHROUGH.md b/examples/tutorial/WALKTHROUGH.md new file mode 100644 index 0000000..b902fdd --- /dev/null +++ b/examples/tutorial/WALKTHROUGH.md @@ -0,0 +1,127 @@ +# Molt CPU RL Walkthrough + +[Overview](./README.md) | [中文教程](./WALKTHROUGH_CN.md) + +## Motivation + +Molt is designed to make RL experimentation easy, but the production quick starts necessarily bring up Ray, vLLM, AutoModel, FSDP, and a GPU cluster before a new contributor can step through the reward-to-update logic. + +This tutorial provides a small CPU learning harness for that logic. It reuses Molt's original dataset, agent, environment, trajectory, experience, advantage, replay-buffer, and loss implementations, while replacing only the distributed scheduler and GPU model-execution boundaries. The result is intentionally not a second training backend: it is a readable, single-process path for learning, debugging, and validating lightweight changes before moving them to a production recipe. + +The same structure also helps coding agents trace a change from input row to prompt, rollout, reward, experience, loss, gradient, and updated policy without having to emulate a cluster. + +## What stays real + +- The Qwen3 tokenizer and chat template loaded through Molt's `get_tokenizer()`. +- Hugging Face `Dataset`, `PromptDataset`, and `StatefulDataLoader`. +- `StepEnvRunner`, the math environment and grader, `Result`, and `Trajectory`. +- Group completeness checks, dynamic filtering, and `Trajectory -> Experience` conversion. +- Old-policy and frozen-reference forwards through `RemoteExperienceMaker`. +- `reinforce_baseline`, replay-buffer collation, `PolicyLoss`, KL-as-loss, autograd, gradient clipping, AdamW, and the learning-rate scheduler. + +The local policy is a real CPU `Embedding -> Linear` model rather than a random shape stub. The fake generation transport samples tokens from its softmax, records the corresponding behavior-policy log probabilities, and shares the same model object with training, so an optimizer update is immediately visible to the next forward. + +## What is replaced + +- Ray actors, queues, placement, and `remote/wait` scheduling. +- vLLM serving and transport. +- AutoModel/FSDP/CUDA/NCCL model execution. +- The `PolicyTrainer`/`FsdpStrategy` shell that is coupled to those distributed runtimes. The walkthrough spells out its corresponding CPU forward, loss, backward, clipping, optimizer, and scheduler operations. +- FSDP-to-vLLM weight broadcast. In one process, rollout and training share one model object. + +Every replacement is visible near the top of the single walkthrough file. Ray queue scheduling and vLLM `SamplingParams` fail immediately if the tutorial accidentally enters those production paths. + +## Files and source correspondence + +- `cpu_rl_walkthrough.py` is the only executable entry point. It is deliberately linear so it can be debugged from the first line. +- `requirements-cpu.txt` contains the pinned CPU dependencies and omits Ray and GPU runtimes. + +No Molt implementation is copied. The import boundary follows the eager imports in `molt/models/__init__.py`, `samples_generator.py`, and `experience_maker.py`. `TinyPolicy` and the fake engine implement the interfaces consumed by `Actor.forward` and `StepEnvRunner`. The explicit update block corresponds to `PolicyTrainer.training_step` and the CPU-meaningful parts of `FsdpStrategy.backward/optimizer_step`. + +The existing `molt/`, `tests/`, and production examples are unchanged; all adaptation code lives in this directory. + +## Run + +The tutorial requirements are intentionally separate from the full repository requirements: + +```bash +python3.12 -m venv .venv +.venv/bin/python -m pip install -r examples/tutorial/requirements-cpu.txt +.venv/bin/python examples/tutorial/cpu_rl_walkthrough.py +``` + +The script adds the repository root to `sys.path`, so it does not need `pip install -e . --no-deps` and does not create incomplete `molt` package metadata. Direct dependencies are pinned; Linux also queries the official PyTorch CPU wheel index. + +The tokenizer is pinned to revision `c1899de289a04d12100db370d81485cdf75e47ca` of `Qwen/Qwen3-0.6B`. `snapshot_download()` allows only config, tokenizer, vocabulary, and merge files. The first run downloads about 15 MB and no model weights. + +After the cache is warm, the tutorial can run explicitly offline: + +```bash +HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \ + .venv/bin/python examples/tutorial/cpu_rl_walkthrough.py +``` + +## Executed path + +```text +Hugging Face Dataset.from_list (one local row) + -> PromptDataset.__getitem__ / collate_fn + -> StatefulDataLoader(batch_size=1) + -> _collect_prompt_batch + -X Ray dispatch / vLLM transport (direct runner + TinyVllmEngine call) + -> examples/python/agents/math.py + -> StepEnvRunner.execute -> MathEnv.step -> Result -> Trajectory + -> SamplesGenerator._filter_group (complete-group check + dynamic filtering) + -> SamplesGenerator._process_response_into_experience + -> RemoteExperienceMaker.build_experiences + -> TinyPolicy old-policy forward + frozen-reference forward + -> reinforce_baseline advantage + -> balance_experiences + -> NaiveReplayBuffer -> torch DataLoader + -X PolicyTrainer / AutoModel / FSDP Actor (explicit CPU update + TinyPolicy) + -> log_probs_from_logits / PolicyLoss / compute_approx_kl / agg_loss + -> loss.backward -> clip_grad_norm_ -> AdamW -> scheduler + -X FSDP-to-vLLM broadcast (one shared model object) +``` + +`-X` marks every cut boundary. Local `ray.get` only unwraps synchronous local results; real `wait`/queue scheduling and vLLM `SamplingParams` remain disabled. `Result`, `Trajectory`, and `Experience` are the original Molt records, not tutorial dictionaries. + +## Deliberate small-scale changes + +- The dataset is one in-memory Hugging Face row, with the same `prompt` and `reward_model` fields as the Qwen3 math quick start. +- The production recipe's eight samples per prompt are reduced to two. Per-token inverse-CDF draws deterministically produce one `\boxed{4}` and one `\boxed{5}` while preserving the autoregressive policy/log-probability contract. +- The two rollouts share a group ID, so the real `reinforce_baseline` estimator produces positive and negative advantages. +- The quick-start settings for dynamic filtering, old-policy/reference forwards, KL-as-loss (`k2`, coefficient `0.001`), PPO clipping, and importance-sampling correction remain active. Actor and reference are identical on the first step, so the KL value is zero even though the complete branch executes. +- DP, TP, and CP are one; dynamic batching is disabled. These settings remove throughput and memory scheduling without changing the single-rank tensor semantics being studied. +- Pinned memory is disabled on CPU. The prompt loader is still `StatefulDataLoader`, and training still uses `NaiveReplayBuffer + DataLoader`. +- AdamW uses the quick-start betas, epsilon, zero weight decay, constant scheduler, and `max_norm=1.0`. Only the learning rate is increased from `1e-6` to `0.01` so one tutorial step produces a visible update. + +## Suggested breakpoint order + +1. `PromptDataset.__getitem__`: prompt rendering before tokenization. +2. `_collect_prompt_batch`: the five DataLoader columns. +3. `StepEnvRunner.execute`: tokenizer, fake generation transport, and real environment step. +4. `SamplesGenerator._filter_group`: rollout completeness and the `0.5` group score passing dynamic filtering. +5. `SamplesGenerator._process_response_into_experience`: token axis `T` becoming next-token axis `T-1`. +6. `RemoteExperienceMaker.make_experience`: local old-policy/reference forwards and log-probability fields. +7. `RemoteExperienceMaker.compute_advantages_and_returns`: rewards becoming positive and negative advantages. +8. `NaiveReplayBuffer.append/collate_fn`: rollout batches becoming training microbatches. +9. `PolicyLoss.forward`, then `compute_approx_kl/agg_loss`: PPO, importance correction, and KL-as-loss. +10. `total_loss.backward()`: gradients followed by clipping, AdamW, scheduler, and zero-grad. + +## Validation + +The walkthrough has been verified on macOS arm64 with Python 3.12 in a fresh environment containing no installed `molt`, Ray, or vLLM package. `pip check`, online tokenizer bootstrap, cached offline execution, and the end-to-end assertions all pass. Linux resolution selects the PyTorch `2.11.0+cpu` wheel; a Linux runtime smoke test remains desirable in CI. + +The full Molt test environment still requires the production vLLM dependency. This tutorial requirements file is only for the CPU walkthrough, not a replacement development environment for the entire repository. + +## Roadmap + +A planned follow-up is a coding-agent skill built around this CPU path. The intended workflow is: + +1. make and inspect a lightweight change against the CPU walkthrough; +2. trace the affected Molt records, tensors, branches, and metrics locally; +3. transfer the change to the corresponding GPU/distributed path; +4. validate only the remaining distributed and hardware-specific behavior on the server. + +Keeping the CPU path structurally close to the production code should make that transfer easier for both users and coding agents. diff --git a/examples/tutorial/WALKTHROUGH_CN.md b/examples/tutorial/WALKTHROUGH_CN.md new file mode 100644 index 0000000..607ee98 --- /dev/null +++ b/examples/tutorial/WALKTHROUGH_CN.md @@ -0,0 +1,127 @@ +# Molt CPU RL 逐行教程 + +[说明](./README_CN.md) | [English walkthrough](./WALKTHROUGH.md) + +## 动机 + +Molt 的目标是让 RL 实验更方便，但生产 quick start 会先启动 Ray、vLLM、AutoModel、FSDP 和 GPU 集群，刚接触仓库的贡献者很难直接逐行观察 reward 到参数更新的逻辑。 + +这个教程为该逻辑提供了一条小型 CPU 学习路径。它复用 Molt 原有的数据集、agent、环境、trajectory、experience、优势估计、replay buffer 和 loss 实现，只替换分布式调度与 GPU 模型执行边界。它并不是第二套训练后端，而是一条适合学习、调试和本地验证轻量修改的可读单进程路径。 + +同样的结构也让 coding agent 无需模拟集群，就能把一项修改从输入数据一路追踪到 prompt、rollout、reward、experience、loss、梯度和更新后的 policy。 + +## 哪些部分保持真实 + +- 通过 Molt `get_tokenizer()` 加载的 Qwen3 tokenizer 与 chat template。 +- Hugging Face `Dataset`、`PromptDataset` 和 `StatefulDataLoader`。 +- `StepEnvRunner`、math 环境与 grader、`Result` 和 `Trajectory`。 +- 分组完整性检查、动态过滤以及 `Trajectory -> Experience` 转换。 +- 通过 `RemoteExperienceMaker` 执行的旧策略与冻结参考策略 forward。 +- `reinforce_baseline`、replay buffer 拼批、`PolicyLoss`、KL-as-loss、autograd、梯度裁剪、AdamW 与学习率 scheduler。 + +本地 policy 不是只返回随机形状的 stub，而是一个真正运行在 CPU 上的 `Embedding -> Linear` 模型。本地生成 transport 从它的 softmax 中采样 token，并记录对应的行为策略 log probability。生成与训练共享同一个模型对象，因此优化器更新会直接反映到下一次 forward 中。 + +## 哪些部分被替换 + +- Ray actor、队列、placement 以及 `remote/wait` 调度。 +- vLLM serving 与 transport。 +- AutoModel/FSDP/CUDA/NCCL 模型执行。 +- 与上述分布式 runtime 耦合的 `PolicyTrainer`/`FsdpStrategy` 外壳。教程显式写出其对应的 CPU forward、loss、backward、梯度裁剪、optimizer 和 scheduler 操作。 +- FSDP 到 vLLM 的权重广播。单进程中 rollout 与训练共享同一个模型对象。 + +所有替换都集中展示在单个教程文件开头附近。如果代码意外进入 Ray 队列调度或 vLLM `SamplingParams` 等生产路径，它会立即失败，而不是静默跳过。 + +## 文件与源码对应关系 + +- `cpu_rl_walkthrough.py` 是唯一可执行入口。它刻意保持线性，便于从第一行开始调试。 +- `requirements-cpu.txt` 包含固定版本的 CPU 依赖，不安装 Ray 或 GPU runtime。 + +教程不复制 Molt 的实现。import 边界对应 `molt/models/__init__.py`、`samples_generator.py` 和 `experience_maker.py` 中的 eager import。`TinyPolicy` 与本地生成 engine 实现 `Actor.forward` 和 `StepEnvRunner` 所消费的接口。显式更新代码块对应 `PolicyTrainer.training_step` 与 `FsdpStrategy.backward/optimizer_step` 中在 CPU 上仍有意义的部分。 + +现有的 `molt/`、`tests/` 和生产 examples 保持不变；所有适配代码均位于本目录。 + +## 运行方法 + +教程依赖与完整仓库依赖刻意分开： + +```bash +python3.12 -m venv .venv +.venv/bin/python -m pip install -r examples/tutorial/requirements-cpu.txt +.venv/bin/python examples/tutorial/cpu_rl_walkthrough.py +``` + +脚本会把仓库根目录加入 `sys.path`，因此不需要运行 `pip install -e . --no-deps`，也不会产生依赖不完整的 `molt` 包元数据。直接依赖均固定版本；Linux 还会查询 PyTorch 官方 CPU wheel index。 + +tokenizer 固定在 `Qwen/Qwen3-0.6B` 的 revision `c1899de289a04d12100db370d81485cdf75e47ca`。`snapshot_download()` 只允许下载 config、tokenizer、词表和 merge 文件。首次运行下载约 15 MB，不下载模型权重。 + +缓存准备好以后，可以显式离线运行： + +```bash +HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \ + .venv/bin/python examples/tutorial/cpu_rl_walkthrough.py +``` + +## 实际执行链 + +```text +Hugging Face Dataset.from_list（本地一条数据） + -> PromptDataset.__getitem__ / collate_fn + -> StatefulDataLoader(batch_size=1) + -> _collect_prompt_batch + -X Ray dispatch / vLLM transport（直接调用 runner + TinyVllmEngine） + -> examples/python/agents/math.py + -> StepEnvRunner.execute -> MathEnv.step -> Result -> Trajectory + -> SamplesGenerator._filter_group（完整分组检查 + 动态过滤） + -> SamplesGenerator._process_response_into_experience + -> RemoteExperienceMaker.build_experiences + -> TinyPolicy 旧策略 forward + 冻结参考策略 forward + -> reinforce_baseline advantage + -> balance_experiences + -> NaiveReplayBuffer -> torch DataLoader + -X PolicyTrainer / AutoModel / FSDP Actor（显式 CPU 更新 + TinyPolicy） + -> log_probs_from_logits / PolicyLoss / compute_approx_kl / agg_loss + -> loss.backward -> clip_grad_norm_ -> AdamW -> scheduler + -X FSDP 到 vLLM 的权重广播（共享一个模型对象） +``` + +`-X` 标记每一个切断的边界。本地 `ray.get` 只负责同步解包本地返回值；真正的 `wait`/队列调度和 vLLM `SamplingParams` 保持禁用。`Result`、`Trajectory` 和 `Experience` 都是 Molt 原有的数据记录，不是教程自定义字典。 + +## 有意做的小规模调整 + +- 数据集是 Hugging Face 内存中的一条数据，字段与 Qwen3 math quick start 相同，包含 `prompt` 和 `reward_model`。 +- 生产 recipe 每个 prompt 的 8 次采样缩小为 2 次。逐 token 的 inverse-CDF 抽样会确定性地产生一个 `\boxed{4}` 和一个 `\boxed{5}`，同时保留自回归策略与 log probability 的对应关系。 +- 两条 rollout 共用一个 group ID，因此真实的 `reinforce_baseline` 会产生正、负 advantage。 +- 保留 quick start 中的动态过滤、旧策略/参考策略 forward、KL-as-loss（`k2`，系数 `0.001`）、PPO clipping 和 importance-sampling correction。第一次更新时 actor 与 reference 相同，所以 KL 数值为零，但完整分支确实执行。 +- DP、TP、CP 均为 1，关闭 dynamic batching。这些设置去掉吞吐量与显存调度，不改变这里要观察的单 rank 张量语义。 +- CPU 上关闭 pinned memory。prompt loader 仍是 `StatefulDataLoader`，训练仍使用 `NaiveReplayBuffer + DataLoader`。 +- AdamW 保留 quick start 的 betas、epsilon、零 weight decay、constant scheduler 与 `max_norm=1.0`。只有学习率从 `1e-6` 提高为 `0.01`，使一次教程更新产生可见变化。 + +## 建议断点顺序 + +1. `PromptDataset.__getitem__`：tokenization 前的 prompt 渲染。 +2. `_collect_prompt_batch`：DataLoader 输出的五个字段。 +3. `StepEnvRunner.execute`：tokenizer、本地生成 transport 与真实环境 step。 +4. `SamplesGenerator._filter_group`：rollout 完整性，以及均值为 `0.5` 的分组通过动态过滤。 +5. `SamplesGenerator._process_response_into_experience`：token 轴 `T` 如何变为 next-token 轴 `T-1`。 +6. `RemoteExperienceMaker.make_experience`：本地旧策略/参考策略 forward 与 log probability 字段。 +7. `RemoteExperienceMaker.compute_advantages_and_returns`：reward 如何变为正、负 advantage。 +8. `NaiveReplayBuffer.append/collate_fn`：rollout batch 如何变为训练 microbatch。 +9. `PolicyLoss.forward`，然后是 `compute_approx_kl/agg_loss`：PPO、importance correction 与 KL-as-loss。 +10. `total_loss.backward()`：梯度、裁剪、AdamW、scheduler 与 zero-grad。 + +## 验证情况 + +教程已在 macOS arm64、Python 3.12 的全新环境中验证，该环境没有安装 `molt`、Ray 或 vLLM。`pip check`、在线 tokenizer 初始化、缓存后的离线执行以及端到端断言均通过。Linux 依赖解析会选择 PyTorch `2.11.0+cpu` wheel，仍建议在 CI 中增加 Linux runtime smoke test。 + +完整 Molt 测试环境仍需要生产依赖 vLLM。此处的 requirements 只服务于 CPU 教程，不替代整个仓库的开发环境。 + +## 后续计划 + +计划中的下一步是围绕这条 CPU 路径提供一套 coding-agent skill，工作流为： + +1. 在 CPU 教程中实现并观察轻量修改； +2. 在本地追踪受影响的 Molt 数据记录、张量、分支与指标； +3. 把修改迁移到对应的 GPU/分布式路径； +4. 只在服务器上验证剩余的分布式与硬件行为。 + +CPU 路径与生产代码保持相近结构，可以让用户和 coding agent 更容易完成这一步迁移。 diff --git a/examples/tutorial/cpu_rl_walkthrough.py b/examples/tutorial/cpu_rl_walkthrough.py new file mode 100644 index 0000000..6b9afed --- /dev/null +++ b/examples/tutorial/cpu_rl_walkthrough.py @@ -0,0 +1,458 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Run Molt's real RL data path with CPU replacements for Ray and the GPU training stack.""" + +# ruff: noqa: E402 + +import asyncio +import copy +import sys +from collections import defaultdict +from pathlib import Path +from pprint import pprint +from types import ModuleType, SimpleNamespace +from unittest.mock import Mock + +import torch +from datasets import Dataset +from huggingface_hub import snapshot_download +from torch.utils.data import DataLoader +from torchdata.stateful_dataloader import StatefulDataLoader + +repo_root = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(repo_root)) + +import molt + +# The production package exports GPU model classes eagerly. Keep its package path so +# the tutorial imports the original CPU-safe loss and utility modules directly. +models_package = ModuleType("molt.models") +models_package.__file__ = str(repo_root / "molt/models/__init__.py") +models_package.__package__ = "molt.models" +models_package.__path__ = [str(repo_root / "molt/models")] +sys.modules["molt.models"] = models_package +molt.models = models_package + +# Rollout code imports Ray/vLLM at module scope. Local ``get`` unwraps synchronous +# results; queue scheduling and SamplingParams construction still fail immediately. +ray = ModuleType("ray") +ray.get = lambda value: value +ray.wait = Mock(side_effect=RuntimeError("The CPU tutorial does not run Ray scheduling.")) +ray.__cpu_tutorial_stub__ = True +sys.modules["ray"] = ray + +vllm = ModuleType("vllm") +vllm.SamplingParams = Mock(side_effect=RuntimeError("The CPU tutorial does not run vLLM dispatch.")) +vllm.__cpu_tutorial_stub__ = True +sys.modules["vllm"] = vllm + +from molt.agents.base import load_agent_runner +from molt.datasets import PromptDataset +from molt.models.loss import PolicyLoss, agg_loss +from molt.models.utils import compute_approx_kl, log_probs_from_logits +from molt.trainer.algorithm import FixedKLController, NaiveReplayBuffer +from molt.trainer.algorithm.experience import balance_experiences +from molt.trainer.rollout.experience_maker import RemoteExperienceMaker +from molt.trainer.rollout.samples_generator import SamplesGenerator, _collect_prompt_batch +from molt.utils import get_tokenizer + + +class TinyPolicy(torch.nn.Module): + """CPU replacement for AutoModel/FSDP with Actor-compatible outputs.""" + + def __init__(self, vocab_size, transition_sequences, hidden_size=16, target_logit=20.0): + super().__init__() + self.embedding = torch.nn.Embedding(vocab_size, hidden_size) + self.cls = torch.nn.Linear(hidden_size, vocab_size) + + transitions = {} + for sequence in transition_sequences: + for previous, target in zip(sequence, sequence[1:]): + transitions.setdefault(previous, set()).add(target) + if len(transitions) > hidden_size: + raise ValueError("TinyPolicy hidden_size is too small for the requested token transitions") + with torch.no_grad(): + self.embedding.weight.zero_() + self.cls.weight.zero_() + self.cls.bias.zero_() + for hidden_index, (previous, targets) in enumerate(transitions.items()): + self.embedding.weight[previous, hidden_index] = 1.0 + for target in targets: + self.cls.weight[target, hidden_index] = target_logit + + def forward(self, sequences, action_mask=None, attention_mask=None): + del attention_mask + logits = self.cls(self.embedding(sequences)) + next_tokens = torch.roll(sequences, shifts=-1, dims=1) + log_probs = log_probs_from_logits(logits, next_tokens)[:, :-1] + output = {"logits": logits, "log_probs": log_probs} + if action_mask is not None: + output["action_log_probs"] = log_probs[:, -action_mask.shape[1] :] * action_mask.float() + return output + + @torch.no_grad() + def async_run_method_batch( + self, + method_name, + sequences, + action_mask, + attention_mask, + mm_train_inputs_list=None, + routed_experts=None, + **kwargs, + ): + """Synchronous stand-in for RayActorGroup's batched model forward.""" + del mm_train_inputs_list, routed_experts + if method_name != "forward" or kwargs: + raise RuntimeError(f"Unsupported local model call: {method_name}, extras={list(kwargs)}") + return [ + [ + self(sequence, mask, attention_mask=attention)["action_log_probs"].cpu() + for sequence, mask, attention in zip(sequences, action_mask, attention_mask) + ] + ] + + +class TinyVllmEngine: + """Duck-typed vLLM engine that uses the shared TinyPolicy on CPU.""" + + def __init__(self, model, tokenizer, uniform_sequences): + self.model = model + self.tokenizer = tokenizer + self.uniform_sequences = iter(uniform_sequences) + + async def generate(self, prompt_token_ids, sampling_params, multi_modal_data=None, session_id=None): + del multi_modal_data, session_id + uniforms = iter(next(self.uniform_sequences)) + context = list(prompt_token_ids) + answer_ids = [] + generation_log_probs = [] + + with torch.no_grad(): + for _ in range(sampling_params.max_tokens): + uniform = next(uniforms) + sequences = torch.tensor([context], dtype=torch.long) + next_logits = self.model(sequences)["logits"][0, -1] / sampling_params.temperature + next_log_probs = next_logits.log_softmax(dim=-1) + cdf = next_log_probs.exp().cumsum(dim=-1) + token_id = torch.searchsorted(cdf, torch.tensor(uniform)).clamp_max(cdf.numel() - 1).item() + answer_ids.append(token_id) + generation_log_probs.append({token_id: SimpleNamespace(logprob=next_log_probs[token_id].item())}) + context.append(token_id) + + generation = SimpleNamespace( + token_ids=answer_ids, + text="", + finish_reason="length", + logprobs=generation_log_probs, + routed_experts=None, + ) + return SimpleNamespace(outputs=[generation], prompt_routed_experts=None), 0 + + @torch.no_grad() + def mean_answer_log_probability(self, prompt_token_ids, answer): + answer_ids = self.tokenizer(answer, add_special_tokens=False)["input_ids"] + sequences = torch.tensor([list(prompt_token_ids) + answer_ids], dtype=torch.long) + first_action_step = len(prompt_token_ids) - 1 + action_log_probs = self.model(sequences)["log_probs"][0, first_action_step:] + return action_log_probs.mean().item() + + +torch.manual_seed(7) +device = torch.device("cpu") +max_length = 256 +correct_answer = r"\boxed{4}" +wrong_answer = r"\boxed{5}" + +# These are real Molt config branches, reduced to one CPU rank and two rollouts. +args = SimpleNamespace( + data=SimpleNamespace( + input_key="prompt", + label_key="reward_model", + tools_key="tools", + image_key="images", + apply_chat_template=True, + ), + train=SimpleNamespace( + dynamic_batch_enable=False, + force_on_policy=False, + colocate_fsdp_models=False, + ), + rollout=SimpleNamespace(batch_size=1, micro_batch_size=1, n_samples_per_prompt=2), + actor=SimpleNamespace(num_nodes=1, num_gpus_per_node=1), + fsdp=SimpleNamespace(cp_size=1, tp_size=1), + algo=SimpleNamespace( + dynamic_filtering_enable=True, + dynamic_filtering_range=(0.01, 0.99), + advantage=SimpleNamespace( + estimator="reinforce_baseline", + gamma=1.0, + lam=1.0, + no_whiten=False, + ), + kl=SimpleNamespace(init_coef=0.001, use_loss=True, estimator="k2"), + ), + reward=SimpleNamespace(clip_range=(-10.0, 10.0)), +) +strategy = SimpleNamespace(args=args) +tokenizer_revision = "c1899de289a04d12100db370d81485cdf75e47ca" +tokenizer_path = snapshot_download( + "Qwen/Qwen3-0.6B", + revision=tokenizer_revision, + allow_patterns=["config.json", "tokenizer*", "merges.txt", "vocab.json"], +) +tokenizer = get_tokenizer(tokenizer_path, model=None, padding_side="left") + +print("\n1. Real PromptDataset and the same StatefulDataLoader class as rl_trainer.prepare_datasets") +rows = Dataset.from_list( + [ + { + "datasource": "local_math", + "prompt": [{"role": "user", "content": r"What is 2 + 2? Answer with \boxed{}."}], + "reward_model": {"ground_truth": "4", "style": "rule"}, + } + ] +) +runner = load_agent_runner(str(repo_root / "examples/python/agents/math.py")) +prompt_dataset = PromptDataset(rows, tokenizer, strategy, prerender=runner.PRERENDER_PROMPT) +prompt_dataloader = StatefulDataLoader( + prompt_dataset, + batch_size=1, + pin_memory=False, + shuffle=True, + drop_last=True, + collate_fn=prompt_dataset.collate_fn, + num_workers=0, +) +loader_batch = next(iter(prompt_dataloader)) +pprint(dict(zip(("datasources", "prompts", "labels", "images", "tools"), loader_batch))) + +print("\n2. Real SamplesGenerator prompt-batch collector") +prompts, labels, images, tools, exhausted = _collect_prompt_batch(iter([loader_batch]), num_prompts=1) +pprint({"prompt": prompts[0], "label": labels[0], "exhausted": exhausted}) + +print("\n3. Tiny CPU model and fake vLLM transport; the model object is shared with training") +prompt_token_ids = tokenizer(prompts[0], add_special_tokens=False, return_tensors="pt")["input_ids"][0].tolist() +correct_answer_ids = tokenizer(correct_answer, add_special_tokens=False)["input_ids"] +wrong_answer_ids = tokenizer(wrong_answer, add_special_tokens=False)["input_ids"] +model = TinyPolicy( + vocab_size=len(tokenizer), + transition_sequences=[prompt_token_ids[-1:] + correct_answer_ids, prompt_token_ids[-1:] + wrong_answer_ids], +).to(device) +reference_model = copy.deepcopy(model).eval().requires_grad_(False) +engine = TinyVllmEngine( + model, + tokenizer, + uniform_sequences=[ + [0.10, 0.30, 0.50, 0.25, 0.70], + [0.20, 0.40, 0.60, 0.75, 0.80], + ], +) +margin_before = engine.mean_answer_log_probability( + prompt_token_ids, correct_answer +) - engine.mean_answer_log_probability(prompt_token_ids, wrong_answer) +pprint({name: tuple(parameter.shape) for name, parameter in model.named_parameters()}) + +print("\n4. Real StepEnvRunner.execute -> real MathEnv.step -> real Result -> real Trajectory") +sampling_params = SimpleNamespace( + max_tokens=len(correct_answer_ids), + min_tokens=1, + logprobs=1, + temperature=1.0, + top_p=1.0, + top_k=-1, +) +trajectories = [] +for rollout_index in range(args.rollout.n_samples_per_prompt): + trajectory = asyncio.run( + runner.execute( + prompt=prompts[0], + label=labels[0], + sampling_params=sampling_params, + max_length=max_length, + hf_tokenizer=tokenizer, + llm_engine=engine, + images=images[0], + tools=tools[0], + ) + ) + # AgentRunnerActor.run_group normally adds these ids after Ray returns. + trajectory.group_id = "local-prompt-0" + trajectory.rollout_id = f"local-rollout-{rollout_index}" + trajectories.append(trajectory) + pprint( + { + "rollout_id": trajectory.rollout_id, + "answer": tokenizer.decode( + trajectory.observation_tokens[trajectory.action_ranges[0][0] :], skip_special_tokens=False + ), + "reward": trajectory.reward, + "truncated": trajectory.truncated, + "action_ranges": trajectory.action_ranges, + "num_tokens": len(trajectory.observation_tokens), + } + ) + +print("\n5. Real SamplesGenerator group completeness, dynamic filter, and Trajectory -> Experience conversion") +samples_generator = SamplesGenerator(strategy, prompt_dataloader, None, tokenizer, agent_runners=[]) +drop_counts = defaultdict(int) +rollout_samples = samples_generator._filter_group( + trajectories, + dynamic_filtering=args.algo.dynamic_filtering_enable, + drop_counts=drop_counts, + max_len=max_length, + n_samples_per_prompt=args.rollout.n_samples_per_prompt, +) +assert not drop_counts +assert len({sample.rollout_ids[0] for sample in rollout_samples}) == args.rollout.n_samples_per_prompt +for sample in rollout_samples: + pprint( + { + "sequences": tuple(sample.sequences.shape), + "action_mask": tuple(sample.action_mask.shape), + "rollout_log_probs": tuple(sample.rollout_log_probs.shape), + "reward": sample.rewards.tolist(), + "truncated": sample.truncated.tolist(), + } + ) + +print("\n6. Real RemoteExperienceMaker old-policy/reference forwards, KL fields, and grouped advantage") +experience_maker = RemoteExperienceMaker( + actor_model_group=model, + initial_model_group=reference_model, + kl_controller=FixedKLController(args.algo.kl.init_coef), + strategy=strategy, + tokenizer=tokenizer, +) +experiences = experience_maker.build_experiences(rollout_samples) +for experience in experiences: + pprint( + { + "index": experience.index, + "reward": experience.rewards.tolist(), + "masked_advantage": experience.advantages[experience.action_mask].unique().tolist(), + "old_action_log_probs": tuple(experience.action_log_probs.shape), + "base_action_log_probs": tuple(experience.base_action_log_probs.shape), + } + ) + +print("\n7. Real single-rank balancing -> NaiveReplayBuffer -> training DataLoader") +balanced_experiences = balance_experiences(experiences, args) +replay_buffer = NaiveReplayBuffer(sample_batch_size=1, cpu_offload=True, dynamic_batch=False) +for experience in balanced_experiences: + replay_buffer.append(experience) +train_dataloader = DataLoader( + replay_buffer, + batch_size=replay_buffer.sample_batch_size, + shuffle=True, + drop_last=True, + pin_memory=False, + collate_fn=replay_buffer.collate_fn, +) +microbatches = list(train_dataloader) +batch_num_tokens = torch.stack([experience.action_mask.sum() for experience in microbatches]).sum() +pprint( + {"buffer_items": len(replay_buffer), "microbatches": len(microbatches), "action_tokens": batch_num_tokens.item()} +) + +print("\n8. Tiny Actor forward -> real PolicyLoss + KL loss -> autograd -> AdamW/scheduler") +# The recipe uses 1e-6; this one-step tutorial enlarges it so the update is visible. +optimizer = torch.optim.AdamW(model.parameters(), lr=0.01, betas=(0.9, 0.95), eps=1e-8, weight_decay=0.0) +scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lambda _: 1.0) +loss_fn = PolicyLoss( + clip_eps_low=0.2, + clip_eps_high=0.27, + dual_clip=10.0, + is_correction_level="geo", + is_correction_threshold=[0.99, 1.01], +) +optimizer.zero_grad(set_to_none=True) +loss_rows = [] +for experience in microbatches: + experience.to_device(device) + model_output = model( + experience.sequences, + experience.action_mask, + attention_mask=experience.attention_mask, + ) + action_log_probs = model_output["action_log_probs"] + old_action_log_probs = experience.action_log_probs + if old_action_log_probs is None: + old_action_log_probs = action_log_probs.detach() + actor_loss, reported_loss, clip_ratio, policy_kl, vllm_kl, filter_ratio = loss_fn( + action_log_probs, + old_action_log_probs, + experience.advantages, + action_mask=experience.action_mask, + rollout_log_probs=experience.rollout_log_probs, + dp_size=1, + batch_num_tokens=batch_num_tokens, + ) + approx_kl = compute_approx_kl( + action_log_probs, + experience.base_action_log_probs, + kl_estimator=args.algo.kl.estimator, + ) + kl_loss = agg_loss( + approx_kl, + experience.action_mask, + "token-mean", + dp_size=1, + batch_num_tokens=batch_num_tokens, + ) + total_loss = actor_loss + kl_loss * args.algo.kl.init_coef + total_loss.backward() + loss_rows.append( + { + "policy_loss_contribution": actor_loss.item(), + "kl_loss_contribution": (kl_loss * args.algo.kl.init_coef).item(), + "total_loss_contribution": total_loss.item(), + "reported_loss": reported_loss.item(), + "clip_ratio": clip_ratio.item(), + "policy_kl": policy_kl.item(), + "vllm_kl": None if vllm_kl is None else vllm_kl.item(), + "filter_ratio": None if filter_ratio is None else filter_ratio.item(), + } + ) +pprint(loss_rows) +grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) +pprint( + { + "total_grad_norm_before_clip": grad_norm.item(), + "learning_rate": scheduler.get_last_lr()[0], + "parameter_gradients": { + name: {"shape": tuple(parameter.grad.shape), "norm": parameter.grad.norm().item()} + for name, parameter in model.named_parameters() + }, + } +) +optimizer.step() +scheduler.step() +optimizer.zero_grad(set_to_none=True) + +print("\n9. Shared model replaces FSDP -> vLLM weight broadcast in this one-process tutorial") +margin_after = engine.mean_answer_log_probability( + prompt_token_ids, correct_answer +) - engine.mean_answer_log_probability(prompt_token_ids, wrong_answer) +pprint({"correct_minus_wrong_logprob_before": margin_before, "correct_minus_wrong_logprob_after": margin_after}) + +assert [trajectory.reward for trajectory in trajectories] == [1.0, 0.0] +assert all(trajectory.truncated for trajectory in trajectories) +assert all(sample.sequences.shape[1] == sample.action_mask.shape[1] + 1 for sample in rollout_samples) +assert margin_after > margin_before +assert sys.modules["ray"].__cpu_tutorial_stub__ +assert sys.modules["vllm"].__cpu_tutorial_stub__ +print("\nDONE: Ray scheduling and the GPU model/trainer stack were replaced; the RL records and math stayed real.\n") diff --git a/examples/tutorial/requirements-cpu.txt b/examples/tutorial/requirements-cpu.txt new file mode 100644 index 0000000..7a27a88 --- /dev/null +++ b/examples/tutorial/requirements-cpu.txt @@ -0,0 +1,19 @@ +# Based on the CPU-usable subset of ../../requirements.txt, pinned to the +# macOS arm64 / Python 3.12 environment used to verify this walkthrough. +--extra-index-url https://download.pytorch.org/whl/cpu + +# Ray, vLLM, vLLM-router, NeMo AutoModel, and other GPU runtimes are intentionally omitted. +aiohttp==3.14.1 +datasets==5.0.0 +fastapi==0.139.0 +huggingface-hub==1.23.0 +numpy==2.5.1 +pillow==12.3.0 +pylatexenc==2.10 +socksio==1.0.0 +sympy==1.14.0 +torch==2.11.0 +torchdata==0.11.0 +tqdm==4.68.4 +transformers==5.13.1 +uvicorn==0.51.0