[Feature Request] Add official Qwen3.5 Sokoban agentic example

[yingyibiao]

Hi ROLL team,

Since support for Qwen3.5 has been added, could you please provide an official example for the Sokoban agentic pipeline (similar to the existing qwen2.5-vl-3B-agentic one)?

Specifically, a reference script like run_agentic_pipeline_sokoban.sh optimized for Qwen3.5 would be extremely helpful for the community to:

Verify the integration of the new model in RL environments.

Have a baseline configuration for training/evaluating Qwen3.5 on agentic tasks.

It would be great if this could be included in the examples/ directory.

Thanks!

[PanAndy]

The parallel example configuration for Qwen3.5 is provided at https://github.com/alibaba/ROLL/blob/main/examples/qwen3.5-27B-rlvr_megatron/rlvr_megatron_80GB.yaml and https://github.com/alibaba/ROLL/blob/main/examples/qwen3.5-35BA3-rlvr_megatron/rlvr_megatron_80GB.yaml.

The agentic example only has specific configurations for agentic trajectory sampling; there is nothing special about the parallel configuration for each role—just adjust as needed.

[yingyibiao]

Hi @PanAndy and the ROLL team,

I understand the parallel configuration part. However, when adapting the existing qwen2.5-vl-3B-agentic example to Qwen3.5-4B for the Sokoban agentic pipeline, I encountered several Qwen3.5-specific issues that are not just about parallel config. I'd like to share them in case they help the team or other users:

1. Qwen3_5Config missing pad_token_id attribute

File: roll/models/model_providers.py (lines 518, 609, 680)

Issue: Qwen3_5Config doesn't have pad_token_id as a standard attribute, causing AttributeError on model.config.pad_token_id is None.

Fix: We fixed it with getattr(model.config, "pad_token_id", None).

2. ValueError: 760 is not in list in token_mask_utils.py

File: roll/pipeline/agentic/env_manager/token_mask_utils.py

Issue: Qwen3.5's chat template automatically strips thinking part from intermediate assistant turns (those before the last user query). The original code uses tokenizer.encode(message["content"]) to find the first content token in token_ids, but since the thinking part has been stripped by the chat template, the token doesn't exist in token_ids.

3. mm_token_type_ids causes tensor concat failure

File: roll/datasets/collator.py

Issue: Qwen3.5 processor outputs a new field mm_token_type_ids with shape (1, seq_len). This field is not in padded_keys and not removed, so it ends up in multi_modal_inputs.

Problem: During training, torch.concat(dim=0) fails because different samples have different seq_len.

Fix: We fixed it by adding mm_token_type_ids to the removal list alongside position_ids and rope_deltas.

4. Thinking mode not controllable in agentic pipeline

Issue: The data_args.template config (e.g., qwen3_nothink) is not used by the agentic pipeline — vl_traj_env_manager.py calls tokenizer.apply_chat_template() directly without passing enable_thinking.

Impact: This means there's no way to control thinking mode from the YAML config for agentic tasks. With default thinking mode enabled and max_tokens_per_step: 64, the thinking content consumes most of the token budget, leaving insufficient room for the actual answer output.

A minimal working example for Qwen3.5 + Sokoban would save a lot of debugging effort for the community.

[SF-evil]

I fix 2 and 4 by modifying the code in vl_traj_env_manager.py as follows:

enable_thinking = self.env_config["enable_thinking"] if "enable_thinking" in self.env_config else False
lm_input_texts = self.tokenizer.apply_chat_template(
    messages, add_generation_prompt=add_generation_prompt, tokenize=False, enable_thinking=enable_thinking
)

Then I can control the thinking mode directly from the YAML configuration file by setting:

enable_thinking: false

[yingyibiao]

@SF-evil Thanks for your reply! Have you tried to use the thinking mode of Qwen3.5 to run_agentic_pipeline_sokoban.sh? Did you encounter any problems?

[SF-evil]

I encountered an issue when using Qwen 3.5's thinking mode with run_agentic_pipeline_sokoban.sh. As you mentioned, the model generates a large amount of thinking content, making it nearly impossible to see the <answer>. Additionally, it’s difficult to tune the max_tokens_per_step parameter to prevent the <answer> from being truncated.

[yingyibiao]

@SF-evil Yes, I encountered the same problem when training the thinking mode of Qwen 3.5.
See the reward plot below:

[yingyibiao]

I fix 2 and 4 by modifying the code in vl_traj_env_manager.py as follows:

enable_thinking = self.env_config["enable_thinking"] if "enable_thinking" in self.env_config else False
lm_input_texts = self.tokenizer.apply_chat_template(
    messages, add_generation_prompt=add_generation_prompt, tokenize=False, enable_thinking=enable_thinking
)

Then I can control the thinking mode directly from the YAML configuration file by setting:

enable_thinking: false

I'm new to ROLL framework and agentic rl training.

I just want to make sure that your modification is also valid to solve problem 2 & 4 when applied to the thinking mode of Qwen 3.5. And what's your solution to problem 1 & 3, is it same to my fix?

[SF-evil]

Yes, it is valid.
The solution for Problem 1 is the same as yours. But I haven’t encountered Problem 3 myself.

[yingyibiao]

Yes, it is valid. The solution for Problem 1 is the same as yours. But I haven’t encountered Problem 3 myself.

I have tried your solution but still encountered Problem 2. Which version (commit) of ROLL are you using?

[SF-evil]

345edea

The parameters max_tokens_per_step and enable_thinking should be at the same configuration level. You may check whether the model output still contains the thinking-related content.