[Qwen3.5 MoE] Add hybrid decoder model (GatedDeltaNet + full attention + MoE)#2545
Open
gali-leilei wants to merge 2 commits into
Open
[Qwen3.5 MoE] Add hybrid decoder model (GatedDeltaNet + full attention + MoE)#2545gali-leilei wants to merge 2 commits into
gali-leilei wants to merge 2 commits into
Conversation
…n + MoE) Upstreams Qwen3.5 MoE support from the internal fork. This is a hybrid decoder: GatedDeltaNet (linear attention) for most layers + full attention every N layers, with MoE + gated shared expert FFN. ## New model: torchtitan/models/qwen3_5_moe/ - model.py: OffsetRMSNorm, RMSNormGated, GatedDeltaNet (FLA optional, torch_naive fallback), Attention (output gating + partial RoPE), TransformerBlock, Model. Uses nn.init.trunc_normal_ (no common/utils.py dep). - parallelize.py: TP/FSDP/CP/EP support. DTensor-safe wrappers for depthwise conv1d (_DTensorSafeConv1d) and FLA kernel dispatch (_install_dtensor_safe_dispatch). Registers softplus as DTensor pointwise op. - state_dict_adapter.py: HF<->TorchTitan conversion. Handles expert gate_up_proj [E,2I,D] <-> w1+w3 split/concat. - config_registry.py: debugmodel, 35b-a3b, 35b-a3b-sdpa, 35b-a3b-varlen configs. - __init__.py: model configs (debugmodel, 35b-a3b, 122b-a10b, 397b-a17b, 397B_A19B), build(), model_registry(). ## Related changes - torchtitan/models/__init__.py: add "qwen3_5_moe" to _supported_models - torchtitan/models/qwen3/config_registry.py: add qwen3_moe_30b_a3b() config - torchtitan/tools/utils.py: add L20/L20X GPU peak flops (989 TFLOPS, same as H200) - tests/unit_tests/test_qwen3_5_moe.py: construction + forward + state dict tests - tests/integration_tests/models.py: qwen3_5_moe FSDP+TP+EP integration test Validated on real 35B-A3B checkpoints (HF<->DCP roundtrip: 19/19 tensors exact) and L20X GPUs (debugmodel + 35B-A3B, 10 steps, no errors). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…py L20 flops Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gali-leilei
requested review from
fegin,
tianyu-l,
wconstab and
wwwjn
as code owners
|
Hi @gali-leilei! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
meta-cla
Bot
added
the
CLA Signed
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
12 tasks
akeshet
added a commit
to akeshet/torchtitan
that referenced
this pull request
Apr 1, 2026
…rch#2545) Back-ports the Qwen3.5 MoE model (GatedDeltaNet + full attention + MoE) from pytorch#2545 to v0.2.0 APIs. Core model math preserved; registration/parallelization adapted to TrainSpec/BaseModelArgs/ModelProtocol. Includes: model definition, parallelization (TP/EP/FSDP with DTensor-safe wrappers for GatedDeltaNet), HF state dict adapter, 5 model flavors (debugmodel through 397B), and debug training config. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Any reason this PR is not merged yet? |
Contributor
|
We are evolving Qwen3 VL to Qwen3.5 here #3371 Due to the fact that the repo is under active refactoring, we felt it'd be easier to do them together. Feedback is welcome. |
shuhuayu
added a commit
that referenced
this pull request
Jun 10, 2026
Qwen3.5 supersedes Qwen3-VL with a hybrid attention architecture: 75% GatedDeltaNet (linear attention) + 25% full attention with output gating and partial RoPE. Model changes: - Hybrid decoder with GatedDeltaNet and Qwen35Attention - Head-sharded TP on GatedDeltaNet projections (`ColwiseParallel`/`RowwiseParallel`) - OffsetRMSNorm, RMSNormGated, MoE with shared expert - Removed DeepStack Parallelisms: fsdp, tp+sp, ep, cp (text-only, full attention only), pp, verified identical logits via numerical tests (`scripts/checkpoint_conversion/numerical_tests_qwen3_5_shard.py`). Numerical parity: kl ~3e-7 against hf models (4b, multimodal) and 100% top-1/top-5 match (`scripts/checkpoint_conversion/numerical_tests_qwen3_5.py`). Many thanks to @gali-leilei for initiating the effort of enabling qwen3.5 decoder in torchtitan in #2545, some components are reused in this pr.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upstreams Qwen3.5 MoE support from the internal fork. Qwen3.5 MoE is a hybrid decoder: GatedDeltaNet (linear attention) for most layers + full attention every N layers, with MoE + gated shared expert FFN. Validated on real 35B-A3B checkpoints (HF↔DCP roundtrip: 19/19 tensors exact) and L20X GPUs (debugmodel + 35B-A3B, 10 steps, no errors).
New model:
torchtitan/models/qwen3_5_moe/OffsetRMSNorm,RMSNormGated,GatedDeltaNet(FLA optional,torch_naivefallback),Attention(output gating + partial RoPE),TransformerBlock,Model_DTensorSafeConv1d) and FLA kernel dispatch (_install_dtensor_safe_dispatch). Registerssoftplusas DTensor pointwise op.gate_up_proj [E,2I,D]↔w1+w3split/concat.debugmodel,35b-a3b,35b-a3b-sdpa,35b-a3b-varlentraining configs__init__.py: model configs (debugmodel, 35b-a3b, 122b-a10b, 397b-a17b, 397B_A19B),build(),model_registry()Related changes
torchtitan/models/__init__.py: add"qwen3_5_moe"to_supported_modelstests/integration_tests/models.py:qwen3_5_moeFSDP+TP+EP integration test (4 GPUs)Test plan
pre-commit run --all-files— flake8, ufmt, pydoclint, codespell all pass--module qwen3_5_moe --config qwen3_5_moe_debugmodelwith FSDP+TP+EP🤖 Generated with Claude Code