test: cover the hybrid-attention VLM architecture (Qwen35) - #51
Merged
Conversation
Follow-up to #49/#50. MLXVLM/Models/Qwen35.swift was, alongside LFM2VL.swift, one of the largest remaining zero-coverage files in the model layer. Its language model is a hybrid: layers alternate between full self-attention and a linear-attention block (GatedDeltaNet) on a fixed period, full_attention_interval. DecoderLayer.isLinear is (layerIdx + 1) % fullAttentionInterval != 0, so a config that never turns the linear branch on would leave GatedDeltaNet entirely dead code. Setting the interval to the layer count (2/2) makes layer 0 linear and layer 1 full attention, exercising both branches deliberately rather than by chance. Vision config reuses Qwen3VLConfiguration.VisionConfiguration directly (Qwen35 is literally typealiased to it), so it inherits the same two silent constraints already covered in #50: num_position_embeddings must be a perfect square, and hidden_size must divide evenly by num_heads. The forward-pass test uses the model's own newCache(parameters:), not nil — that is what routes the linear layer to a MambaCache and the attention layer to a standard KV cache, which is model-specific behaviour worth exercising against the same config that built the model rather than assuming it lines up. MoE fields (num_experts, decoder_sparse_step, ...) are left at their dense defaults; the routed-MoE path is a separate shape from the attention hybrid this covers. Verified with coverage: MLXVLM/Models/Qwen35.swift moves from 0% to 66.4% line coverage. Full suite: 120 tests, 21 suites, passing (112 before, +8 across this PR and its sibling for LFM2VL). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solderzzc
added a commit
to SharpAI/SwiftLM
that referenced
this pull request
Aug 15, 2026
…149) Points at c09851f, which carries both: - SharpAI/mlx-swift-lm#51 — the hybrid full-attention/GatedDeltaNet split in MLXVLM/Models/Qwen35.swift, gated by a fixed period (full_attention_interval) - SharpAI/mlx-swift-lm#52 — the hybrid attention/short-conv split in MLXVLM/Models/LFM2VL.swift, gated by explicit layer indices (full_attn_idxs) Both were, alongside Gemma4 (#147) and Qwen3VL (#148), among the largest zero-coverage files in the model layer. Coverage: Qwen35 0% → 66.4%, LFM2VL 0% → 34.9%. No SwiftLM-side code changes; verified the umbrella builds clean against the bumped pointer. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to #(LFM2VL PR, opening next). Follow-up to #49/#50 —
MLXVLM/Models/Qwen35.swiftwas one of the largest remaining zero-coverage files in the model layer.What's different about it
Its language model is a hybrid: layers alternate between full self-attention and a linear-attention block (
GatedDeltaNet) on a fixed period,full_attention_interval.DecoderLayer.isLinearis(layerIdx + 1) % fullAttentionInterval != 0, so a config that never turns the linear branch on leavesGatedDeltaNetentirely dead code in the test. Setting the interval to the layer count (2/2) makes layer 0 linear and layer 1 full attention — both branches exercised on purpose, not by chance.Vision config reuses
Qwen3VLConfiguration.VisionConfigurationdirectly (Qwen35 is literally typealiased to it), so it inherits the two silent constraints already covered in #50:num_position_embeddingsmust be a perfect square,hidden_sizemust divide evenly bynum_heads.The forward-pass test uses the model's own
newCache(parameters:), notnil— that's what routes the linear layer to aMambaCacheand the attention layer to a standard KV cache, so it checks construction and cache selection actually agree rather than assuming it.MoE fields (
num_experts, ...) stay at their dense defaults — the routed-MoE path is a separate shape from the attention hybrid this covers.Verification
🤖 Generated with Claude Code