test: add the first unit tests for anything under MLXVLM - #49
Merged
Conversation
Coverage measurement (SharpAI/SwiftLM#146) put the entire VLM half of the model layer at 0.3% over 13,304 lines, with zero test files under Tests/MLXLMTests touching MLXVLM at all. MLXVLM/Models/Gemma4.swift — where #45's changes landed, and the file issue ml-explore#128 named specifically — is 1,771 lines and none of them ran. Vision models are exercised end to end by the vision and omni CI jobs, so the gap was never "this code doesn't run" — it was that a shape or dtype fault inside one surfaces as a wrong answer about an image rather than a failing assertion, discovered downstream and expensively. Four tests, tiny random-init configs in the style of the existing Gemma4Tests: config decoding for both the text and vision halves, instantiation, a text-only forward pass (the path every request without an image attachment takes), and determinism. Establishes shape and finiteness, not numerical correctness — real checkpoints remain the only way to judge output quality. Full suite: 108 tests, 18 suites, passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solderzzc
added a commit
to SharpAI/SwiftLM
that referenced
this pull request
Aug 15, 2026
Points at acb6020 (SharpAI/mlx-swift-lm#49), which adds VLMGemma4Tests — the first unit tests for anything under MLXVLM. Measured coverage had that half of the model layer at 0.3% over 13,304 lines; this is the first shape/config/ forward-pass test against MLXVLM/Models/Gemma4.swift, the largest untested file in the model layer and where SharpAI/mlx-swift-lm#45's changes landed. No SwiftLM-side code changes; verified the umbrella builds clean against the bumped pointer. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
solderzzc
added a commit
that referenced
this pull request
Aug 15, 2026
Follow-up to #49. Measured coverage after that PR still showed the VLM half of the model layer almost entirely untested; Qwen3VL.swift (1770 lines) was the new largest zero-coverage file in the model layer. Its config shape differs from Gemma4's in a way worth testing on its own terms: text and vision decode as two independent nested structs (Qwen3VLConfiguration.TextConfiguration / .VisionConfiguration) rather than one flat shape sharing fields, so a misplaced field looks like a wrong nesting level rather than a missing top-level key. The vision tower also has two constraints a hand-written config can violate silently — numPositionEmbeddings must be a perfect square (a grid side is derived via sqrt) and hiddenSize must divide evenly by numHeads — both exercised here by construction succeeding at all. Four tests: config decoding across both nested structs, instantiation, text-only forward pass shape (the vision tower is built at init but must not be required for a plain-text request, which is what most chat traffic is), determinism. Verified with coverage: Qwen3VL.swift moves from 0% to 26.0% line coverage. Full suite: 112 tests, 19 suites, passing (108 before, +4 here). Co-authored-by: Aegis AI Assistant <simba@aegis-ai.dev> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 15, 2026
solderzzc
added a commit
that referenced
this pull request
Aug 15, 2026
Follow-up to #49/#50. MLXVLM/Models/Qwen35.swift was, alongside LFM2VL.swift, one of the largest remaining zero-coverage files in the model layer. Its language model is a hybrid: layers alternate between full self-attention and a linear-attention block (GatedDeltaNet) on a fixed period, full_attention_interval. DecoderLayer.isLinear is (layerIdx + 1) % fullAttentionInterval != 0, so a config that never turns the linear branch on would leave GatedDeltaNet entirely dead code. Setting the interval to the layer count (2/2) makes layer 0 linear and layer 1 full attention, exercising both branches deliberately rather than by chance. Vision config reuses Qwen3VLConfiguration.VisionConfiguration directly (Qwen35 is literally typealiased to it), so it inherits the same two silent constraints already covered in #50: num_position_embeddings must be a perfect square, and hidden_size must divide evenly by num_heads. The forward-pass test uses the model's own newCache(parameters:), not nil — that is what routes the linear layer to a MambaCache and the attention layer to a standard KV cache, which is model-specific behaviour worth exercising against the same config that built the model rather than assuming it lines up. MoE fields (num_experts, decoder_sparse_step, ...) are left at their dense defaults; the routed-MoE path is a separate shape from the attention hybrid this covers. Verified with coverage: MLXVLM/Models/Qwen35.swift moves from 0% to 66.4% line coverage. Full suite: 120 tests, 21 suites, passing (112 before, +8 across this PR and its sibling for LFM2VL). Co-authored-by: Aegis AI Assistant <simba@aegis-ai.dev> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
solderzzc
added a commit
that referenced
this pull request
Aug 15, 2026
Follow-up to #49/#50. MLXVLM/Models/LFM2VL.swift was, alongside Qwen35.swift, one of the largest remaining zero-coverage files in the model layer. Its hybrid mechanism differs from Qwen35's fixed-period one: layers are attention only at explicit indices, full_attn_idxs (or derived from layer_types), and a short causal convolution (LFM2ShortConv) everywhere else — a sparser, more arbitrary pattern than a modulus. Setting full_attn_idxs: [1] over 2 layers makes layer 0 the conv block and layer 1 the attention block, so both branches of LFM2DecoderLayer build and run. newCache reads fullAttnIdxs independently of how the layers were built to decide MambaCache versus KVCacheSimple per layer, so the forward-pass test uses it directly (model.newCache(parameters: nil)) rather than assuming construction and cache selection stay in agreement — that agreement is exactly what this exercises. The vision half carries the same silent constraint as Qwen3VL's tower (#50): num_patches must be a perfect square, since a grid side is derived via sqrt. Verified with coverage: MLXVLM/Models/LFM2VL.swift moves from 0% to 34.9% line coverage. Full suite: 120 tests, 21 suites, passing (112 before, +8 across this PR and its sibling for Qwen35). Co-authored-by: Aegis AI Assistant <simba@aegis-ai.dev> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Coverage measurement (SharpAI/SwiftLM#146) put the whole VLM half of the model layer at 0.3% over 13,304 lines, with zero test files under
Tests/MLXLMTeststouchingMLXVLMat all.MLXVLM/Models/Gemma4.swift— 1,771 lines, where #45's changes landed and the file SharpAI/SwiftLM#128 named specifically — had none of them run.Why this matters despite CI already running vision/omni
Those end-to-end jobs mean the code isn't dormant. What's missing is that a shape or dtype fault inside a VLM model surfaces as a wrong answer about an image, discovered downstream, rather than as a failing assertion at the point of the bug. Same failure shape ml-explore#128 was opened about.
What this adds
Four tests, tiny random-init configs in the style of the existing
Gemma4Testsfor the LLM side:text_configandvision_confighalves, including the wrapper's fallback to text-half values when top-level ones are absentThese establish shape and finiteness, not numerical correctness — the weights are random. Real checkpoints remain the only way to judge output quality; this closes the "never executed at all" gap, not the "produces the right answer" one.
Full suite: 108 tests, 18 suites, passing.
🤖 Generated with Claude Code