test: add a coverage report that names where the suites are blind - #146
Merged
Conversation
Tier 4 of #128, which asked for `--enable-code-coverage` to find blind spots objectively rather than by eye. This runs it and aggregates the result into something readable: coverage per area, plus the largest files no test executes. Both suites are measured, because reading either alone misleads. From the umbrella package the model architectures look like 0%, which is not because they are untested but because their tests live in the submodule. Measured separately: SwiftLM package server 24.4%, inference core 57.7%, SwiftBuddy 5.3% mlx-swift-lm submodule LLM models 24.1%, VLM models 0.3%, MLXLMCommon 41.3% Two findings worth acting on. #128 named `MLXVLM/Models/Gemma4.swift` as having no tests despite being where SharpAI/mlx-swift-lm#45's changes landed. Measurement confirms it and adds that it is the single largest untested file in the model layer, at 1771 lines. More striking is the pattern around it: the VLM half of the model layer is at **0.3% over 13,304 lines** — Qwen3VL, Qwen35, LFM2VL, GlmOcr, Qwen25VL, Gemma3, Pixtral, FastVLM are all at zero. CI does run vision and omni end-to-end tests, so these paths are exercised; nothing touches them at unit level, which is why a shape or dtype mistake inside one surfaces only as a wrong image answer. Deliberately not wired into CI. A coverage gate rejects work for touching an already-thin file and invites tests written to raise a number, and the run costs a full debug rebuild of both packages. This is a tool to answer "where are we blind" when someone asks, not a check to pass. The script says so itself: coverage measures execution, not correctness — a covered line may be asserted on by nothing at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 15, 2026
solderzzc
added a commit
to SharpAI/mlx-swift-lm
that referenced
this pull request
Aug 15, 2026
Coverage measurement (SharpAI/SwiftLM#146) put the entire VLM half of the model layer at 0.3% over 13,304 lines, with zero test files under Tests/MLXLMTests touching MLXVLM at all. MLXVLM/Models/Gemma4.swift — where #45's changes landed, and the file issue ml-explore#128 named specifically — is 1,771 lines and none of them ran. Vision models are exercised end to end by the vision and omni CI jobs, so the gap was never "this code doesn't run" — it was that a shape or dtype fault inside one surfaces as a wrong answer about an image rather than a failing assertion, discovered downstream and expensively. Four tests, tiny random-init configs in the style of the existing Gemma4Tests: config decoding for both the text and vision halves, instantiation, a text-only forward pass (the path every request without an image attachment takes), and determinism. Establishes shape and finiteness, not numerical correctness — real checkpoints remain the only way to judge output quality. Full suite: 108 tests, 18 suites, passing. Co-authored-by: Aegis AI Assistant <simba@aegis-ai.dev> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tier 4 of #128 — which asked for
--enable-code-coverageto find blind spots objectively rather than by eye. This runs it and aggregates the output into something readable.Why both suites
Reading either alone misleads. From the umbrella package the model architectures look like 0% — not because they are untested, but because their tests live in the submodule. Measured separately:
Two findings
#128's named example is confirmed, and it's the worst case.
MLXVLM/Models/Gemma4.swift— where SharpAI/mlx-swift-lm#45's changes landed — has no tests, and at 1771 lines it is the single largest untested file in the model layer.The pattern around it is the bigger story. The VLM half of the model layer sits at 0.3% across 13,304 lines. Every one of these is at zero:
CI does run vision and omni end-to-end tests, so these paths are exercised — but nothing touches them at unit level. That is why a shape or dtype mistake inside one surfaces as a wrong answer about an image rather than as a failing assertion, which is a considerably more expensive way to find out.
Deliberately not wired into CI
A coverage gate rejects work for touching an already-thin file, and invites tests written to move a number rather than to catch anything. The run also costs a full debug rebuild of both packages. This is a tool for answering "where are we blind" when someone asks — not a check to pass.
The caveat, which the script prints itself
Coverage measures execution, not correctness. A covered line may be asserted on by nothing at all. Treat a low number as a question and a high number as no answer.
🤖 Generated with Claude Code