fix(tests): gemma4 assistant mtp fixture - #369
Draft
a-ghorbani wants to merge 3 commits into
Draft
a-ghorbani wants to merge 3 commits into
a-ghorbani wants to merge 3 commits into
Conversation
…of skip on missing MTP support The gemma4 fixture entry claimed mtp=true, but download.sh never fetched the separate gemma4-assistant draft gguf that assistant-type (mem-shared) MTP requires — the harness only ever exercised that branch on machines where the file had been placed by hand. Worse, an MTP init failure was recorded as a passing 'skipped' check, so the suite stayed green while mem-shared MTP went untested everywhere else. - download.sh: fetch ggml-org/gemma-4-E2B-it-GGUF mtp-gemma-4-E2B-it-Q8_0.gguf as gemma4.assistant.gguf (the name the tests auto-wire). ggml-org's mtp-* files are the canonical llama.cpp conversion of Google's safetensors-only assistant weights; spec entries can now carry their own repo (repo|file). - kv_cache_reuse_test: draft-init failure on an mtp=true model is now a FAIL with a re-download hint instead of a skip marked as pass. Verified: 9/9 MTP checks with the pinned draft + bartowski Q4_K_M target (draft acceptance 0.66-0.89); missing assistant now exits 1.
loadModel unconditionally overwrote the draft's devices with speculative.draft.devices, which is always empty (not settable from JS). A CPU-only target override (devices: ['CPU']) was therefore reset to the default device list for the draft, whose layers (n_gpu_layers default -1) then landed on the GPU — on the iOS simulator this aborts in the Metal draft-decode buffer alloc while the target loads clean. Only override when the draft carries its own device list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Assistant MTP tests would pass even though the assistant draft gguf was never downloaded, so the test couldn't actually happen (MTP init failure was recorded as a passing "skipped" check).
This PR ensures the test really runs:
models/download.shnow fetchesggml-org/gemma-4-E2B-it-GGUF | mtp-gemma-4-E2B-it-Q8_0.gguf.kv_cache_reuse_testnow FAILs (exit 1, with a re-download hint) when a model markedmtp=truecan't init MTP, instead of skip-marked-as-pass.