Model-Saver: Write the SWA pattern, 15 more architectures roundtrip - #29042
Open
ServeurpersoCom wants to merge 2 commits into
Open
ServeurpersoCom wants to merge 2 commits into
ServeurpersoCom wants to merge 2 commits into
Conversation
Add llama_model_base::load_swa_pattern(), which reads sliding_window_pattern either as one flag per layer or as a period expanded by set_swa_pattern(), and use it in every loader that reads the key as a period. These loaders silently ignored an array and applied their default period, although the converters of olmo2, gemma3n and exaone4 write arrays. The published GGUFs match the defaults, so their outputs do not change. The loaders that already accepted both forms lose their duplicated scalar-then-array block, and use their declared default period when the key is absent.
Write sliding_window_pattern as one flag per layer, nextn layers included, for every model using SWA. The array is never collapsed to a scalar, since the loaders read a scalar as a period. Also write the MLA key/value lengths and KV LoRA rank of the SWA layers, required by dots3note. This enables the saver for plamo3, gemma3, cohere2, cohere2moe, olmo2, exaone-moe, afmoe, mimo2, spark2_5, muse-glimmer, mellum, laguna, granite_swa, dots3note and maple, all passing the bit-exact roundtrip of test-llama-archs.
This was referenced Sep 17, 2026
Draft
CISC
approved these changes
Sep 17, 2026
Contributor
Author
|
I already have the follow-up for four of the five remaining architectures ready, small saver omissions only; step35 needs a bit more thought, so it gets its own PR. That splits the review into three small pieces instead of one big one. |
ServeurpersoCom
added a commit
to ServeurpersoCom/llama.cpp
that referenced
this pull request
Sep 17, 2026
Write decoder_start_token_id as uint32 like the converter and the loader, write the xielu arrays one value per layer, save tok_embd when the model has no output tensor, and save the altup and per layer projection tensors of gemma3n. Follow-up of ggml-org#29042, step35 is now the last one left to enable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Improves CI coverage: the model saver now handles the SWA pattern, so 15 more architectures get the dummy models and the roundtrip, save/load state and fusion tests that come with them.
The diff touches 21 files but 18 of them are the same one-line substitution in a loader; the whole change to read is load_swa_pattern() and the saver hunk. The CI runs the roundtrip on every architecture, and a full local run of test-llama-archs on CPU and CUDA gives 254 roundtrips green with no failure.
Additional information
The missing sliding_window_pattern key is the usual reason an architecture ends up excluded from the saver (#27000 review), it got the line dropped from #25505 in review, and #24423 and #25731 are about to grow the exclusion list for the same reason.
The saver writes the pattern as one flag per layer and never collapses it to a scalar, since the loaders read a scalar as a period. The loaders that only understood a period now go through a shared load_swa_pattern() helper that accepts either form, which also fixes them silently dropping the arrays the olmo2, gemma3n and exaone4 converters already write (the published GGUFs match the default period, so no output changes). Three loaders lose their duplicated scalar-then-array block.
Enables plamo3, gemma3, cohere2, cohere2moe, olmo2, exaone-moe, afmoe, mimo2, spark2_5, muse-glimmer, mellum, laguna, granite_swa, dots3note and maple, all bit-exact on the test-llama-archs roundtrip. The 5 still excluded (gemma3n, bitnet, t5, apertus, step35) fail for unrelated reasons, each a small follow-up.
Requirements