feat(gguf-tools, cli): add --strip-nextn for community fine-tunes and prompt support in /read - #6
Open
Trecto34 wants to merge 3 commits into
Open
Conversation
…ipping Community fine-tunes and abliterated models (e.g. Huihui-Qwen3.6-35B-A3B-abliterated) often package an extra 41st speculative MTP layer (blk.40.*), block_count = 41, and nextn_predict_layers = 1. These cause dimension and layer mismatches in Q36. Add --strip-nextn (alias --strip-mtp) to qwen36-quantize: - Strips blk.40.* and .nextn_ tensors during parsing - Strips nextn_predict_layers KV metadata record - Safely patches GGUF block_count from 41 to 40 via strict key and type checking - Fully non-breaking: default behavior when flag is omitted remains 100% untouched - Documents the quantization workflow and runtime commands in gguf-tools/README.md
When using /read with an image in interactive mode, the user prompt was hardcoded to "". Allow specifying an optional user prompt after the file path, and support single or double quotes for paths containing whitespace: /read photo.jpg What is written on this sign? /read "path/with spaces/photo.png" Describe this image Also synchronize the help and usage text in q36_cli.c and q36_help.c.
- Document optional prompt and quoted path support in /read FILE [PROMPT] - Add reference to --strip-nextn for quantizing community fine-tunes and abliterated models in gguf-tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces two enhancements to improve compatibility with community fine-tunes and interactive vision usability without affecting default model paths:
gguf-tools/qwen36-quantize(--strip-nextn/--strip-mtp):Huihui-Qwen3.6-35B-A3B-abliterated, RavenX, etc.) often include a 41st speculative MTP layer (blk.40.*), settingblock_count = 41andnextn_predict_layers = 1.--strip-nextn:blk.40.*and.nextn_tensors during parsing.nextn_predict_layersKV metadata record.block_countfrom 41 to 40 via strict key and type checking (ends(key, ".block_count") && type == GGUF_TYPE_UINT32 && val == 41).q36_cli(/read FILE [PROMPT]):/read photo.jpg Describe what is in this image/read "path with spaces/image.png" What text is written here?q36_cli.candq36_help.c.Documentation:
README.mdandgguf-tools/README.mdwith instructions for quantizing and running community fine-tunes and abliterated models.Benchmarks & Verification
Tested on AMD BC-250 APU (RADV GFX1013 Vulkan, 15.35 GiB UMA).
1. Build Verification
All targets built cleanly with 0 errors and 0 warnings:
2. Correctness Regression Tests
3. Throughput Benchmark Comparison (
q36-bench)Smoke benchmark command:
Default Model (
q36moe.gguf- official Qwen3.6-35B-A3B IQ2_XXS/Q2_K):Abliterated Community Model (
Huihui-Qwen3.6-35B-A3B-Abliterated-Q36-IQ2XXS.ggufquantized with--strip-nextn):