Skip to content

feat(gguf-tools, cli): add --strip-nextn for community fine-tunes and prompt support in /read - #6

Open
Trecto34 wants to merge 3 commits into
Ninnix:mainfrom
Trecto34:feature/strip-nextn-and-cli-read-prompt
Open

feat(gguf-tools, cli): add --strip-nextn for community fine-tunes and prompt support in /read#6
Trecto34 wants to merge 3 commits into
Ninnix:mainfrom
Trecto34:feature/strip-nextn-and-cli-read-prompt

Conversation

@Trecto34

Copy link
Copy Markdown

Summary

This PR introduces two enhancements to improve compatibility with community fine-tunes and interactive vision usability without affecting default model paths:

  1. gguf-tools/qwen36-quantize (--strip-nextn / --strip-mtp):

    • Community fine-tunes and abliterated models (e.g. Huihui-Qwen3.6-35B-A3B-abliterated, RavenX, etc.) often include a 41st speculative MTP layer (blk.40.*), setting block_count = 41 and nextn_predict_layers = 1.
    • Running these directly causes runtime layer and dimension mismatches in Q36.
    • Adding --strip-nextn:
      • Strips blk.40.* and .nextn_ tensors during parsing.
      • Strips nextn_predict_layers KV metadata record.
      • Safely patches GGUF block_count from 41 to 40 via strict key and type checking (ends(key, ".block_count") && type == GGUF_TYPE_UINT32 && val == 41).
      • Non-breaking: Default quantization path remains completely untouched when the flag is omitted.
  2. q36_cli (/read FILE [PROMPT]):

    • Allows passing an optional user prompt alongside image files in interactive CLI mode:
      • /read photo.jpg Describe what is in this image
      • /read "path with spaces/image.png" What text is written here?
    • Supports quoted paths (single/double quotes) containing spaces.
    • Synchronizes usage and help strings across q36_cli.c and q36_help.c.
  3. Documentation:

    • Updated README.md and gguf-tools/README.md with instructions for quantizing and running community fine-tunes and abliterated models.

Benchmarks & Verification

Tested on AMD BC-250 APU (RADV GFX1013 Vulkan, 15.35 GiB UMA).

1. Build Verification

All targets built cleanly with 0 errors and 0 warnings:

make clean
make vulkan-generic
make vulkan-bc250
make cpu
make -C gguf-tools

2. Correctness Regression Tests

./q36_test --server
# Output:
# server: OK
# q36 tests: ok

3. Throughput Benchmark Comparison (q36-bench)

Smoke benchmark command:

./q36-bench -m <model> --prompt-file tests/long_context_story_prompt.txt --ctx-start 32 --ctx-max 64 --ctx-alloc 128 --gen-tokens 8

Default Model (q36moe.gguf - official Qwen3.6-35B-A3B IQ2_XXS/Q2_K):

ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,kvcache_bytes
32,32,152.96,8,91.63,140
64,32,228.12,8,92.50,268

Abliterated Community Model (Huihui-Qwen3.6-35B-A3B-Abliterated-Q36-IQ2XXS.gguf quantized with --strip-nextn):

ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,kvcache_bytes
32,32,125.12,8,91.71,140
64,32,227.46,8,91.20,268
  • Generation/decode throughput is virtually identical (~91.6 vs ~91.7 t/s).
  • Prefill scaling matches the default model (~228 vs ~227 t/s at 64 tokens).

…ipping

Community fine-tunes and abliterated models (e.g. Huihui-Qwen3.6-35B-A3B-abliterated)
often package an extra 41st speculative MTP layer (blk.40.*), block_count = 41,
and nextn_predict_layers = 1. These cause dimension and layer mismatches in Q36.

Add --strip-nextn (alias --strip-mtp) to qwen36-quantize:
- Strips blk.40.* and .nextn_ tensors during parsing
- Strips nextn_predict_layers KV metadata record
- Safely patches GGUF block_count from 41 to 40 via strict key and type checking
- Fully non-breaking: default behavior when flag is omitted remains 100% untouched
- Documents the quantization workflow and runtime commands in gguf-tools/README.md
When using /read with an image in interactive mode, the user prompt was
hardcoded to "".

Allow specifying an optional user prompt after the file path, and support
single or double quotes for paths containing whitespace:
  /read photo.jpg What is written on this sign?
  /read "path/with spaces/photo.png" Describe this image

Also synchronize the help and usage text in q36_cli.c and q36_help.c.
- Document optional prompt and quoted path support in /read FILE [PROMPT]
- Add reference to --strip-nextn for quantizing community fine-tunes
  and abliterated models in gguf-tools
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant