Eval bug: Wrong number of tensors when using Qwen3.6 / Ternary-Bonsai

[david568303]

Name and Version

Im running the latest windows beellama, and if i use the official script file of the PrismMl-Eng llama.cpp to convert dspark to dflash which goes smoothly, and then i try to use dflash i get the following error, also, i can't replicate this issue with the official main repo of llama.cpp, i was only able to replicate it with beellama. C:\Users\David\Desktop\BeeLlama>.\beellama.cpp-v0.4.6\llama-server.exe -m .\models\Ternary-Bonsai-27B-heretic-ja-Q2_g64.gguf --spec-type draft-dflash --spec-draft-model .\models\Ternary-Bonsai-27B-dflash-bf16.gguf
0.00.102.003 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.102.247 W srv llama_server: -----------------
0.00.102.250 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.102.251 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.102.252 W srv llama_server: more info: ggml-org#25655
0.00.102.252 W srv llama_server: -----------------
0.00.158.757 I cmn common_specu: DFlash: omitted --spec-draft-n-max defaults to the drafter block depth (3); pass the flag to override
0.00.158.785 I srv load_model: loading model '.\models\Ternary-Bonsai-27B-heretic-ja-Q2_g64.gguf'
0.00.915.795 E llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 77, got 73
0.00.915.803 E llama_model_load_from_file_impl: failed to load model
0.00.915.832 W operator(): failed to measure the memory of the extra model, fitting without it: failed to load model
0.03.830.489 I cmn init: llama threadpool init, n_threads = 16
0.03.921.060 I common_speculative_init_result: loading draft model '.\models\Ternary-Bonsai-27B-dflash-bf16.gguf'
0.04.261.508 E llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 77, got 73
0.04.261.540 E llama_model_load_from_file_impl: failed to load model
0.04.261.541 E common_speculative_init_result: failed to load draft model, '.\models\Ternary-Bonsai-27B-dflash-bf16.gguf'
0.04.261.549 E srv load_model: failed to load draft model, '.\models\Ternary-Bonsai-27B-dflash-bf16.gguf'
0.04.261.564 I srv operator(): operator(): cleaning up before exit...
0.04.262.888 E srv llama_server: exiting due to model loading error

C:\Users\David\Desktop\BeeLlama>

Operating systems

Windows

GGML backends

CUDA

Hardware

msi Rtx 5090 32GB

Models

No response

Problem description & steps to reproduce

run llama-server with Ternary-Bonsai using a dspark file converted to dflash

First Bad Commit

No response

Relevant log output

C:\Users\David\Desktop\BeeLlama>.\beellama.cpp-v0.4.6\llama-server.exe -m .\models\Ternary-Bonsai-27B-heretic-ja-Q2_g64.gguf --spec-type draft-dflash --spec-draft-model .\models\Ternary-Bonsai-27B-dflash-bf16.gguf
0.00.102.003 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.102.247 W srv llama_server: -----------------
0.00.102.250 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.102.251 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.102.252 W srv llama_server: more info: ggml-org#25655
0.00.102.252 W srv llama_server: -----------------
0.00.158.757 I cmn common_specu: DFlash: omitted --spec-draft-n-max defaults to the drafter block depth (3); pass the flag to override
0.00.158.785 I srv load_model: loading model '.\models\Ternary-Bonsai-27B-heretic-ja-Q2_g64.gguf'
0.00.915.795 E llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 77, got 73
0.00.915.803 E llama_model_load_from_file_impl: failed to load model
0.00.915.832 W operator(): failed to measure the memory of the extra model, fitting without it: failed to load model
0.03.830.489 I cmn init: llama threadpool init, n_threads = 16
0.03.921.060 I common_speculative_init_result: loading draft model '.\models\Ternary-Bonsai-27B-dflash-bf16.gguf'
0.04.261.508 E llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 77, got 73
0.04.261.540 E llama_model_load_from_file_impl: failed to load model
0.04.261.541 E common_speculative_init_result: failed to load draft model, '.\models\Ternary-Bonsai-27B-dflash-bf16.gguf'
0.04.261.549 E srv load_model: failed to load draft model, '.\models\Ternary-Bonsai-27B-dflash-bf16.gguf'
0.04.261.564 I srv operator(): operator(): cleaning up before exit...
0.04.262.888 E srv llama_server: exiting due to model loading error

C:\Users\David\Desktop\BeeLlama>

But, now with official default llama.cpp

C:\Users\David\Desktop\BeeLlama>run-light.bat

C:\Users\David\Desktop\BeeLlama>.\llama.cpp\llama-server.exe -m .\models\Ternary-Bonsai-27B-heretic-ja-Q2_g64.gguf -c 700000 -np 1 --cache-type-k q4_0 --cache-type-v q4_0 -fa on --rope-scaling yarn --rope-scale 2.6705 --yarn-orig-ctx 262144 --override-kv qwen35.context_length=int:700000 --spec-type draft-dflash --spec-draft-model .\models\Qwen3.6-27B-DFlash-Q8_0.gguf --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 --load-mode mmap --reasoning-preserve --host 127.0.0.1 -fit off --flash-attn on --port 8070
0.00.087.827 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.088.068 W srv llama_server: -----------------
0.00.088.071 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.088.072 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.088.072 W srv llama_server: more info: ggml-org#25655
0.00.088.072 W srv llama_server: -----------------
0.00.104.885 I srv load_model: loading model '.\models\Ternary-Bonsai-27B-heretic-ja-Q2_g64.gguf'
0.02.925.830 W llama_context: n_ctx_seq (700160) > n_ctx_train (700000) -- possible training context overflow
0.03.131.265 I cmn init: llama threadpool init, n_threads = 16
0.03.232.936 I common_speculative_init_result: loading draft model '.\models\Qwen3.6-27B-DFlash-Q8_0.gguf'
0.04.168.656 W llama_context: n_ctx_seq (700160) > n_ctx_train (262144) -- possible training context overflow
0.04.235.803 W srv load_model: the slot context (700160) exceeds the training context of the model (700055) - capping
0.04.296.717 I srv load_model: initializing, n_slots = 1, n_ctx_slot = 700055, kv_unified = 'false'
0.04.296.749 I common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
0.04.296.752 I common_speculative_impl_draft_dflash: - n_max=3, n_min=0, p_min=0.00
0.04.296.753 I common_speculative_impl_draft_dflash: - block_size=16, mask_token_id=248070, n_extract=5, sample_from_anchor=true
0.04.447.891 I srv llama_server: model loaded
0.04.447.900 I srv llama_server: listening on http://127.0.0.1:8070
0.05.877.618 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
0.05.878.471 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
0.09.183.379 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 9269, progress = 0.95, t = 3.25 s / 2848.02 tokens per second
0.09.389.428 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 9774, progress = 1.00, t = 3.45 s / 2836.05 tokens per second
0.09.449.894 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 9781, progress = 1.00, t = 3.56 s / 2749.86 tokens per second
0.09.885.302 I slot print_timing: id 0 | task 0 | prompt eval time = 3635.34 ms / 9785 tokens ( 0.37 ms per token, 2691.63 tokens per second)
0.09.885.308 I slot print_timing: id 0 | task 0 | eval time = 370.73 ms / 41 tokens ( 9.27 ms per token, 107.90 tokens per second)
0.09.885.309 I slot print_timing: id 0 | task 0 | total time = 4006.06 ms / 9826 tokens
0.09.885.310 I slot print_timing: id 0 | task 0 | graphs reused = 17
0.09.885.314 I slot print_timing: id 0 | task 0 | draft acceptance = 0.45098 ( 23 accepted / 51 generated), mean len = 2.35
0.09.885.882 I slot release: id 0 | task 0 | stop processing: n_tokens = 9825, truncated = 0
0.09.921.633 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 9799927
0.10.038.169 I slot launch_slot_: id 0 | task 26 | processing task, is_child = 0
0.13.205.213 I slot print_timing: id 0 | task 26 | n_gen = 432, tg = 142.97 t/s, tg_3s = 143.27 t/s
0.16.218.488 I slot print_timing: id 0 | task 26 | n_gen = 837, tg = 138.69 t/s, tg_3s = 134.41 t/s
0.16.469.451 I slot print_timing: id 0 | task 26 | prompt eval time = 151.82 ms / 252 tokens ( 0.60 ms per token, 1659.81 tokens per second)
0.16.469.461 I slot print_timing: id 0 | task 26 | eval time = 6278.88 ms / 893 tokens ( 7.04 ms per token, 142.06 tokens per second)
0.16.469.462 I slot print_timing: id 0 | task 26 | total time = 6430.70 ms / 1145 tokens
0.16.469.462 I slot print_timing: id 0 | task 26 | graphs reused = 405
0.16.469.469 I slot print_timing: id 0 | task 26 | draft acceptance = 0.42324 ( 499 accepted / 1179 generated), mean len = 2.27
0.16.469.525 I slot release: id 0 | task 26 | stop processing: n_tokens = 1144, truncated = 0
0.22.986.728 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 16383563
0.23.075.833 I slot launch_slot_: id 0 | task 422 | processing task, is_child = 0
0.25.877.823 I slot print_timing: id 0 | task 422 | prompt eval time = 205.21 ms / 34 tokens ( 6.04 ms per token, 165.69 tokens per second)
0.25.877.830 I slot print_timing: id 0 | task 422 | eval time = 2596.20 ms / 264 tokens ( 9.87 ms per token, 101.30 tokens per second)
0.25.877.831 I slot print_timing: id 0 | task 422 | total time = 2801.40 ms / 298 tokens
0.25.877.832 I slot print_timing: id 0 | task 422 | graphs reused = 540
0.25.877.836 I slot print_timing: id 0 | task 422 | draft acceptance = 0.30657 ( 126 accepted / 411 generated), mean len = 1.92
0.25.878.382 I slot release: id 0 | task 422 | stop processing: n_tokens = 10078, truncated = 0
0.25.914.093 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 25792420
0.26.067.811 I slot launch_slot_: id 0 | task 563 | processing task, is_child = 0
0.29.266.919 I slot print_timing: id 0 | task 563 | n_gen = 456, tg = 151.32 t/s, tg_3s = 151.63 t/s
0.32.279.796 I slot print_timing: id 0 | task 563 | n_gen = 827, tg = 137.22 t/s, tg_3s = 123.14 t/s
0.35.291.060 I slot print_timing: id 0 | task 563 | n_gen = 1260, tg = 139.41 t/s, tg_3s = 143.79 t/s
0.38.297.765 I slot print_timing: id 0 | task 563 | n_gen = 1728, tg = 143.47 t/s, tg_3s = 155.65 t/s
0.39.667.276 I slot print_timing: id 0 | task 563 | prompt eval time = 191.75 ms / 451 tokens ( 0.43 ms per token, 2351.96 tokens per second)
0.39.667.283 I slot print_timing: id 0 | task 563 | eval time = 13407.14 ms / 1964 tokens ( 6.83 ms per token, 146.41 tokens per second)
0.39.667.284 I slot print_timing: id 0 | task 563 | total time = 13598.89 ms / 2415 tokens
0.39.667.285 I slot print_timing: id 0 | task 563 | graphs reused = 1363
0.39.667.290 I slot print_timing: id 0 | task 563 | draft acceptance = 0.45312 ( 1131 accepted / 2496 generated), mean len = 2.36
0.39.667.368 I slot release: id 0 | task 563 | stop processing: n_tokens = 2414, truncated = 0
0.39.667.654 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 39581406
0.39.762.022 I slot launch_slot_: id 0 | task 969 | processing task, is_child = 0
0.43.024.928 I slot print_timing: id 0 | task 969 | n_gen = 283, tg = 93.65 t/s, tg_3s = 93.96 t/s
0.43.589.989 I slot print_timing: id 0 | task 969 | prompt eval time = 251.08 ms / 207 tokens ( 1.21 ms per token, 824.44 tokens per second)
0.43.589.995 I slot print_timing: id 0 | task 969 | eval time = 3576.29 ms / 350 tokens ( 10.25 ms per token, 97.59 tokens per second)
0.43.589.997 I slot print_timing: id 0 | task 969 | total time = 3827.37 ms / 557 tokens
0.43.589.998 I slot print_timing: id 0 | task 969 | graphs reused = 1550
0.43.590.003 I slot print_timing: id 0 | task 969 | draft acceptance = 0.28219 ( 160 accepted / 567 generated), mean len = 1.85
0.43.590.539 I slot release: id 0 | task 969 | stop processing: n_tokens = 10367, truncated = 0
0.45.996.153 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.731 (> 0.100 thold), f_keep = 0.966
0.45.997.231 I slot launch_slot_: id 0 | task 1591 | processing task, is_child = 0
0.47.023.783 W srv stop: cancel task, id_task = 1591
0.47.031.755 I slot release: id 0 | task 1591 | stop processing: n_tokens = 13188, truncated = 0

[Anbeeld]

This is not a BeeLlama bug, and not BeeLlama-specific either: the GGUF your conversion produced uses a tensor set that only PrismML's fork implements, and stock llama.cpp rejects the same file with the same message.

Your drafter belongs to PrismML's GIDD log-SNR variant. Their gguf_dspark_to_dflash.py renames dspark.log_snr_fc1/fc2.* to log_snr_fc1/fc2.* and carries over dspark.dspark.log_snr_conditioning. Those four tensors (log_snr_fc1.weight, log_snr_fc1.bias, log_snr_fc2.weight, log_snr_fc2.bias) and the dflash.log_snr_conditioning key exist only in PrismML's fork, which re-applies the log-SNR delta on top of upstream dflash (PrismML-Eng#114).

The numbers in your log are exactly that gap. The published source drafter (prism-ml/Ternary-Bonsai-27B-gguf, Ternary-Bonsai-27B-dspark-bf16.gguf) has 79 tensors, and your 77 implies the conversion dropped token_embd.weight and output.weight with --drop-shared-tensors. The dflash arch claims 66 block tensors (6 layers x 11) plus fc.weight, enc.output_norm.weight, output_norm.weight, markov_w1.weight, markov_w2.weight, conf_proj.weight and conf_proj.bias, which is 73. The four leftovers are the log-SNR ones, and llama_model_loader::done_getting_tensors() refuses any file with unclaimed tensors.

The mainline run did not hit this because it used Qwen3.6-27B-DFlash-Q8_0.gguf, a stock DFlash drafter with no Markov head, no confidence head and no log-SNR tensors. The tensor-creation code in src/models/dflash.cpp matches upstream master here, and the only local change in llama-model-loader.cpp is dropping the removed Bee TQ3/TQ4 ftype ids, so upstream rejects this file too.

I reproduced it on the current tree: I converted the published Ternary-Bonsai-27B-dspark-bf16.gguf with PrismML's script (keeping the real header and tensor table, skipping the roughly 7 GB payload copy) and got

E llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 77, got 73

After removing those four tensors and the dflash.log_snr_conditioning key, the count check passes and tensor creation completes. The load then stops on dflash requires ctx_other to be set, which is simply what happens when a draft model is loaded as a standalone target.

So the real fix is upstream support for the log-SNR tensors (log_snr_fc1/fc2 weights and biases, plus dflash.log_snr_conditioning, min_log_snr and max_log_snr). BeeLlama does not add fork-private DFlash tensors, so there is nothing to patch here.

As a workaround, strip those four tensors and the flag from the converted file. It loads and drafts, but without conditioning, and PrismML reports that costs real acceptance rate: 82.6% with conditioning versus 51.7% stripped. Also pass --spec-type draft-dspark (or omit --spec-type and let it auto-detect from markov_w1.weight), and use -fit off if memory fitting still tries to measure the draft.

One thing worth confirming: did you run the converter with --drop-shared-tensors? That is what the 77 in your log implies, but keeping the embeddings and head would give 79 and 75 instead.

[david568303]

As i specifyed, the output dflash file conversed using the script from PrismML, works on ggml-org/llama.cpp, soo i think it somehow is or could be a BeeLlama bug, from the 2 outputs i pasted on my other message, the last one is from ggml-org/llama.cpp, not PrismML fork.

Answering to your question, i did try using --drop-shared-tensors and not using it, and i had the same results.

Also, thanks for taking a deep look into the issue, i saw you were able to replicate it.