[Bug]: llm-llamacpp never runs qvac-fabric --fit (sharded path skips it, single-file path aborts); >VRAM MoE model loads 78 GB onto a 24 GB GPU and runs at 2.5 tok/s

[yuranich]

🐛 Bug Description

@qvac/llm-llamacpp never applies qvac-fabric's --fit (automatic GPU/CPU placement) to a loaded model. For a MoE model larger than VRAM this makes qvac serve / loadModel() unusable (2.5 tok/s) although the same addon reaches 22 tok/s when the placement --fit would have chosen is passed by hand.

Two independent causes in the native addon:

1. Sharded GGUFs skip fit entirely. initFromConfig (packages/inference-addon-cpp/src/inference-addon-cpp/LlamacppUtils.hpp) sends multi-shard models through initFromShards() → llama_model_load_from_splits() and never calls common_fit_params; only the single-file, non-streaming path goes through common_init_from_params(), which is where fabric runs the fit.
2. Single-file fit aborts anyway. With the shards merged into one GGUF (llama-gguf-split --merge) and gpu_layers: -1 (fabric's "unset" sentinel, so the fit is allowed to touch n_gpu_layers), the load logs
   common_fit_params: failed to fit params to free device memory: did not provide buffer to set tensor_buft_overrides, abort.
   LoadFitNormalization.cpp re-implements common_params_parse but does not pad params.tensor_buft_overrides to llama_max_tensor_buft_overrides() entries the way fabric's common/arg.cpp (// pad tensor_buft_overrides for llama_params_fit) does, so the fit has nowhere to write its placement.

On top of that, @qvac/inference injects gpu_layers: 99 by default (LLM_CONFIG_DEFAULTS), which fabric treats as user-pinned, so even a working fit would abort with "n_gpu_layers already set by user".

🔄 Steps to Reproduce

Model: unsloth Qwen3.8-Flash-Next-UD-Q4_K_XL (103.7 GiB, 4 shards, 125B total / 6B active, 512 experts), local path, <base>.tensors.txt generated.

{ "serve": { "models": { "qwen38": { "src": ".../Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf", "type": "llm",
  "config": { "device": "gpu", "ctx_size": 16384, "verbosity": 3 } } } } }

qvac serve --openai, then one /v1/chat/completions request (44-token prompt, 256 max tokens).

✅ Expected Behavior

Same placement llama-cli/llama-server pick with --fit on this box: all 49 layers' attention/DeltaNet/dense/shared-expert weights plus the routed experts of layers 0–9 on the GPU (20.3 GB), routed experts of layers 10–48 in system RAM — ~22 tok/s on Vulkan, 26 tok/s on CUDA.

❌ Actual Behavior

Load log: offloaded 49/49 layers to GPU, Vulkan0 model buffer size = 78056.39 MiB on a 24 GB card; nvidia-smi shows 24.1/24.6 GB used and the NVIDIA Vulkan driver silently spills ~54 GB to host memory. Generation: 2.5 tok/s, prompt 2.6 tok/s.

Best the SDK config can currently do is plain layer offload (gpu_layers: 12, 18.5 GB VRAM, attention of the other 37 layers on CPU): 4.7 tok/s.

Comparison (same box, same model, ctx 16384, 44-token prompt → generation tok/s; 11K-token document → prompt tok/s):

Path	gen	prompt
qvac serve, defaults (gpu_layers 99)	2.5	3
qvac serve, gpu_layers: 12	4.7	7
@qvac/llm-llamacpp 0.49.2 direct from Bare, gpu_layers: 99 + override-tensor: "blk\.(1[0-9]|[2-4][0-9])\.ffn_(up|down|gate)_exps=CPU" + threads: 8 (= what fit picks)	22	88
qvac-fabric v10549 llama-server --fit-target 512 -t 8 (Vulkan)	20.7	104
llama.cpp b10927 llama-server --fit-target 512 -t 8 (CUDA)	23.9	174

💻 Platform / OS · ⚙️ Runtime · 🏷️ SDK Version

Ubuntu 26.04, Intel Core Ultra 7 265KF (8P+12E), 128 GB DDR5-4800, NVIDIA RTX 3090 24 GB (driver 595.91, Vulkan 1.4). Node 24.21 / Bare 1.32. @qvac/cli 0.13.0, @qvac/sdk + @qvac/inference 0.19.1, @qvac/llm-llamacpp 0.49.2 (fabric v10297.1.1 prebuild); same behaviour with @qvac/llm-llamacpp 0.52.0 (fabric v10549.0.0).

🩹 Workaround

Bypass the SDK: bare script against @qvac/llm-llamacpp with gpu_layers: 99, override-tensor and threads (any fabric flag passes through there). Not possible from qvac serve / loadModel() because the config schema rejects those keys (#4431).

📎 Likely component

@qvac/llm-llamacpp native addon (LoadFitNormalization.cpp padding, LlamacppUtils.hpp sharded path) and the gpu_layers: 99 default in @qvac/inference. Suggested fix: pad tensor_buft_overrides, call common_fit_params before llama_model_load_from_splits (with the first shard path), and stop injecting gpu_layers: 99 so fabric's -1 default lets the fit run.

[yuranich]

Cross-reference after going through the open fit work (#4415, #4174, #4178, packages/model-fit):

@qvac/model-fit already runs fabric's fitter and returns the full plan, including the per-tensor buftOverrides (the expert-FFN-to-CPU pattern) and nGpuLayers; its README is explicit that a SUCCESS "is conditional on applying the placement it returns". But @qvac/inference's advisory-fit.ts is advisory by design ("no verdict changes the load"), and the released SDK 0.19.1 does not ship @qvac/model-fit at all, so on the shipped path there is no fit of any kind and on main the plan is computed and then dropped while the addon loads with gpu_layers: 99.

So the shortest fix may not be inside the addon: take nGpuLayers + buftOverrides from the advisory plan and pass them to @qvac/llm-llamacpp as gpu_layers + override-tensor (both already accepted by the addon; override-tensor only needs to pass the SDK schema, #4431). The addon-side fixes above would still be needed for anyone using the addon without the SDK.