Conversation
…rg#28127) * model: add Tencent Hy 4 (hy_v4) preview architecture support Adds support for the Tencent Hy 4 model (Hugging Face architecture HYV4ForCausalLM, GGUF arch hy_v4): Add HF -> GGUF conversion script (conversion/hy_v4.py) and wire it into the conversion registry Register hy_v4 GGUF constants, arch enum, and writer support Implement the hy-v4 model graph, hparams, vocab and context changes Register the new arch in llama-arch and models registry Extend arch tests to cover hy_v4 Assisted by Claude Opus 5 * Update convert_hf_to_gguf_update.py Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> * Update conversion/base.py Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> * convert : move hy_v4 entry to the same place as in convert_hf_to_gguf_update.py * model : apply changes related to n_ff_exp becoming per-layer in Hy4-preview * n_layer_all --------- Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* move more jobs to ccache-buckets * add venv deps * also jq
* opencl: add Adreno xmem SDPA path Assisted-by: Codex * Removed the Adreno-specific queue profiling override * Clean up formatting * 修复数值误差优化gqa/mask attn Assisted-by: Codex * add env GGML_OPENCL_XMEM_SDPA Assisted-by: OpenAI Codex --------- Co-authored-by: happyyzy <happyyzy@users.noreply.github.com>
…g#27633) * opencl: add extended elementwise unary ops (sgn, step, elu, hardswish, hardsigmoid, floor, ceil, round, trunc) Adds nine GGML_UNARY_OP_* elementwise ops that were falling back to CPU on the OpenCL backend, following the same variant shape as the existing ABS op: f32, f32_4 (vec4), f16, f16_4 (vec4), and stride-addressed f32_nc / f16_nc for non-contiguous inputs. New kernels/unary_ext.cl (macro-generated), a shared ggml_cl_unary_ext dispatch helper mirroring ggml_cl_abs, the supports_op cases, and the compute-forward cases. Values are computed in float (the f16 variants read/write half and convert), so the conditional ops (step, elu) match the CPU reference; the vec4 forms use select() for the branch. Validated with test-backend-ops on Adreno 840 and 850 (E17): all nine ops pass every case including the vec4 and non-contiguous variants (8/8 or 14/14). * opencl: dispatch a contiguous f32 copy over the whole device kernel_cpy_f32_f32 maps one workgroup to each (i01,i02,i03) row and strides the row across that workgroup's lanes, and the host launches ne01*MIN(64,ne00) work items. A tensor with few long rows therefore runs on a single workgroup. The mamba2 and gated-delta-net recurrent state cache is one row of 524288 floats, copied once per layer per graph, and lands on 64 work items. When both sides are contiguous the copy is a linear move, so dispatch it over the whole device: one work item per float4. Gated on ggml_is_contiguous for both tensors and equal element counts, so copies already spread over many rows keep the existing path. The kernel is created optionally, so a driver that rejects it falls back rather than aborting. vload4/vstore4 rather than a float4 cast: they require only the scalar type's alignment, and these buffers carry an arbitrary 4-byte view offset. CPY, DUP and CONT are 217/217 on Adreno 840 and 740 with the path enabled and disabled. GGML_OPENCL_CPY_FLAT=0 forces the old kernel. * opencl: support all easy-copy types in CONCAT CONCAT was F32-only. Extend it to every "easy-copy" type -- any non-quantized type with a block size of 1 and an element size of 1, 2, 4 or 8 bytes, i.e. f16/bf16/i8/i16/i32/i64 as well as f32. The kernels are keyed by element SIZE rather than by type, which is what CUDA already does for the same op: one kernel per byte width (b1/b2/b4/b8) plus the packed b4 fast path, instead of one per ggml type. supports_op gates on the same property, so a new type of a supported width is picked up with no further work. Validated with test-backend-ops on Adreno 840 / A8X and X2-90 / X2E.
…25789) (ggml-org#28089) * ui : fix MCP image attachments not displayed in tool block (ggml-org#25789) Fixes regression from ggml-org#25450 where ChatMessageAgenticContent passed message.extra instead of section.toolResultExtras to tool blocks, leaving tool images invisible. Also fixes TOOL_RESULT_JSON_OPEN_REGEX which misclassified "[Attachment saved: ...]" as JSON. Fixes ggml-org#25789 Assisted-by: Muse Spark * Addressed PR comments: 1.- Removed ·?? mesage?extra· as it has no case left to cover 2.- Added ·[\· to cover the case of ·[[1, 2], [3, 4]]· case suggested in the PR comment 3.- Added unit test for covering up this regex case * ui : fix MCP image attachments not displayed in tool block (ggml-org#25789) - Addressed lint error on regex (redundant \)
* addition of m3 in fa_vec_tuned_table * adding q4_0,q4_1,q5_0,q5_1 in ggml-metal-tuning * Fix formatting in ggml-metal-tuning.cpp
…-org#27631) define two new environment variables to better understand how much memory is being allocated, and when. This has been invaluable in inproving the --fit algorithm, and is likely to be useful when debugging other memory-related issues. `-lv 4` will be required to enable the following: GGML_SYCL_MEMTRACE=1 will show per-site memory usage, updated whenever it increases by more than 64MiB. GGML_SYCL_MEMTRACE=2 will show every allocation and deallocation. To change the default 64MiB threshold for reporting memory usage increases, use GGML_SYCL_MEMTRACE_STEP. A sample log line: [SYCL-MEMTRACE] device memory query (dev): total 59493 MiB, free 4494, in use 54998; allocated 0 (buffers 0 + scratch 0), peak 0 MiB
…T support (ggml-org#28016) (ggml-org#28254) * Reapply "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (ggml-org#28184) This reverts commit c845263. * tests : fix unused variable M in test-backend-ops * tests: fix trailing space error and isolate kronecker tests for sycl backend only
Remove the build-time C++ helper and external gzip dependency, simplifying cross-compilation. Keep the generated C++ in templates for readability and preserve fully embedded UI assets. Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* common: add --log-jsonl * rename unknown to none
* ui : update active conversation fields in place updateCurrentNode, applyConversationUpdate, updateConversationTimestamp and the pin toggle replaced the whole activeConversation object, so its identity changed on every send, tool result and rename. ChatMessages tracks that identity to refresh sibling info, so each replacement triggered a full refetch of every message in the conversation. Write the changed fields instead, mirroring updateMessageAtIndex. Assisted-by: pi:zai-org/GLM-5.3 * ui : reuse the conversation load read for sibling info Opening a conversation read every message from the database twice: once in loadConversation for the active path, once in ChatMessages for the sibling map. Hand the freshly read array over once so the chat screen builds sibling info from it, and set the conversation and its messages in one sync block so effects never see the new conversation paired with the previous one's messages. Assisted-by: pi:zai-org/GLM-5.3 * ui : memoize leaf walks in sibling map build buildSiblingInfoMap resolves each sibling's leaf by walking the last-child chain, once per sibling per message, so the walk repeats along the same chains for every message in the conversation ( O(messages^2) on long chats ). Memoize leaf resolution per build with path compression so each edge is walked once. Assisted-by: pi:zai-org/GLM-5.3 * ui : skip sibling refetch for in-place message edits refreshAllMessages refetches every message of the conversation just to rebuild sibling info, but preserve-responses and non-branching assistant edits never create branches, so the sibling map stays valid. Refresh only after actions that branch (editWithBranching kept) or delete. Assisted-by: pi:zai-org/GLM-5.3 * ui : drop unused currentResponse reactive writes Nothing reads chatStore.currentResponse, but setChatStreaming reassigned it on every streamed chunk, so each token paid a reactive write and string assignment for nothing. Remove the field and the clearUIState wrapper that only reset it. Assisted-by: pi:zai-org/GLM-5.3 * ui : reuse completed agentic turn sections during streaming deriveAgenticSections runs in a $derived invalidated per streamed chunk, but re-derived every turn of the session each time, so per-chunk cost grew with session length. Cache completed turns keyed by their assistant message plus reference checks on every field that feeds derivation; only the streaming turn recomputes. Cache hits return the same section objects, so tool block props stay stable and skip their per-chunk re-derive. Assisted-by: pi:zai-org/GLM-5.3 * ui : share markdown block infrastructure Every markdown block duplicated shared work: a full copy of the hljs theme CSS per instance, and the remark/rehype plugin chain rebuilt on every processMarkdown call ( once per block at mount, again per coalesced chunk while streaming ). Use the single theme style element already maintained by SyntaxHighlightedCode, and build pipelines once - shared process-wide for attachment-less blocks, cached by attachments identity otherwise. Assisted-by: pi:zai-org/GLM-5.3 * ui : measure assistant layout only for the last message Every assistant message ran getComputedStyle, getBoundingClientRect and a ResizeObserver over the previous user bubble at mount, even off-screen ones, forcing a layout pass per message while a long conversation renders. The measured vars only feed the :last-child min-height rule, so gate the effect on isLastAssistantMessage; one measurement and one observer remain, and the effect re-runs when the last message changes. Assisted-by: pi:zai-org/GLM-5.3 * ui : trim whole-blob scans in tool block headers Tool block headers parsed their entire blobs at mount, even collapsed, and most tool results and args are large plain text or embedded file content: skip JSON.parse unless the blob starts with a JSON container, prefilter search-result extraction with a Title:/URL: substring check, and match the end-anchored exit-code marker against only the tail of exec outputs. Assisted-by: pi:zai-org/GLM-5.3 * ui : parse write_file and edit_file titles without the content blob Both block headers parsed the full args JSON at mount, even collapsed, and write_file and edit_file args embed the whole file content or edit strings, so every block paid a full-blob JSON parse just to read the path. Split the meta into a title tier that extracts the path with a targeted key match (full parse only as fallback) and a body tier that keeps the full parse; Svelte deriveds are lazy, and the body snippet renders only while the block is expanded, so collapsed blocks no longer parse args. Assisted-by: pi:zai-org/GLM-5.3 * ui : mount chat messages lazily near the viewport Every message row mounted its full component tree on load, so the cycle collector, GC and layout invalidation kept walking every live object and DOM node even for rows the user never scrolls to - which dominated the profile of long conversations. Wrap each row in a placeholder with an IntersectionObserver ( two viewport heights of runway ) that swaps in the real ChatMessage when the row approaches the viewport; the row shell keeps the content-visibility sizing, and rows stay mounted once realized. Rows targeted by the pending-edit flow mount eagerly. Assisted-by: pi:zai-org/GLM-5.3 * ui : smooth the chat navigation animations Slide the centered new-chat form to the bottom edge with a transform instead of a bottom offset - layout-property transitions need the main thread every frame and stutter while a long conversation loads, while transform transitions run on the compositor. Fade the message list in with a CSS animation keyed to the conversation id, disabled under prefers-reduced-motion. Assisted-by: pi:zai-org/GLM-5.3 * ui : follow the svelte runes guidance in chat message code Two effects detected changes with manual previous-value refs and reset flags. The permission request carries object identity, so its dismissal is now a derived comparing the dismissed request; the continue request is a bare boolean, so its dismissal only shrinks to a reset while no request is pending. Also drop a dead if (browser) guard in the markdown theme loader - effects never run on the server. Assisted-by: pi:zai-org/GLM-5.3 * test : pin the chat perf invariants in the unit suite Cover the fixes whose silent regression would be stale or wrong UI rather than a crash: the turn-section cache must reuse unchanged turns yet recompute on every field it compares; the sibling map must resolve the same leaves after the leaf-walk memoization; the active conversation must keep its identity through field updates; and the blob gates ( exec tail window, plain-text result gate, search prefilter ) must keep accepting what they gate. Only the risky invariants are pinned - no coverage for coverage's sake. Assisted-by: pi:zai-org/GLM-5.3 * refactor : address review remarks Name the tool-arg string-field pattern, move the file tools' path field aliases and the JSON container gates into lib/constants, and export the write_file / edit_file meta types from $lib/types instead of the parser modules. Assisted-by: pi:zai-org/GLM-5.3
* Add Spark3 Model * rename spark3 -> spark2_5 Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> Co-authored-by: dongjiang <dongjiang2010@gmail.com>
* models: use flash-linear-attention's l2norm for gated delta net q/k
The GDN q/k normalization is defined by flash-linear-attention as
l2norm(x) = x * rsqrt(sum(x*x) + eps)
with eps inside the root. Every GDN call site in the tree uses ggml_l2_norm
instead, which is x / max(sqrt(sum(x*x)), eps), i.e.
torch.nn.functional.normalize - its CUDA kernel cites that page.
The clamp never engages at these magnitudes, so in practice llama.cpp
normalizes with no epsilon at all where the reference has one inside the
root.
transformers made the same substitution when it first added Qwen3-Next and
corrected it three days later in huggingface/transformers#40842, 'Fix the
misalignment between the l2norm in GDN of Qwen3-Next and the implementation
in the FLA library'. vLLM and SGLang vendor FLA rather than reimplementing
it, so neither ever had the clamp.
eps keeps coming from the checkpoint, exactly as every call site already
passed it. The references hardcode 1e-6 for this norm; that is a separate
question and the two agree on every GDN checkpoint in the wild.
ggml_l2_norm itself is correct and unchanged, as is rwkv7-base, its original
caller, which passes normalize's own default eps of 1e-12.
No new ggml op: rms_norm already carries eps inside the root, so
rms_norm(x, eps/n) * (1/sqrt(n)) is exactly x * rsqrt(sum(x*x) + eps).
* Update src/models/models.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ggml-org#27765) * vulkan: add TQ1_0 support (mm, mat-vec, dequant, get_rows) * vulkan: pack TQ1_0 powers of 3 into a 32-bit constant Replaces the constant array with a packed 32-bit value (7 bits per entry, max 81 < 128) extracted with shift/mask, as suggested in review — avoids a constant array that may not be kept in registers. test-backend-ops on gfx1151: tq1_0 MUL_MAT 11/11, MUL_MAT_ID 6/6, GET_ROWS 4/4, unchanged. * vulkan: address review - shared TQ1_0 decode helpers, fix standalone dequant shader Review feedback from jeffbolznv, all points: - Move the packed-pow3 decode into shared helpers in types.glsl (tq1_0_byte_of / tq1_0_digit_of / tq1_0_trit) and use them from dequant_funcs.glsl, mul_mm_funcs.glsl, dequant_funcs_cm2.glsl and dequant_tq1_0.comp instead of repeating the logic. The cm2 path also drops its constant array for the packed-constant extraction. - Translate all remaining comments to English. - dequant_tq1_0.comp: use dequant_head.glsl. The shader previously declared its own single-field push constant while the pipeline is created with the 5-field layout, so p.ne read the wrong field - confirmed broken, as suspected in review. - Fix wg_denoms for the standalone dequant pipeline: one invocation decodes 4 elements with local_size 256, so a workgroup covers 256*4 elements, not 256*16. With the old value the dispatcher launched a quarter of the required workgroups. Verified by temporarily forcing the dequant + f16 matmul path for TQ1_0 (hack not committed): test-backend-ops MUL_MAT passes through the rewritten standalone shader, and the standard MUL_MAT / MUL_MAT_ID / GET_ROWS tq1_0 cases still pass on Vulkan (AMD gfx1151). * vulkan: address review — English comments, shared tq1_0_trit, trim TQ1_0 test cases - mul_mat_vec_tq1_0.comp: drop leftover non-English comment and the local POW3_PACKED constant; all decode sites now call tq1_0_trit() from types.glsl - types.glsl / dequant_funcs_cm2.glsl: ASCII-only, drop stale reviewer note - test-backend-ops: remove the oversized MUL_MAT_ID case (432 MiB A tensor, ~172 GFLOP reference); move the two remaining ones next to the other backend-specific mul_mat_id one-offs and document why they are needed * metal: decline TQ1_0 for GET_ROWS and mat-mul in supports_op The new TQ1_0 cases in test-backend-ops exposed that the Metal backend claimed support for GET_ROWS/MUL_MAT/MUL_MAT_ID with TQ1_0 sources while having no such kernels (ggml_metal_library_compile_pipeline aborted on the missing kernel_get_rows_tq1_0). Decline the type so the ops fall back to the CPU, matching the existing NVFP4 handling on the same lines. Assisted-by: Claude Fable 5 * vulkan: trim the TQ1_0 comments Addresses @0cc4m's review: keep only what the code does not already say. Removed the block-format recaps (the layout is right there in the struct) and the step-by-step decode walkthrough. Kept the two facts a reader cannot infer: the 8-bit truncation is part of the format, not an optimisation, and the powers of 3 are packed into one uint so they do not end up in a constant array that may miss the registers. No functional change. * vulkan: address review — trim comments, fold Metal check, drop unused _v Per @0cc4m's review: - dequant_funcs.glsl, dequant_funcs_cm2.glsl: drop the "see types.glsl" pointers — they apply to every quant and say nothing specific. - dequant_tq1_0.comp: drop the wg_denoms note. It is a precondition, not information. - mul_mm_funcs.glsl: same pointer removed. - types.glsl: the comment on tq1_0_trit is down to the one fact the code cannot show — the 8-bit truncation is part of the format, matching the C reference, not an optimisation. - dequant_funcs_cm2.glsl: removed dequantFuncTQ1_0_v and its define. You were right that it is optional: it wrapped four scalar decodes and vectorised nothing, and mul_mm_cm2.comp already guards the path with `#if defined(dequantFuncA_v)` (DATA_A_F32 omits it the same way). - ggml-metal-device.m: folded TQ1_0 into the existing NVFP4 check instead of a separate block, and dropped both comments. - test-backend-ops.cpp: the two mul_mat_id cases stay — they cover the block-stride loop and the per-expert base offset that k == 256 alone never reaches — but the comment is now one line instead of five. Kept: the one-line labels on the three block regions in mul_mat_vec_tq1_0.comp and on tq1_0_byte_of(). Those state the 5-trits-per-byte packing, which the loop bounds do not show. Happy to remove them too if you prefer. Re-verified on AMD gfx1151 (Vulkan), test-backend-ops, 2/2 backends passed: MUL_MAT 9 TQ1_0 cases, MUL_MAT_ID 5, GET_ROWS 4 — all OK, no failures. The coopmat2 path is unchanged apart from the removed _v define.
This commit contains a suggestion for handling container images which are currently not semver tagged, they only have build numbers in there tags. The proposed solution here is to first add a check to make sure that there are container images built for the build number of the release and if not fail the build. The container images are build nightly but they can be triggered manually as well. If the the container images check passes then the make-release workflow will re-tag the images with the semver.
Support RMS_NORM + MUL + ADD (+ MUL) and RMS_NORM + VIEW + SET_ROWS. Extend ROPE + VIEW + SET_ROWS to support IMROPE. Worth around 4% in gemma4 on my system.
* ggml-cuda: fix divergent barrier in f16 flash attention * ggml-cuda: avoid duplicate metadata pointer setup
…gml-org#28208) Problem - Loader prefers `<arch>.attention.recurrent_layers`, falls back to `full_attention_interval` if missing - Converter only ever writes the interval. gguf-py has no constant/writer for the array - Interval can only describe evenly spaced full-attention layers. Any non-uniform `layer_types` gets reconstructed wrong - No error, no warning. Model loads, runs, wrong layers get wrong ops. Full-attn layers marked recurrent lose their KV cache - Every published Qwen3.5 checkpoint is uniform so nobody's hit it yet Repro 12 layers, periods 4/3/5: layer: 0 1 2 3 4 5 6 7 8 9 10 11 actual: L L L F L L F L L L L F loader: L L L F L L L F L L L F ^ ^ Layer 6 is full attn, loaded as recurrent. Layer 7 the reverse. 52-layer non-uniform stack: 15/52 mis-typed. Fix - `constants.py`: add `Keys.Attention.RECURRENT_LAYERS` (name already registered in llama-arch.cpp) - `gguf_writer.py`: add `add_recurrent_layers()`, same shape as `add_rope_pattern()` - `conversion/qwen.py`: emit array from `layer_types` in `Qwen3NextModel.set_gguf_parameters` (covers 3-Next, 3.5, 3.5-MoE) Notes - Array is padded with `false` for MTP blocks. `get_key_or_arr` checks length against `n_layer_all`, which includes MTP. Matches the fallback's `i < n_layer()` guard - Interval is still written. Old builds only understand the interval - `layer_types` length != `num_hidden_layers` now raises in converter instead of producing a GGUF that fails at load Tested - End-to-end on a 62-layer non-uniform Qwen3.8-27B (2 linear layers removed). Loader reads the array, 62 blocks, 0 mismatches. Without fix: interval fallback, mis-typed - MTP padding NOT tested on a real MTP model. Reasoned from qwen35.cpp + get_key_or_arr. Would appreciate a check Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…gml-org#28538) This commit adds the printing of the ggml version and commit to the test-cmake example. The motivation is just to be able to quickly verify that the correct version of ggml is being used. Example output: ```console test-cmake] llama.cpp version: 0.4.0-dev, build: 10837 (5202104) [test-cmake] ggml version: 0.23.0, commit: 5202104 [test-cmake] Initializing backend... ... ```
* vulkan: fall back to CPU for GET_ROWS with misaligned offsets
The Vulkan GET_ROWS shader asserts when a tensor's backing-buffer offset
plus view_offs is misaligned w.r.t. minStorageBufferOffsetAlignment
(see init_pushconst_tensor_offsets). Previously this caused a hard crash
on models using ggml_view + ggml_get_rows (e.g. Qwen3-TTS, Qwen3-VL).
Return false from supports_op() in the misaligned case so the scheduler
falls back to CPU, matching the existing pattern for PAD_REFLECT_1D and
other unsupported op/shape combinations.
Repro: llama-tts -m Qwen3-TTS-*.gguf -mm mmproj-*.gguf -ngl 99
Crash: GGML_ASSERT(dst->op != GGML_OP_GET_ROWS || (a_offset == 0 && ...)) failed
* vulkan: trim comment for GET_ROWS misalign fallback
* vulkan: fix file corruption in gated_linear_attn struct
* vulkan: properly handle misaligned offsets in GET_ROWS quantized path
- get_rows_quant.comp was missing get_aoffset()/get_boffset()/get_doffset()
calls that are already present in get_rows.comp, causing GGML_ASSERT crashes
when GET_ROWS operates on views with non-zero view_offs, as produced by
KV cache slices in Qwen3-TTS and Qwen3-VL.
- Remove the defensive misalignment GGML_ASSERT in init_pushconst_tensor_offsets
for the binary push-constants specialization, since both get_rows.comp and
get_rows_quant.comp now correctly apply per-tensor base offsets.
- Remove the workaround CPU fallback in supports_op() for GET_ROWS, since the
Vulkan backend now handles misaligned offsets natively (no more bailout).
- Add backend test coverage with view_src0=true (ggml_view_4d into a padded
tensor) for F32, F16, Q4_0, Q4_K, Q8_0, and I32 types, exercising both the
non-quantized (get_rows.comp) and quantized (get_rows_quant.comp) paths
with non-zero view_offs that reproduce the original Qwen3-TTS crash.
* tests: trim redundant comments in test_get_rows vs0 region
* tests: trim redundant comments in test_get_rows vs0 region (follow-up)
* vulkan: bind tensor base for binary ops, pass full view_offs via push constants
For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, MUL, etc.),
bind the view_src base and pass the full view_offs divided by type_size via
push constant misalign_offsets. This avoids truncation when misalign_bytes is
not a multiple of quantized block size.
ggml_vk_tensor_subbuffer gains a use_view_offs parameter. When false, the
binding points to vk_tensor_offset (base) and size includes view_offs.
init_pushconst_tensor_offsets<binary> computes a/b/d_offset directly from
tensor->view_offs, which is always row-aligned and therefore exact.
Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).
All 223 GET_ROWS tests pass on Vulkan (NVIDIA RTX 5060 Ti).
* vulkan: bind aligned offset for binary ops, pass adjusted misalign via push constants
For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, etc.), bind
the buffer to an aligned position near the view offset (not the tensor base)
and pass the adjusted misalignment via push constants.
ggml_vk_get_adjusted_misalign finds the smallest misalign that is both a
multiple of minStorageBufferOffsetAlignment and type_size, ensuring
misalign/type_size is exact (no truncation for quantized block types).
ggml_vk_tensor_subbuffer gains use_view_offs parameter. When false, binds
to (target - adjusted_misalign) instead of the view_src base, keeping the
offset small enough for 16-bit/8-bit push constant fields.
Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).
All 223 GET_ROWS tests pass on Vulkan (NVIDIA RTX 5060 Ti).
* vulkan: bind aligned offset for binary ops, fix UMA offset mismatch
For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, etc.), bind
the buffer to an aligned position near the view offset (not the tensor base)
and pass the adjusted misalignment via push constants.
Added ggml_vk_tensor_physical_offset to unify physical offset lookup across
UMA and non-UMA devices. On UMA, resolves via ggml_vk_host_get(tensor->data);
otherwise uses vk_tensor_offset(t) + t->view_offs. Both get_misalign_bytes and
the new ggml_vk_get_adjusted_misalign helper build on top of this function,
so buffer bindings and push constant offsets are always consistent regardless
of device memory model.
ggml_vk_get_adjusted_misalign finds the smallest misalign that is both a
multiple of minStorageBufferOffsetAlignment and type_size, ensuring
misalign/type_size is exact (no truncation for quantized block types) while
remaining small enough for 16-bit/8-bit push constant fields
(adjusted_misalign < lcm(align, type_size)).
ggml_vk_tensor_subbuffer gains use_view_offs parameter. When false, binds
to (physical_offset - adjusted_misalign) on both UMA and discrete GPUs,
fixing a bug where the UMA host_get path previously skipped the adjusted
misalign binding and returned the target offset directly.
Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).
All 223 GET_ROWS tests pass on Vulkan (NVIDIA GeForce RTX 5060 Ti).
* finish misalignment fix
* supports_op changes for openvino/webgpu
---------
Co-authored-by: AiChiTuDouPian <15327701848@qq.com>
llama probes weight placement with a rope where all params are 0, so rejecting n_dims == 0 or freq_base == 0 puts rope_freqs on the CPU. That splits the decode graph at every full-attention layer (gemma-4-E2B: 5 splits instead of 2). Assisted-by: Claude Opus 5
implement q6k/q4k kernels Squashed from: feat: implement q6k kernel hex-q6k: improve unpack accuracy hex-q4_k: add support for Q4_K kernels Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* Improve CUDA graph usage for MTP * Rename field * Address review feedback
* switch jobs to ubuntu-slim * ubuntu slim almost takes 15 minutes for check requirements so use something faster
Advise read-mostly + prefetch to the allocating device for unified-memory buffers so lazily-migrated pages settle on the owning GPU. Co-Authored-By: opencode <noreply@opencode.ai>
Defer queued requests when resident-ctx-sum + candidate prompt would reach the threshold; deferred tasks retry on slot release. Auto-slot path only (explicit id_slot bypasses). Vanilla slot accounting: max(prompt-cache tokens, full task prompt length) per processing slot. Env: LLAMA_ARG_PARALLEL_CTX_THRESHOLD. Co-Authored-By: opencode <noreply@opencode.ai>
Mirror the defer SRV_INF with an admit line (resident + candidate < threshold) so boundary tests can show the exact accounting either way. Co-Authored-By: opencode <noreply@opencode.ai>
… exceeding threshold alone The gate deferred any request with resident + candidate >= threshold, including resident == 0. A lone request bigger than the threshold on an otherwise idle pool then defers forever: nothing is resident, so no slot release ever retries it. The threshold guards combined oversubscription between concurrent requests, not a single request's own size (that is bounded by the per-slot cap). Defer now requires resident > 0; a lone request always admits. Admit log distinguishes the lone case so the printed comparison stays accurate. Co-Authored-By: opencode <noreply@opencode.ai>
… one task A release event re-posted a single deferred task (FIFO head, or one explicitly requesting the slot). If that task re-deferred - e.g. the hydra#747 threshold gate deferring a large candidate - the release was burned and every other waiter stayed queued even with a slot idle, while brand-new arrivals kept getting served from the main queue (priority inversion, silent client hangs). Re-post the entire deferred FIFO in order (explicit-slot matches first) to the front of the main queue so every waiter is re-evaluated against the freed capacity before newer arrivals; tasks that still cannot proceed re-defer to the back, preserving FIFO order. Deterministic repro + production case: hydra#747 task-12239 hang (docs/investigations/740-results-report.md). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Assisted-by: Codex
…rowing Port upstream danielhanchen/llama.cpp qwen4exp/mtp: draft MTP block tensors (hc_head_norm/down/up), DECODER_MTP graph, ctx_other borrowing for shared-embd heads, hybrid MTP memory, gemma4-only mem-share gate. Assisted-by: Muse Spark (opencode)
Add --rpc-port flag to llama-server to expose a binary TCP listener that speaks the Hydra RPC protocol (see specs/rpc-protocol.md). New ops (0x30-0x32): STATE_GET (0x30) — serialize slot KV state, send as response payload STATE_PUT (0x31) — receive KV state bytes, restore to slot STATE_META (0x32) — slot metadata only (n_past, state_size, is_processing) Key design: - llama-server knows nothing about Store or sessions - key = slot_id as ASCII string; persistent TCP connections - Agent pipelines llama RPC socket → Store RPC socket (no 2nd 800 MB buffer) - HTTP /slots/:id/state/meta added as debug/curl fallback - Thread safety: is_processing() guard for M0; TODO task-queue in M1 Files changed: tools/server/server-rpc.h NEW — wire-format constants (MAGIC, op/status codes) tools/server/server-context.h add start_rpc_server(int port) declaration tools/server/server-context.cpp add hydra_* helpers + server_context::start_rpc_server tools/server/server-http.h add put() method declaration tools/server/server-http.cpp implement put() mirroring post() tools/server/server.cpp call start_rpc_server after model load; add HTTP meta route common/common.h add rpc_port = 0 to common_params common/arg.cpp add --rpc-port / LLAMA_ARG_RPC_PORT Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…+ split-cuda script Cherry-pick -m 1 of b933c30. Applied: socket_t::connection_id() (transport.cpp/h) + scripts/build-split-cuda.sh. Skipped: ggml-rpc.cpp hunks (sock-based resolve_bound_socket, stale remote_ptr guards, alloc-size fallback). N/A on this line: cdd1102 uses the dispatcher model and lacks the hydra-fork get_socket / re-resolve infra the PR side assumes (same verdict as prior port: rpc guard N/A). Keeping HEAD dispatcher code avoids a duplicate get_device_memory definition and ~15 dangling symbol refs.
…gguf-py) Port upstream ggml-org#28243 shared-tensor borrowing modules onto baseline cdd1102. C++ graph side (speculative.cpp, llama-arch, llama-context, llama-model.h, models.h, qwen4exp.cpp, llama-model.cpp hybrid gate) already matches upstream via local port cdd1102; this commit adds the 11 missing files: - conversion/{base,qwen,qwen4exp,bailingmoe3,command_r,dots3,glm}.py: mtp_shared_embd flag, qwen4exp MTP export (eh_proj fusion, mixer rename, dense compress_ratio trailer, mtp_only PLE skip) - convert_hf_to_gguf.py: --mtp-shared-embd flag - gguf-py/gguf/{constants,tensor_mapping}.py: NEXTN_HC_HEAD_* registration + QWEN4EXP NextN tensor list - src/llama-model-loader.h: whitespace sync with upstream SKIP per brief: overlap/priors/early-router/moe_grouped. No merge; leader merges.
- New tools/llama-engine/ target: thin exe linking server-context + llama-common + llama - include/llama-hydra.h + src/llama-hydra.cpp: Hydra-specific APIs isolated from upstream - llama_state_seq_get_data_to_fd: zero-copy KV state streaming to socket - llama_hydra_peer_reachable: TCP connect test for graceful degradation - Rename --rpc to --rpc-engine (keep --rpc as hidden alias for one release) - Graceful degradation: engine starts in SOLO mode if peer unreachable - Build verified (CPU-only, no warnings) Part of ggml-org#161 (Epic: phase-optimized inference engine) Co-Authored-By: opencode <noreply@opencode.ai>
Adapt llama-hydra to_fd wrapper to get_size+get_data+send instead of 69d49e0 zero-copy path; header comment now states the dropped zero-copy and why (avoid async machinery). Co-Authored-By: opencode <noreply@opencode.ai>
#44) server_context::start_rpc_server() declared hydra_rpc_ctx as an automatic-storage local, then handed hydra_rpc::start() its address to retain in a process-lifetime singleton (hydra_rpc::state().hydra_ctx), read by every future RPC connection from a bounded-thread-pool worker thread. The local went out of scope the moment start_rpc_server() returned, so the singleton retained a dangling pointer into a freed stack frame — a stack-use-after-return. Whether a given request's queue_tasks/queue_results dereference through it "worked" depended entirely on whether the freed stack slot had been reused yet, which explains the intermittent nature of #43 (task processed, result queued, but the response body never reaching the client socket). Fix: give ctx static storage duration. start_rpc_server() only ever runs meaningfully once per process (hydra_rpc::start() itself guards double-start), so `static` gives it exactly the lifetime the singleton already assumes. Also make hydra_send_all/hydra_recv_all log on failure (fd, bytes transferred, errno) instead of silently returning false — every caller already treats a false return as "give up" but had no way to tell a real socket error from `-lv` output, which the issue itself flagged as a diagnostic gap. Verified live on hardware (RTX 5060 Ti, Qwen3.5-2B-Q8_0): INFO (0x41) and CONFIGURE (0x40) previously timed out after 10s against a raw socat client, a raw Python client, and C# RpcClient. All three now round-trip correctly, including a second request on the same persistent connection. test-hydra-rpc-bind, test-hydra-configure-tier, and dotnet Tests.Shared EngineOpcodeTests all still pass. Closes #43 Co-authored-by: Hydra Engineering <hydra-engineering@local> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
…#451 Port of 860101b adapted to this lineage: progress fields (operation/progress/tokens_processed/tokens_total/elapsed_ms) reported by hydra_handle_state_meta from server_slot state/stats. M1 task-queue parts of the original (hydra_state result structs, PREFILL metrics, get_state route) skipped - no M1 base in this tree by design (pick 1b option B). Original: 860101b feat: expose slot progress/status for ggml-org#451 (#55)
Port of f7c8be1. Its checkpoint-before-decode core targets the M1 manual PREFILL batch loop (opcode 0x42), which does not exist in this tree - our prefill path is upstream update_slots, which already creates the checkpoint before llama_decode. No manual PREFILL loop, no hydra_rpc singleton, no set_routes_ptr base: those hunks skipped. Kept the portable prerequisite: guarded hydra_rpc.cpp link + include dir in tools/server/CMakeLists.txt (empty-safe EXISTS guard; no hydra_rpc dir in this tree yet). Original: f7c8be1 fix(hydra): honor checkpoint-before-decode invariant
slot.n_prompt_tokens_cache / slot.n_decoded do not exist on server_slot in this lineage (pre-existing breakage since 79ebdf4cf - the hydra RPC handlers never compiled). Use the stats equivalents the rest of this tree uses (slot.stats.n_prompt_cached / slot.stats.n_gen, cf. the n_decoded/n_prompt_tokens_cache task-result mappings). Found by CPU-only build-check of the 1-series tip (llama + server-context); both targets now build clean. Co-Authored-By: opencode <noreply@opencode.ai>
Upstream hc-ops (ggml-org#28901, in rebase base) moved the HC grouped-norm gamma to [n_embd, hc]; the shared-draft file still carries flat hc_dim gammas. The layer-norm declarations were already ported with ALLOW_RESHAPE during the rebase, but NEXTN_HC_HEAD_NORM was missed: the draft's own blk.*.nextn.hc_head_norm stayed flat and aborted graph_mtp in build_hc_mix (ggml_can_repeat fail, A=[2560,4,4] vs B=[10240]). Same reshape-tolerant treatment; verified by strict-config leg on the rebased tree.
Unmarked intermediates can have buffers recycled before extraction runs, yielding stale contents instead of ids. Stash per-layer selected_experts and set output when HYDRA_TRACE_ROUTES or HYDRA_PIN_FILE requests them. Off by default. Assisted-by: Muse Spark
Runtime-param-gated decode profiler, default OFF, single hot-path branch when off. Emits PROF window lines (64 steps) + final summary on stderr: NVML PCIe gen/width/rx/tx sampled during decode (dlopen, graceful degrade), per-step wall/CPU/device(CUDA-event)/sync split, host-device bytes per step, MTP draft/verify timing and acceptance via llama_profile_note. CUDA hook: record-only event bracket around graph compute, query-guarded out-of-band readback, never syncs. Co-Authored-By: opencode glm-5.3-flash <noreply@opencode.ai>
…p (§93) Prompt-eval step gets its own stderr line (n_tokens, tps, wall/cpu/dev/sync split) instead of being averaged into decode windows — the 26x prefill-gap discriminator. Co-Authored-By: opencode glm-5.3-flash <noreply@opencode.ai>
…tion) cudaEventQuery always failed (decode returns before GPU drain; server syncs at sampling) so t_dev was the -1 sentinel. EventSynchronize is bounded by the sync the caller already forces. Threshold 32->256 keeps the 42-token warmup from stealing the prefill tag. Co-Authored-By: opencode glm-5.3-flash <noreply@opencode.ai>
Splits record start/end per graph_compute call, so the readout only ever saw the last (tiny) CUDA segment — t_dev was ~0 even for real GPU prefill. Span begins before the sched compute and ends after it: one event pair per step. Co-Authored-By: opencode glm-5.3-flash <noreply@opencode.ai>
ddvnguyen
added a commit
that referenced
this pull request
Sep 20, 2026
Owner-authorized epic-scoped merge (not main): --profile-decode in-source decode profiler, gate-verified +0.31% with nil overhead on CUDA0 and 3060. Co-Authored-By: opencode glm-5.3-flash <noreply@opencode.ai>
Owner
Author
|
Owner-authorized epic-scoped merge executed: feat/decode-profiler merged into |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the §92-approved design (docs/design-decode-profiler.md in hydra_vortex superproject).
--profile-decode, default OFF; single hot-path branch when off (no allocation, no timing calls, no sync)llama_profile_notefrom speculative + server verify bracket)CI-gated. Do not merge without owner confirmation (§92 binding). Gate leg (flag-OFF vs pre-profiler binary, ±1.5%) runs before any measurement leg.