ggml-cpu: ternary I2_S matmul kernel (VibeASR, 3/4)

[XsquirrelC]

Stacked on #440 (which is itself stacked on #438). This branch contains both of those plus one commit; only ggml-cpu: ternary I2_S matmul kernel (69de525) is new here.

Adds the CPU matmul for GGML_TYPE_I2_S, the BitNet-style ternary weight type that VibeASR.cpp runs its Qwen2 decoder on. #438 registered the type, its packing, and its (de)quantizers; this is the kernel that makes a ternary weight usable in a graph. The decoder graph, loader, and session are the next PR — this one is numeric only.

Interface

Plain ggml_mul_mat with an I2_S src0 and an F32 src1, F32 out. No new op, no new graph-building call, nothing for a model to opt into.

ggml_compute_forward_mul_mat branches to a dedicated function when src0->type == GGML_TYPE_I2_S. It cannot use the vec_dot_type path: that contract quantizes src1 one row at a time into a fixed row_size and hands the kernel two row pointers, while I2_S needs the per-row activation scale and the per-row int8 sum to survive into the epilogue. Those live in a sidecar after the quantized rows in params->wdata.

The branch is 8 lines in ggml_compute_forward_mul_mat and 10 in ggml_graph_plan's GGML_OP_MUL_MAT case (work size); everything else is new code in ops.cpp / vec.cpp. For reference, upstream's own fork does this by patching the body of the generic ggml_compute_forward_mul_mat inline — including a src1_col_de = wdata + i11*nb11/4 pointer reinterpretation — which is what this avoids.

Arithmetic

I2_S stores the ternary values as the codes {0, 1, 2}, not as {-1, 0, +1}. That is deliberate: unsigned codes let the integer multiply-add instructions (vpmaddubsw, vdot) be used directly on the packed nibbles with no sign extraction. The cost is a bias, which comes out algebraically:

sum(code*q) = sum((w+1)*q) = sum(w*q) + sum(q)

so the kernel returns sum(code*q) and the epilogue subtracts the row's int8 sum. Both scales are multipliers (amax/127 for activations, the weight absmax for I2_S), so they combine once at the end:

dst[oc] = (float)(dot - row_sum) * (weight_scale * act_scale)

Everything up to that last multiply is int32, so the unit test compares bit-exactly against a plain-loop reference rather than with a tolerance — a wrong packing that happens to be numerically close would otherwise pass.

Kernels

ggml_vec_dot_i2_i8 in vec.cpp, alongside ggml_vec_dot_i8_i8 from #438. The packing (128 values per 32-byte group; byte gp holds group-relative positions gp, 32+gp, 64+gp, 96+gp in bit pairs 6/4/2/0) is chosen so the four code lanes of a group line up with four consecutive 32-value slices of the activation row — no shuffling on either side.

• AVX2: one 32-byte load yields all 128 codes; four vpmaddubsw against four 32-byte activation loads. Accumulation widens to int32 every eight groups, not at the end of the row: a lane holds a sum of two code*int8 products, at most 2*2*127 = 508, and eight groups contribute 32 of them, 32*508 = 16256 — inside int16 with room, but 16 groups would not be. Upstream accumulates 32 groups in int16, which can overflow on activations near full scale; test_accumulator_headroom drives exactly that case (every code 2, every activation +127).
• NEON/aarch64: vdotq_s32 when __ARM_FEATURE_DOTPROD is available, otherwise vmull_s8 + vpadalq_s16. Half a group per iteration since the register is 16 bytes wide.
• Scalar: the portable path, and the fallback on targets with neither ISA. Not dead code — it is what the reference values in the test are checked against on any other target.

Threading: activation rows are quantized striped across threads (a row-wide absmax cannot be split), one barrier, then the output features are split. Sweeping the whole batch inside the output-feature loop means a weight row is streamed once and reused across every column, which matters because this op is bandwidth-bound on the weights, not on the activations.

Tests

tests/unittests/test_i2_s_mul_mat.cpp, registered in ctest as i2_s_mul_mat_test. Every case runs at nth=1 and nth=4, since a missing barrier or an overlapping output split shows up as a thread-count-dependent result.

• test_pack_layout — decodes the packing by hand (not via ggml_i2_s_to_float) so the layout the kernel reads is pinned independently of the dequantizer written alongside it, and checks nb[1] == K/4.
• test_mul_mat — K = 128 (one group), 1024 (exactly the eight groups the AVX2 body accumulates in int16), 1152 (that plus a shorter second flush); N = 1 (decode-time lm_head shape), 64 (one output-channel chunk), 100 (two), 70 (unaligned), 2 (fewer output features than threads, so tail threads get an empty range).
• test_mul_mat_batched — 3D src1, so the row indexing walks nb12 and the output walks nb2.
• test_accumulator_headroom — the int16 worst case described above.
• test_degenerate_scales — all-zero weights (scale 0) and a zeroed activation row, neither of which may produce a NaN. This is the guard on the multiplier convention: a reciprocal convention divides by zero in both cases.

Build

Release, gcc, x86-64 AVX2, 24-core: full ctest 69/69 pass, 0 failures (3 skipped for missing model assets, as before). i2_s_mul_mat_test passes at both thread counts. No behaviour change for any existing type — the new branch is reached only when src0->type == GGML_TYPE_I2_S, and the work-size addition is inside the same if.

[XsquirrelC]

Closing in favour of #447. You asked for two PRs in microsoft/VibeASR.cpp#10 — one for the additive ggml changes, one for the model integration — so this four-PR stack has been reorganized into exactly that: #447 (ggml) and #448 (model). Same code, no functional change; sorry for the churn.