ggml: INT8/ternary types and fused CPU ops for the VibeASR pipeline

[XsquirrelC]

Adds the ggml pieces VibeASR's INT8 CPU pipeline needs, ahead of the model port itself. Purely additive: two new types, five new ops, no change to any existing type, op, or code path.

What's here

GGML_TYPE_I8_S / GGML_TYPE_I2_S — int8 and ternary weights with a per-tensor scale, one F32 stored after the payload (padded to 32 bytes so arena alignment holds). Per-tensor rather than per-block is deliberate: a 32-value-block int8 type would just be Q8_0. ggml_type_extra_bytes() is the single place the layout is encoded; it returns 0 for every other type, so nothing existing changes size.

Five fused ops — mul_mat_add, mul_mat_add_relu, add_scaled, rms_norm_scaled, im2col_asym. They keep activations int8 end to end: each op requantizes its own output and writes the new scale in-band, so the chain never detours through F32. im2col_asym exists because ggml_im2col cannot express asymmetric padding, which the VAE's causal convs need.

Requantization is three-phase — stage F32 into params->wdata, ggml_barrier and reduce absmax across threads, then quantize the slice — so results are independent of thread count.

AVX2 + NEON kernels for the I8_S ops. The dot product uses the maddubs sign trick on AVX2 and vdotq_s32 (or vmull_s8 + vpadalq_s16 without dotprod) on NEON, widening to int32 every block rather than batching in int16, since each maddubs lane already reaches 32512. Anything past the vector width is done scalar, so contraction lengths shorter than one step are handled rather than truncated away.

Measurements

x86-64 AVX2, single thread. Three changes, separately attributed:

	512×512×512 matmul	K=8 depthwise, 512ch × 4096pos
scalar baseline	5.39 ms	—
+ vectorized requantize, roundf → rintf	3.99 ms (1.35×)	22.0 ms
+ batched depthwise dot	—	14.4 ms (1.53×)

Most of the matmul gain came from the requantizer, not the dot product: roundf was a PLT call per element. rintf also fixes a consistency problem — it rounds half-to-even, matching _mm256_cvtps_epi32 and vcvtnq_s32_f32, so each vector body agrees with its own scalar tail.

Both mul_mat_add paths batch their dot products, with the operand roles inverted between them: the matmul path batches over output channels, the depthwise path over positions — in each case so the strided operand becomes the batched one and the results land contiguous, which is what lets the scale+bias+absmax pass vectorize too.

Why not TQ2_0

Worth addressing up front, since ggml already ships a ternary type. Every I2_S tensor in the model is shape-compatible with it (n_embd=1536, n_ff=8960, both divisible by 256), and TQ2_0 costs only 3% more (2.0625 vs 2.0000 bit/weight) with finer per-block scales. So it looked like the right answer until measured — same machine, same shapes, single thread:

shape		TQ2_0	I2_S
1536×8960	N=1	0.260 ms	0.222 ms	1.17×
8960×1536	N=1	0.236 ms	0.183 ms	1.29×
1536×8960	N=130	37.06 ms	18.11 ms	2.05×
8960×1536	N=130	31.56 ms	17.86 ms	1.77×

The prefill gap is multi-column GEMM blocking, which TQ2_0's per-column vec_dot path does not have. Over 28 layers that is 3.31 s vs 1.69 s for a 130-token prefill. The I2_S kernels are not in this PR yet — this is the argument for why they are coming rather than being dropped in favour of TQ2_0.

Tests

tests/unittests/test_i8_s_fused_ops.cpp, 24 cases, every op against a plain-loop reference computed from the same dequantized inputs. Comparisons are byte-exact wherever both sides do the same arithmetic; where the op reaches the same value by different arithmetic (integer sum-of-squares vs float normalize) the tolerance is one quantization step.

Shapes are chosen to hit the edges rather than round numbers: IC ∈ {13, 64, 67} for all-scalar / no-tail / has-tail contractions, OC ∈ {48, 100} for chunk wraparound, depthwise K ∈ {7, 36} × N ∈ {19, 150}. Every case runs at 1 and 4 threads, so a missing barrier shows up as a scale that depends on thread count.

Two mutation tests confirm the assertions have teeth: disabling the scalar tail fails IC=67, and reintroducing int16 accumulation fails IC=64 by 31%.

Full suite: 40/40, no skips.

[0xShug0]

@XsquirrelC Thanks for bringing the official VibeASR integration to audio.cpp!

One minor issue. I built the unittests but failed.

cmake -S . -B build/debug \
  -DCMAKE_BUILD_TYPE=Debug \
  -DENGINE_BUILD_TESTS=ON \
  -DAUDIOCPP_MODEL_SET=core \
  -DAUDIOCPP_MODELS='' \
  -DENGINE_ENABLE_CUDA=OFF \
  -DENGINE_ENABLE_VULKAN=OFF

cmake --build build/debug --target i8_s_fused_ops_test -j$(nproc)

[100%] Linking CXX executable bin/i8_s_fused_ops_test
/usr/bin/ld: CMakeFiles/i8_s_fused_ops_test.dir/tests/unittests/test_i8_s_fused_ops.cpp.o:
in function `(anonymous namespace)::test_rms_norm_scaled(int)':
/media/leo/Share/audio.cpp/tests/unittests/test_i8_s_fused_ops.cpp:118:(.text+0xc74):
undefined reference to `ggml_graph_compute_with_ctx'

Could you document the exact CMake command you used to build and run it? That would make the expected validation path clear. Alternatively, the test could run through the public ggml backend registry API so it works in both static and dynamic backend builds, but documenting the intended test build configuration is enough.

[XsquirrelC]

Closing in favour of #447. You asked for two PRs in microsoft/VibeASR.cpp#10 — one for the additive ggml changes, one for the model integration — so this four-PR stack has been reorganized into exactly that: #447 (ggml) and #448 (model). Same code, no functional change; sorry for the churn.