[deepseek_v4] performance: use coli_v4_route_bf16 in moe_token

[weber-software]

Summary

I have let deepseek_v4-flash optimize its own engine. And it found something:

The single-token decode path moe_token was decoding the entire n×d gate matrix (n_routed_experts × hidden_size, e.g. 256×7168 ≈ 1.8M bf16 values) into a freshly-malloc'd float buffer on every token, every layer, then calling the float-gate router coli_v4_route.

Meanwhile the prefill/batch path already used the bf16-gate router coli_v4_route_bf16, which reads the raw bf16 gate directly and avoids both the decode pass and the large allocation.

I changed moe_token to use coli_v4_route_bf16 with the raw bf16 gate, mirroring the batch path. This eliminates the O(n·d) gate-decode pass and the ~7 MB per-call allocation from the decode hot loop.

I have tested the performance with multiple runs:

rm -f ds4flash-0731/.coli_usage && time ./deepseek_v4 ds4flash-0731 "Hi" --thinking --max-tokens 1 --memory-gb 40 --no-dspark

Without the optimization it takes ~13 seconds and with the optimization ~11 seconds (AMD Ryzen 9 3900X 12-Core Processor, 64GB DDR4).
I have deleted the coli_usage before every run and made sure that the experts are in OS cache (single token + enough free memory).

I also tested the generation of longer texts and compared them (with/without the optimization) and they had the same output. I do not cite them because they contained much more time generating (and i think the optimization is only for prefill?).

Validation

• make -C c check
• CUDA changes were tested with make -C c cuda-test (if applicable)
• Performance claims include hardware, commands, and repeatable measurements

Compatibility

• The default CPU build remains dependency-free
• No model files, generated binaries, or benchmark artifacts are included

Disclaimer

deepseek_v4-flash has done the work and i'm unsure if my validations are correct. So please check them yourself too.