server: add KV cache quantization flags (--kv-bits, --kv-group-size, --quantized-kv-start)#1353
Open
soobrosa wants to merge 4 commits into
Open
server: add KV cache quantization flags (--kv-bits, --kv-group-size, --quantized-kv-start)#1353soobrosa wants to merge 4 commits into
soobrosa wants to merge 4 commits into