Cant disable KV Cache Type

[jxs714]

Hey guys,

For some reason there is no way to disable KV Cache at all in the GUI. Am I missing something?

[pwilkin]

KV cache type should default to F16 since that's the actual default.

[jxs714]

Thank you. So this defaults to non-cache mode?

[pwilkin]

KV cache is always enabled - that's a core attention mechanism that guarantees model processing does not take ages, I don't know of any engine that allows fully disabling KV cache these days. The setting only determines if the cache is quantized - reduced to a lesser precision to reduce memory usage.

[jxs714]

True, thanks so much for the information and response. Keep up the great work.