Reduce peak memory for prompt_logprobs requests - #907

Open

50h100a wants to merge 3 commits into

mainfrom

prompt_logprobs_mem

Commits on Dec 16, 2024

do not use cached chunks for prompt_logprobs
50h100a
committed
bc1a2bd
reduce sampler peak memory usage
50h100a
committed
d49ead7

Commits on Mar 12, 2025

Merge branch 'main' into prompt_logprobs_mem
AlpinDale
authored
07a80c0