server: remove two host side stalls in the decode loop at high concurrency - #200

Draft

danielhanchen wants to merge 10 commits into

perf/server-result-queue

Commits on Sep 6, 2026

server: deliver a result to the one thread waiting for it
danielhanchen
committed
fb16b8f
server: only keep per token probabilities when the request asked for them
danielhanchen
committed
755fb62

Commits on Sep 9, 2026

server: trim comments in the result queue change
danielhanchen
committed
29fd150
server: do not abort the process when a request's ids have left the waiting list
danielhanchen
committed
18abc68
server: widen the result queue test to isolation, teardown, broadcast and churn
danielhanchen
committed
170a675
server: filter results by requested id, wake on registration, honour terminate()
danielhanchen
committed
cdb0ecb
server: wake every subset receiver, and keep the response test buildable on Windows
danielhanchen
committed
2af4f90
server: wake a reader whose ids span several waiters
danielhanchen
committed
4e86543
server: keep arrival order across waiters, and wake a reader whose waiter is discarded
danielhanchen
committed
031b480
server: keep a partly registered receive on the shared condition
danielhanchen
committed
5488483