server: tell a streaming client when its slot is parked and restored - #190

Draft

danielhanchen wants to merge 25 commits into

feat/server-side-preemptionfrom

feat/server-side-preemption-notify

Commits on Sep 5, 2026

server: tell a streaming client when its slot is parked and restored
danielhanchen
committed
65f8f54

Commits on Sep 6, 2026

server: make the result queue timeout a deadline, so a parked stream is kept alive
danielhanchen
committed
5a791e0
Merge branch 'feat/server-side-preemption' into feat/server-side-preemption-notify
danielhanchen
committed
e9aeb3a
server: tell the stream about a park made as a last resort
danielhanchen
committed
5864dae
Merge branch 'feat/server-side-preemption' into feat/server-side-preemption-notify
danielhanchen
committed
301f480
Merge branch 'feat/server-side-preemption' into feat/server-side-preemption-notify
danielhanchen
committed
76baf1a
server: keep the park notices of a multi-prompt stream apart
danielhanchen
committed
7fb42b5
Merge branch 'feat/server-side-preemption' into feat/server-side-preemption-notify
danielhanchen
committed
53c59a1
server: a stream parked before its first token starts with the notice; the rotation park is announced
danielhanchen
committed
ab40f16
Merge feat/server-side-preemption: keep only the shared prefix of a reused slot before it is sized or parked
danielhanchen
committed
356978a
Merge feat/server-side-preemption: planner accounting for reused slots and waiting children
danielhanchen
committed
3306d4b
Merge feat/server-side-preemption: the rotation parks the resident that lets the head in
danielhanchen
committed
7efef20
Merge feat/server-side-preemption: a parked slot survives an aborted round; a rotation counts the head's bytes as leaving
danielhanchen
committed
3d179fd
Merge feat/server-side-preemption: the leader is measured by what a reused slot keeps
danielhanchen
committed
a06419d
Merge feat/server-side-preemption: parent and child alone in the pool test
danielhanchen
committed
1d52885

Navigation Menu

server: tell a streaming client when its slot is parked and restored - #190

server: tell a streaming client when its slot is parked and restored#190
danielhanchen wants to merge 25 commits into
feat/server-side-preemptionfrom
feat/server-side-preemption-notify

Commits on Sep 5, 2026

server: tell a streaming client when its slot is parked and restored

Commits on Sep 6, 2026

server: make the result queue timeout a deadline, so a parked stream is kept alive

Merge branch 'feat/server-side-preemption' into feat/server-side-preemption-notify

server: tell the stream about a park made as a last resort

Merge branch 'feat/server-side-preemption' into feat/server-side-preemption-notify

Merge branch 'feat/server-side-preemption' into feat/server-side-preemption-notify

server: keep the park notices of a multi-prompt stream apart

Merge branch 'feat/server-side-preemption' into feat/server-side-preemption-notify

server: a stream parked before its first token starts with the notice; the rotation park is announced

Merge feat/server-side-preemption: keep only the shared prefix of a reused slot before it is sized or parked

Merge feat/server-side-preemption: planner accounting for reused slots and waiting children

Merge feat/server-side-preemption: the rotation parks the resident that lets the head in

Merge feat/server-side-preemption: a parked slot survives an aborted round; a rotation counts the head's bytes as leaving

Merge feat/server-side-preemption: the leader is measured by what a reused slot keeps

Merge feat/server-side-preemption: parent and child alone in the pool test

Commits on Sep 7, 2026

Merge feat/server-side-preemption: recurrent caches served without preemption; the rotation holds both states at once

Merge feat/server-side-preemption: a started slot's reservation counts from the prefix it keeps

Merge feat/server-side-preemption: the shift and the planner inside the guarded step

server: a parked stream keeps a shorter ping interval the request asked for

Merge feat/server-side-preemption: what a started slot keeps is decided by one rule

Merge feat/server-side-preemption: the resume order read on every load

Merge feat/server-side-preemption: the rotation-budget test keeps its resident cycling past the rotation delay

Merge feat/server-side-preemption: the resume order belongs to the context; the recurrent flag assigned on every load

Merge /mnt/disks/unslothai/daniel3/workspace_12/llamacpp_fork into feat/server-side-preemption-notify

server : validate a started slot's prompt before it can be parked