server: preemption notices, asynchronous parks and exact concurrency together - #197

Draft

danielhanchen wants to merge 503 commits into

feat/server-side-preemptionfrom

feat/server-side-preemption-integration

This pull request is big! We're only showing the most recent 250 commits

Commits on Sep 6, 2026

Commits on Sep 7, 2026

Commits on Sep 8, 2026

hexagon: add RELU and LEAKY_RELU ops (#28585)
tboinovski1
authored
050dde5
model : support Kimi-K3 recurrent-state rollback (#28466)
soulmachine
authored
9dcf84e
llama, server : apply a recorded context shift before a slot is parked
danielhanchen
committed
f926c3f
server : validate every prompt of a request before any task is queued
danielhanchen
committed
2fdfbe1
server : park a model with a recurrent half synchronously
danielhanchen
committed
63b6e6f
common : refuse exact concurrency for an M-RoPE model with a projector
danielhanchen
committed
d811fe8
tests : fewer preemption tests, the same behaviours
danielhanchen
committed
668e138
server : report a slot with a copy in flight apart from a parked one
danielhanchen
committed
7e650c4
server : park by dropping the cells when the preempt-ram budget is spent
danielhanchen
committed
168007d
server : reserve a whole media chunk before the planner lets it decode
danielhanchen
committed
fb634ca
Merge branch 'feat/preemption-fixes-2' into feat/server-side-preemption-integration
danielhanchen
committed
1c2c675
ggml : update ggml_prec specification (#26675)
ggerganov
authored
5a6caa0
opencl: properly handle non-contiguous inputs to conv2d (#28503)
lhez
authored
7d701b5
chat : split specialized parsers into common/parsers (#27764)
pwilkin
authored
895c045
Merge branch 'feat/preemption-tests' into feat/server-side-preemption-integration
danielhanchen
committed
3f0afe4
Fix Vulkan-Hpp handle usage on 32-bit targets. (#22892)
miyanyan
authored
f014bfe
vulkan : fuse UNARY(GELU|SIGMOID|SILU|SOFTPLUS) + MUL (#27220)
Ankk98
authored
64e9bce
llama : reject a shared-token batch under exact concurrency as invalid, not as a full cache
danielhanchen
committed
522e336
llama : reject repeated positions of one sequence under exact concurrency as invalid input
danielhanchen
committed
f8b2f51
server : /props says whether exact concurrency runs, a recompute park keeps its draft bound, its media and its prompt validation, and replays decoded tokens one per step under the mode
danielhanchen
committed
73cc64b
llama : add missing headers (#28566)
peppergrayxyz
authored
ca86fb2
ci : add PYTEST_WORKERS=1 to fix server-self-hosted job (#28603)
danbev
authored
1744c6b
ci : disable npm gha cache (#28600)
CISC
authored
03fa73c
spec: single device drafter should create meta backend wrapper (#28390)
am17an
authored
415e909
metal : fix idle threads in mul_mv_iq3_xxs for ne00 < 1024 (#28086)
masterFoad
authored
88ada91
server : apply checkpoint min-step eviction only when the checkpoint list is full (#28302)
masterFoad
authored
5d806aa
Revert "ggml-cuda : restore prop.integrated on HIP builds (#24233)" (#28604)
IMbackK
authored
d4389a4
ggml : fix msvc+clang ggml_vld1q_u32 (#28284)
sarahwu185
authored
9113cc1
llama: disable lazy tensor loading by default on iGPUs (#28326)
0cc4m
authored
f3f1a8f

Commits on Sep 9, 2026

Commits on Sep 10, 2026