rpc: one tensor read per decode step and asynchronous split copies - #193

Draft

danielhanchen wants to merge 20 commits into

feature/rpc-stage2

Commits on Sep 5, 2026

server: add --pipeline-groups to run the slots over several contexts
danielhanchen
committed
c238f82
server: harden the pipeline group decode loop
danielhanchen
committed
f96bb35
rpc: serialise the client connection and track the stored graph per connection
danielhanchen
committed
2bd1359
rpc: batch the tensor reads of a step and move the split tensors asynchronously
danielhanchen
committed
4dc1b57
rpc: keep the asynchronous copy to the backend's own buffer type
danielhanchen
committed
c2f055a

Commits on Sep 8, 2026

reduce comment volume in the rpc stage2 changes
danielhanchen
committed
ec74a60

Commits on Sep 9, 2026