[Serve][LLM][SGLang] RL weight sync and inference-trainer coordination

[eicherseiji]

The ingress mixins for RL (collective_rpc, sleepable, pausable) call engine-protocol methods that SGLangServer doesn't implement. /collective_rpc, /sleep, /pause fail on SGLang deployments. This issue tracks implementing those methods plus an ObjectRef-based weight-transfer path and session affinity for multi-turn rollouts.

Subitems

• SGLangServer implements collective_rpc / sleep / wakeup / is_sleeping / pause / resume / is_paused.
• ObjectRef-based weight-transfer wrapper for SGLang.
• Session / multi-turn affinity for SGLang: routing pins follow-up requests to the same replica, exposed through a small SGLang-scoped session API.
• SGLang-specific reference RL driver exercising deploy → open session → stream → pause → sync → resume.
• SGLang RL weight-sync release test: bounded sync latency, no requests stuck across a sync boundary.

Open questions

• Whether the inference-trainer handoff becomes a documented first-class contract for RL frameworks, or stays implicit in the mixin surface.
• Where session affinity lives: Serve routing, engine-level session manager, or both.
• Weight-sync latency budget. Drives whether broadcast-and-reload is acceptable or delta sync is needed.
• Composition with PD disaggregation (separate rollout-decode, shared prefill pool): natural fallout or dedicated design pass?

Upstream coordination

• update_weights_from_object_ref-style API accepting Ray references directly.
• abort_request acknowledgement so cancellation is deterministic.

[0z5a]

Hi @SohamRajpure — I'd like to take a focused state-contract regression slice under #62794, following the control-plane implementation merged in #63021 (not the separate ObjectRef/session-affinity work).

On current Ray master 317c2888eade3c294c4fdb46eff9d9ec290b08f2, SGLangServer.wakeup(tags=[]) forwards the empty list successfully but does not clear _sleeping_tags. SGLang's resume_memory_occupation treats both None and [] as all memory regions; Ray currently only clears its tracked state for None. This can leave is_sleeping() returning True after an acknowledged full wakeup.

I have a local patch changing that state check to if not config.tags, documenting the empty-list meaning, and extending the existing SGLang release suite (rather than adding a competing control-plane implementation). A 24-case CPU contract test uses the real Ray class and real SGLang request types, with only the tokenizer-manager calls mocked. It covers omitted/None/empty tags, selective wakeup, backend errors/cancellation, and state updates only after acknowledgement:

• Unmodified Ray: 4 failures, 20 passes, all four failures are stale state after empty-list wakeup.
• Patched Ray: 24 passes.
• Command: python -m pytest release/llm_tests/serve/test_llm_serve_sglang.py -k TestSGLangSleepState -q.
• SGLang source: 0bcd822377da7b5718e674eaf9c870d349424dd1; local Python 3.12.13 / Torch 2.13.0 / Transformers 5.12.1. These CPU adapter-state tests mock backend calls; the GPU checks below do not.

Validation update (2026-09-06 UTC): The final test fixture now enables memory saving and weight CPU backup, obtains the installed preload library through torch_memory_saver.configure_subprocess(), and passes it into the Ray actor's LLMConfig.runtime_env. The driver/container itself has no LD_PRELOAD override.

• Real NVIDIA L20 / Qwen2.5-0.5B-Instruct (7ae557604adf67be50417f59c2c2f167def9a775), Torch 2.13.0+cu130 / SGLang 0.5.19: the same 14 control tests yield 4 failures / 10 passes on unmodified Ray, 14 passes on the fix, zero skips. Three baseline failures are the empty-wakeup cases; the fourth is the following selective-tag test seeing their stale state, not an independent selective-wakeup defect.
• A separate final-source run including the existing streaming and tokenization tests gives 19 passes, zero skips. Each of the nine omitted/None/empty sleep-wakeup cycles checks at least 256 MiB release and restoration plus exact greedy-output equality. Recorded GPU totals are approximately 38,384 → 1,732 → 38,384 MiB. This is real Serve/SGLang execution, not a mocked backend.
• Final-source Linux CPU confirmation: 24 passes. Full applicable pre-commit hooks pass. The Ray core is nightly wheel dede511b61fbf6383f5b03cdcc1fa2c0128efafa, using the documented Python-only development setup for the source under test; not a full C++ source build.
• Selection for the expanded run: python -m pytest release/llm_tests/serve/test_llm_serve_sglang.py -k '(sleep or pause or reset or test_sglang_serve_e2e or streaming or tokenize or batched) and not TestSGLangSleepState and not multi and not pipeline' -q --timeout=600. The validation harness first initializes Ray with one GPU and overrides the model path to the verified offline checkpoint.

These results qualify the adapter state fix and its single-GPU test fixture, not full RL training, ObjectRef weight transfer, sessions, distributed recovery, or performance. Initial dependency/preload startup failures were retained separately and are not counted as defect reproductions.

I checked the issue comments and open PRs for 62794, SGLang, sleep, and wakeup and did not find the same fix in flight. Would this be a useful bounded subtask to assign, or is there an existing branch you would prefer me to contribute the regression to?

AI assistance was used for this audit/patch/tests. Publication update (2026-09-07 UTC): after the human submitter confirmed line-by-line review and personally running the relevant tests, the unchanged validated commit 1c5ea3ee38ed10ec4eb84becc81e9cbeb9293135 was pushed from the submitter's fork as PR #65969. It contains this bounded state fix and release-test coverage only; #62794 as a whole is not completed. No permission to modify anyone else's branch is assumed.