[Serve] [SGLang] PD disaggregation by limarkdcunha · Pull Request #63741 · ray-project/ray

limarkdcunha · 2026-05-30T03:23:32Z

Description

A POC PR for the my proposal (#63257) related to Ray Serve SGLang PD disaggregation support.

Related issues - #62792 #63257

gemini-code-assist

Code Review

This pull request introduces SGLang Prefill-Decode (PD) disaggregated LLM serving by adding SGLangPDPrefillServer and SGLangPDDecodeServer deployments, and updating the SGLang engine to pass bootstrap coordination fields. Feedback on these changes highlights critical issues: first, setting attributes directly on Pydantic v2 models will raise runtime errors, so object.__setattr__ should be used instead; second, the prefill generator must be consumed in a background task to prevent premature garbage collection and engine hangs; third, asyncio needs to be imported to support this background task; and finally, the new servers should be added to the __all__ list in deployment.py to be properly exposed in the public API.

jeffreywang88 · 2026-06-02T04:04:56Z

can you add a target in release_tests.yaml? example: https://github.com/ray-project/ray/blob/master/release/release_tests.yaml#L4643-L4661

limarkdcunha · 2026-06-04T00:59:30Z

Added the target in release_tests.yaml. Thanks

jeffreywang88

Left some design questions for you to consider while revising the RFC.

jeffreywang88 · 2026-06-24T18:00:06Z

+to the right place. The decode server generates the bootstrap_room upfront and
+dispatches both sides simultaneously — it does not wait for a prefill response
+before starting decode.


Looks similar to the parallel handoff pattern (https://github.com/ray-project/ray/pull/63950/changes#diff-ccc2365e3ea6c5f4fd309223ecbaaa15d44deb1d510f0acf985604060a7f1747R56). Could you assess the feasibility of leveraging the established concurrent handoff mechanism?

jeffreywang88 · 2026-06-24T18:01:25Z

+        # LLMServer.get_deployment_options calls get_engine_config(), which
+        # unconditionally imports vLLM. The SGLang byod image uninstalls vLLM,


We will need to somehow decouple vLLM from the common paths. It might require a refactor. Could you explore some options?

jeffreywang88 · 2026-06-24T18:01:59Z

+         carrying the same prefill bootstrap host/port/room — the decode
+         KVReceiver connects to that prefill bootstrap server and blocks
+         internally waiting for the KV cache to arrive.
+      5. Streams the decode response back to the client.


This is exactly what we need, nice!

jeffreywang88 · 2026-06-24T18:02:50Z

+    For each chat/completions request it:
+      1. Reads the PREFILL node's bootstrap_host and bootstrap_port, fetched
+         from the prefill deployment at init (the bootstrap server lives there).
+      2. Generates a unique bootstrap_room integer.


In the design doc, it'd be great to clarify what bootstrap_room does and where it lives.

jeffreywang88 · 2026-06-24T18:03:56Z

+        # TODO: Users currently need to set disaggregation_mode manually in engine_kwargs.
+        # The builder should set this automatically since it already knows which
+        # config is prefill and which is decode.


We should emulate ray serve LLM's PD API for vLLM here. The user API should be similar.

jeffreywang88 · 2026-06-24T18:04:38Z

+
+        Unlike the vLLM flow, we do not wait for a prefill response before
+        starting decode — the bootstrap_room is established upfront and both
+        sides coordinate directly via SGLang's bootstrap server.


Is this bootstrap server strictly required?

jeffreywang88 · 2026-06-24T18:05:01Z

    pass


+@PublicAPI(stability="beta")


We should start from alpha.

jeffreywang88 · 2026-06-24T18:08:40Z

+        - UCX_TLS=all
+        - UCX_NET_DEVICES=all


Could you help me understand why we need these?

When we use SGLang with NIXL, it uses a networking library called UCX to move data directly from one GPU to another.

By default, our testing environment (the CI box) has strict or limited network paths. Without these two settings, UCX gets confused, picks a network path that doesn't actually have access to the GPUs, and the data transfer crashes.

Here is what these two lines specifically tell UCX to do:

UCX_TLS=all: Tells it, 'You are allowed to use any available transport method (like shared memory, TCP, or CUDA-IPC) to move this data.'

UCX_NET_DEVICES=all: Tells it, 'You are allowed to look at all network devices on this machine to find a working path.'

Basically, it forces UCX to stop being picky and use whatever path works on our test machines so the test doesn't fail. Once we know the absolute bare minimum network settings needed for this specific CI box, we can narrow this down."

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

cursor

Stale Bugbot comment from a previous run.

cursor · 2026-08-15T14:34:29Z


+    # Does this handle when every worker is full ?
+    # Ans - it actually does it the correct way by parking the request and going to sleep,
+    # rather than spinning and wasting CPU.


Accidental personal notes in router

Low Severity

A Q&A-style personal note (Does this handle... / Ans - ...) and a # background dispatcher aside landed in the core Serve request router. These read as scratch notes rather than intentional docs and do not belong in production code.

Additional Locations (1)

python/ray/serve/_private/request_router/request_router.py#L1375-L1376

^{Reviewed by Cursor Bugbot for commit e467352. Configure here.}

0z5a · 2026-09-06T18:54:21Z

Hi @limarkdcunha — I saw that this POC is still in progress, including the hardware-availability note. I'd like to help with a bounded validation slice on your existing connector instead of starting another P/D implementation.

I inspected head e4673524cf3e0067bbd5f9ca5475315388517ee0. The connector-level bootstrap tests currently use _req() with a SimpleNamespace and a model_copy that returns another empty namespace. Could I contribute a small extension using the actual SGLang-backed Ray ChatCompletionRequest and CompletionRequest models? Proposed checks are matching host/port/room across prefill and decode, single-token/non-streaming clamping only on the prefill copy, preservation of the decode/original request (including nested fields), and missing-peer rejection before mutating the request.

This would validate the concrete bootstrap/request-shaping contract, not duplicate your builder, orchestrator, room-uniqueness tests, or claim that CPU tests replace NIXL transfer or the required two-node release test. I can also help with a separately reported single-host GPU smoke once the environment is ready; I cannot claim two-node coverage from one shared L20 host.

Is this a useful subtask to take on, and is this head still the right integration target? I will keep any preparation in a local branch until we agree on how to integrate it; I am not assuming permission to push to your branch. AI assistance is being used, and Ray's human review/local-test submission requirements will still apply before opening any code PR.

Preparation update (2026-09-06): the bounded local extension now passes 38 CPU cases (10 existing + 28 using the real request models), with the five GPU release cases explicitly deselected. All applicable pre-commit hooks pass. Production code is unchanged. This result does not establish GPU/NIXL/KV-transfer or two-node correctness; the initial new-test failures were test-side nested-message assumptions, not upstream defects. The patch remains local and unpushed pending integration agreement and the required human review.

Single-host GPU follow-up (2026-09-06 UTC): the unchanged PR head e4673524cf3e0067bbd5f9ca5475315388517ee0 now passes all 15 original release-module tests, zero failures/errors/skips, on two L20 GPUs. That is the five real NIXL P/D chat/completion/streaming/concurrent-request tests plus ten original connector tests, separate from the 38 CPU cases above. No test assertions or PR production code were changed. The Qwen2.5-0.5B-Instruct checkpoint (revision 7ae557604adf67be50417f59c2c2f167def9a775) was served from a hash-verified local cache; 267 source files and the actual imported builder/connector/engine paths were checked.

Environment: Python 3.12, Ray nightly core dede511b61fbf6383f5b03cdcc1fa2c0128efafa with this PR's llm and serve/llm linked using the documented setup-dev.py method, SGLang 0.5.19, Torch 2.13.0+cu130, Transformers 5.12.1, NIXL/nixl-cu13 1.3.2. This is not a full Ray C++ build. I used the release CI CUDA-visibility/UCX variables; because the validation container has no external network, the supported SGLANG_HOST_IP=127.0.0.1 override was needed. An earlier setup attempt failed automatic IP detection, and a prior attempt was deliberately interrupted to correct launcher configuration; both logs are retained and neither is being reported as an upstream defect.

The successful container exited normally and released both GPUs. Command: python -m pytest release/llm_tests/serve/test_llm_serve_sglang_pd.py -vv -s --tb=short --timeout=600 --maxfail=1 --junitxml=/results/gpu.xml (only MODEL_ID redirected to the unchanged local checkpoint). Results archive SHA256: 13f8b2169d3bed95e5289e51956f9d7021d74f09afb78fb2037f711c0733f150. This adds original-test single-node NIXL smoke evidence, not two-node/RDMA, exact monolithic-vs-P/D output parity, or performance qualification. Integration agreement and the human-review requirements above remain unchanged.

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

cursor

Stale Bugbot comment from a previous run.

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

jeffreywang88 · 2026-09-25T20:30:03Z

FYI @xyuzh

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

cursor

Stale Bugbot comment from a previous run.

cursor · 2026-09-26T18:24:54Z

        )


+# this is where routing happens from


Scratch note committed in proxy

Low Severity

A personal navigation note was left on ProxyActor in core Serve. It is unrelated to PD disaggregation and does not document behavior.

^{Reviewed by Cursor Bugbot for commit b201b25. Configure here.}

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{Reviewed by Cursor Bugbot for commit f4207a4. Configure here.}

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

limarkdcunha changed the title ~~[Serve] POC ready for PD dissaggregation~~ [Serve] [SGLang] POC PD disaggregation May 30, 2026

gemini-code-assist Bot reviewed May 30, 2026

View reviewed changes

jeffreywang88 reviewed Jun 24, 2026

View reviewed changes

limarkdcunha marked this pull request as ready for review July 2, 2026 22:41

limarkdcunha requested review from a team, MengjinYan, SongGuyang, alimaazamat, andrewsykim, dayshah, edoakes, kfstorm, marosset, matthewdeng, pcmoritz, raulchen, richardliaw, ryanaoleary and thomasdesr as code owners July 2, 2026 22:41

test: Added Ray Serve SGLang PD disaggregation support

1205202

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

cursor Bot reviewed Aug 9, 2026

View reviewed changes

Comment thread python/ray/llm/_internal/serve/serving_patterns/prefill_decode/pd_server.py Outdated

limarkdcunha added 3 commits August 9, 2026 12:13

tracer code fix

ee6c8fc

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

setting stream_batching_interval_ms to 0

feee4e7

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

testing skip slot reservation for thesis testing

86a6633

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

cursor Bot reviewed Aug 9, 2026

View reviewed changes

Comment thread python/ray/serve/_private/router.py Outdated

limarkdcunha added 3 commits August 9, 2026 16:48

fix in router as per feedback

ea41f52

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

forward thread caps to pd replicas always

b794ee4

Undoing experimental changes

765a056

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

cursor Bot reviewed Aug 9, 2026

View reviewed changes

Comment thread bench/launch_ray_pd.py Outdated

limarkdcunha added 2 commits August 9, 2026 18:14

reverted legit change

ce0eaf5

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

added trace to measure one specific flow

f3b030e

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

cursor Bot reviewed Aug 10, 2026

View reviewed changes

Comment thread python/ray/serve/_private/request_router/request_router.py

resolved as per feedback

d8012a5

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

limarkdcunha changed the title ~~[Serve] [SGLang] [Draft] POC PD disaggregation~~ [Serve] [SGLang] [POC] PD disaggregation Aug 11, 2026

log queue wait and probe time as aggregates

e467352

cursor Bot reviewed Aug 15, 2026

View reviewed changes

0z5a mentioned this pull request Sep 7, 2026

[Serve LLM] Fix SGLang sleep state after empty-tag wakeup #65969

Open

limarkdcunha added 2 commits September 10, 2026 23:44

Added more logging steps

72f085e

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

Some more logging

c1f01a3

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

cursor Bot reviewed Sep 11, 2026

View reviewed changes

Comment thread python/ray/serve/_private/router.py Outdated

fix to last logging

ff404ac

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

Buffer PD trace logging off hot path; fix sweep harness

b201b25

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

cursor Bot reviewed Sep 26, 2026

View reviewed changes

limarkdcunha added 2 commits September 26, 2026 14:30

Write PD trace rows direct to file; drain on start and exit

0048e22

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

Report server token counts and support ignore_eos in PD benchmarks

f4207a4

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

cursor Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread bench/sweep.py Outdated

Count output chunks on both arms, not server token counts

94f1194

Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>

limarkdcunha changed the title ~~[Serve] [SGLang] [POC] PD disaggregation~~ [Serve] [SGLang] PD disaggregation Sep 27, 2026

		# LLMServer.get_deployment_options calls get_engine_config(), which
		# unconditionally imports vLLM. The SGLang byod image uninstalls vLLM,

		pass


		@PublicAPI(stability="beta")

Conversation

limarkdcunha commented May 30, 2026

Description

Related issues - #62792 #63257

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jeffreywang88 commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

limarkdcunha commented Jun 4, 2026

Uh oh!

jeffreywang88 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 15, 2026

Choose a reason for hiding this comment

Accidental personal notes in router

Uh oh!

0z5a commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jeffreywang88 commented Sep 25, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Sep 26, 2026

Choose a reason for hiding this comment

Scratch note committed in proxy

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

jeffreywang88 commented Jun 2, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

0z5a commented Sep 6, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading