Conversation
…subgroups The configured timeout reaches init_process_group (the world group) and the gloo CPU groups, but the NCCL subgroups were created without it, so TP and PP silently kept PyTorch's 600 s default. On a cold boot under pipeline parallelism the first stage compiles Triton kernels for minutes while the next stage waits in its receive; the watchdog then kills the waiting rank and the boot dies at a timeout instead of an error. The subgroups now follow the configured value through a helper next to the existing CPU-timeout helper; unset keeps PyTorch's default. Signed-off-by: Peuqui <peuqui@github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
--distributed-timeout-secondsis documented as the timeout for thedistributed groups, and
init_distributed_environmentdoes hand it toinit_process_group(the world group) and to every gloo CPU group. Thedevice groups that
GroupCoordinatorcreates for TP, PP and the othersubgroups were created without it, so they silently kept PyTorch's
default of 600 s no matter what was configured.
On a cold boot under pipeline parallelism that default is too short: the
first stage compiles its Triton kernels for minutes while the next stage
already sits in its first receive. The NCCL watchdog then kills the waiting
rank and the boot dies at a timeout instead of finishing the compile
(Qwen3.8 Flash-Next, TP2 × PP2,
--distributed-timeout-seconds 3600configured and ignored).
The change is one helper next to the existing
get_cpu_distributed_timeout_or_none()indistributed/utils.py, readingparallel_config.distributed_timeout_secondsfrom the current vLLM configthe same way, and the device subgroups are created with that timeout.
GroupCoordinatoris constructed insideset_current_vllm_configon theworker (
worker_base.py,init_device), so the value is available there.Unset keeps
timeout=None, i.e. PyTorch's default, exactly as before.Test Plan
pre-commit run --files vllm/distributed/parallel_state.py vllm/distributed/utils.py tests/distributed/test_group_coordinator_timeout.pyand
pre-commit run mypy-3.10 --hook-stage manual --files <same>.tests/distributed/test_group_coordinator_timeout.py:torch.distributed.new_groupis replaced by a recorder, aGroupCoordinatoris built underset_current_vllm_config; the NCCLgroup receives
distributed_timeout_seconds, the gloo groupcpu_distributed_timeout_seconds, and both fall back toNonewith thefields unset or without a config.
torch.distributed.new_group(..., timeout=None)resolves the timeout in_new_group_with_tagvia_get_default_timeout(backend), which returnsthe constant
default_pg_nccl_timeout(600 s) and never the timeout theworld group was initialised with. A configured
--distributed-timeout-secondstherefore did not reach any subgroup.Test Result
distributed_c10d.py(_new_group_with_tag,_get_default_timeout).Not a duplicate
Checked on 2026-09-12 against
1CatAI/1Cat-vLLM:gh pr list --state open --searchfor "distributed_timeout", "NCCL timeout subgroup", "new_grouptimeout" and
gh issue list --search "timeout pipeline parallel boot"return nothing related;
gh pr diff --name-onlyover every open PR showsno PR touching
vllm/distributed/parallel_state.py.AI assistance (Claude) was used to trace the timeout path and prepare the
change; I reviewed every line and ran the tests above.
🤖 Generated with Claude Code