Skip to content

[Bugfix][Distributed] Pass --distributed-timeout-seconds to the NCCL subgroups - #619

Open
Peuqui wants to merge 1 commit into
1CatAI:mainfrom
Peuqui:nccl-subgroup-timeout
Open

Peuqui wants to merge 1 commit into
1CatAI:mainfrom
Peuqui:nccl-subgroup-timeout

Conversation

@Peuqui

@Peuqui Peuqui commented Sep 13, 2026

Copy link
Copy Markdown

Purpose

--distributed-timeout-seconds is documented as the timeout for the
distributed groups, and init_distributed_environment does hand it to
init_process_group (the world group) and to every gloo CPU group. The
device groups that GroupCoordinator creates for TP, PP and the other
subgroups were created without it, so they silently kept PyTorch's
default of 600 s no matter what was configured.

On a cold boot under pipeline parallelism that default is too short: the
first stage compiles its Triton kernels for minutes while the next stage
already sits in its first receive. The NCCL watchdog then kills the waiting
rank and the boot dies at a timeout instead of finishing the compile
(Qwen3.8 Flash-Next, TP2 × PP2, --distributed-timeout-seconds 3600
configured and ignored).

The change is one helper next to the existing
get_cpu_distributed_timeout_or_none() in distributed/utils.py, reading
parallel_config.distributed_timeout_seconds from the current vLLM config
the same way, and the device subgroups are created with that timeout.
GroupCoordinator is constructed inside set_current_vllm_config on the
worker (worker_base.py, init_device), so the value is available there.
Unset keeps timeout=None, i.e. PyTorch's default, exactly as before.

Test Plan

  1. pre-commit run --files vllm/distributed/parallel_state.py vllm/distributed/utils.py tests/distributed/test_group_coordinator_timeout.py
    and pre-commit run mypy-3.10 --hook-stage manual --files <same>.
  2. New CPU-only tests/distributed/test_group_coordinator_timeout.py:
    torch.distributed.new_group is replaced by a recorder, a
    GroupCoordinator is built under set_current_vllm_config; the NCCL
    group receives distributed_timeout_seconds, the gloo group
    cpu_distributed_timeout_seconds, and both fall back to None with the
    fields unset or without a config.
  3. PyTorch behaviour, checked in the installed torch 2.10.0:
    torch.distributed.new_group(..., timeout=None) resolves the timeout in
    _new_group_with_tag via _get_default_timeout(backend), which returns
    the constant default_pg_nccl_timeout (600 s) and never the timeout the
    world group was initialised with. A configured
    --distributed-timeout-seconds therefore did not reach any subgroup.

Test Result

  1. All applicable hooks passed; mypy-3.10 passed.
  2. 3 passed.
  3. Confirmed by reading distributed_c10d.py (_new_group_with_tag,
    _get_default_timeout).

Not a duplicate

Checked on 2026-09-12 against 1CatAI/1Cat-vLLM: gh pr list --state open --search for "distributed_timeout", "NCCL timeout subgroup", "new_group
timeout" and gh issue list --search "timeout pipeline parallel boot"
return nothing related; gh pr diff --name-only over every open PR shows
no PR touching vllm/distributed/parallel_state.py.

AI assistance (Claude) was used to trace the timeout path and prepare the
change; I reviewed every line and ran the tests above.

🤖 Generated with Claude Code

…subgroups

The configured timeout reaches init_process_group (the world group) and
the gloo CPU groups, but the NCCL subgroups were created without it, so
TP and PP silently kept PyTorch's 600 s default. On a cold boot under
pipeline parallelism the first stage compiles Triton kernels for minutes
while the next stage waits in its receive; the watchdog then kills the
waiting rank and the boot dies at a timeout instead of an error. The
subgroups now follow the configured value through a helper next to the
existing CPU-timeout helper; unset keeps PyTorch's default.

Signed-off-by: Peuqui <peuqui@github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant