ci: diagnose hanging tests on PR #1476 (do not merge) - #1493
Closed
zengleixin-amd wants to merge 5 commits into
Closed
zengleixin-amd wants to merge 5 commits into
zengleixin-amd wants to merge 5 commits into
Conversation
Updates VLLM_VERSION default in install_baremetal.sh, plus every doc/example that names the pinned version: docs/compatibility.rst, docs/install/install.md, the four example SKILL.md recipes, assets/slurm/models.tsv, and the test_baremetal_doc_version_consistency.py comments that mirror it. VLLM_ROCM_VARIANT stays rocm723: wheels.vllm.ai only publishes a rocm723 build for 0.28.0, same as 0.27.1. Verified against the real upstream vLLM v0.28.0 tag that TraceLens' config_vllm_v0.28.0.patch applies cleanly via git apply --check, so the TraceLens vLLM profiler-config patch path is unaffected by this bump (unlike the SGLang v0.5.18 case, see docs/compatibility.rst). Also confirmed the vllm/vllm-openai-rocm:v0.28.0 Docker image exists. Ran the full provenance/manifest/server-patcher/baremetal-doc test suite (347 tests) plus the doc-consistency guard; all pass except one pre-existing unrelated failure (test_baremetal_profiler_hotfix_accepts_an_atom_only_host, reproduced unmodified on main).
Raise the bare-metal vLLM default to 0.28.0, document Ubuntu 24.04 as the recommended host OS, gate SGLang profiler hotfix to ROCm 7.2.0 only, and teach hyperloom-setup to steer older Ubuntu hosts away from vLLM 0.28.0+ installs.
Reject vLLM >= 0.28.0 installs when host glibc is below 2.39 (including check-only and dry-run), revert the SGLang hotfix gating to a follow-up PR, trim hyperloom-setup to installer-backed UX, and fix ruff format.
Branch from PR 1476 head 7f04704 and add verbose progress plus non-terminating faulthandler stack dumps after 180 seconds. Preserve every test, the original walkthrough depth, and coverage gates. Co-Authored-By: Claude <noreply@anthropic.com>
Install the graph replay probe from the already imported torch module instead of importing through its own hook. Isolate import hooks and exit callbacks in probe tests, and cover bounded repeated imports, lazy readiness, replay counting and cleanup without reducing test depth. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
Author
|
The verified root-cause fix has been cherry-picked into PR #1476 as 3975327. Only the two-file import-hook recursion/test-isolation fix was transferred; diagnostic pytest flags were not copied. Closing this diagnostic PR as requested. Validation of the fix on this branch: https://github.com/AMD-AGI/Hyperloom/actions/runs/34604481222 (all test shards and coverage passed). The original PR will run its own CI on the transferred commit. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Diagnostic reproduction branch forked directly from PR #1476 head 7f04704. This PR intentionally includes the original PR changes so the full merge test exercises that code, not just main. Do not merge this diagnostic PR independently.
Diagnostic change
Only pyproject.toml differs from the source PR head: add -vv and -o faulthandler_timeout=180 to CI pytest arguments. Dumps do not terminate tests. No test selection, 300-pass walkthrough depth, coverage gate, shard or worker count changed.
Validation
Boundaries
Original PR #1476 branch is untouched. Native faulthandler dumps Python threads, not all suspended async tasks or nested child processes, and does not cover collection/session-exit hangs.
🤖 Generated with Claude Code