Conversation
Validation updateContainer init (8GPU, no workflow-side dind): validated on run 31652689759 — Initialize containers passed on rocOps: no infra changes needed — production End-to-end build + test validation: triggered run 31652941629 ( |
|
End-to-end validation (build + gfx950.8 pytest): run 31652956563 — Cancelled duplicate continuous run 31652941629 to avoid overlapping ROCm builds. |
|
Release backport opened: #841 → |
|
Nodind manual-test label added to workflow_dispatch only. Validation dispatch: https://github.com/ROCm/jax/actions/runs/31653907730 (runner: \�md-do-linux.jax.gpu.gfx950.8-nodind). Requires rocOps app deploy first. |
d4a3ae2 to
be81179
Compare
Drop 43 of the 45 test-target exclusions in build/rocm/ci_test_targets.txt, leaving only two: - buffer_callback_test_gpu, whose py_import deps are incompatible with pre-built plugin wheels (build_jaxlib=false). - pallas:ops_test_mgpu_gpu, a multi-accelerator test that is excluded from single_gpu runs anyway. Most of the dropped exclusions traced back to a complex-GEMM gap that no longer reproduces: C64 and C128 matmul return correct results on ROCm 7.14. The three teardown-hang exclusions (cholesky_update_test, pallas:pallas_vmap_test and pallas:gpu_paged_attention_test) each passed five consecutive runs with no hang. This brings the suite to 114 single-GPU GPU targets. Validated on MI300X (gfx942); the bazel CI pool is gfx950, so CI is the real gate.
Upstream CI has stopped publishing jax/jaxlib nightly wheels to GCS (gs://jax-nightly-artifacts/latest), leaving the ROCm nightly test legs running against missing/stale base wheels. On the ROCm fork, build the base jax and jaxlib wheels on the same amd-do-linux.jax.cpu runners as the ROCm plugin/pjrt wheels and upload them to the same S3 prefix, so the ROCm test legs consume freshly built wheels instead of the broken GCS bucket. Upstream (jax-ml) behavior is unchanged. - ci/build_rocm_artifacts.sh: allow "jax"/"jaxlib" artifacts; skip auditwheel for the pure-python jax wheel. - wheel_tests_nightly_release.yml: new fork-only build-rocm-jax-jaxlib job; wire it into the ROCm test legs and select S3 as the jax/jaxlib source on the fork. - pytest_rocm.yml / bazel_rocm.yml: new optional jax-jaxlib-source input (default gcs), forwarded to the download action. - download-jax-rocm-wheels: when source is s3, fetch jax/jaxlib from the CloudFront listing alongside the ROCm wheels instead of GCS. (cherry picked from commit 8654db6)
- Label the nightly and continuous ROCm matrices `N (7.14.0)` and `N+1 (therock-latest)` so the release track is identifiable from the job name. - Report the ROCm release version rather than the wheel packaging major in the build, pytest and Bazel job names, and report the GPU count as `gfx950 <n>-GPU`. - Remove the duplicate continuous Bazel ROCm leg: `gfx950.8` ran the same `--config=single_gpu` suite as `gfx950.4` under an identical job name. - Apply the same labels to the fork-only `build-rocm-jax-jaxlib` nightly job. The Bazel job name sits inside a Presubmit Naming Check block, so the presubmit job names change from `ROCM=7` to `ROCM=7.14.0`. The internal github-rocm-presubmits and github-rocm-blocking-presubmits checks need to be updated to match. wheel_tests_rocm_release.yml keeps its `TheRock <version>` labels: it validates one specific ROCm release, so it is single-track.
Bind-mount the runner workspace into GitHub container: jobs so checkout and artifacts work on ARC runners that use hostPath _work volumes. Co-authored-by: Cursor <cursoragent@cursor.com>
48bad50 to
262203f
Compare
…port Co-authored-by: Cursor <cursoragent@cursor.com>
79d884e to
ed8289a
Compare
360638e to
e156ed6
Compare
564a88e to
a53a18e
Compare
a53a18e to
9714167
Compare
4 participants
Summary
/home/runner/_work:/home/runner/_workvolume mount to thecontainer:section inpytest_rocm.yml.volumes:block inbazel_rocm.yml.This ensures the GitHub Actions workspace is available inside ROCm container jobs.
Test plan