CI/CD - Fix Image build for cuda11.1.1 and py unit test pipelines #771

polarG · 2026-01-06T22:55:48Z

Description

When building the CUDA 11.1.1 image, pip (Python 3.8) cannot find a pre-built wheel for the latest wandb release (v0.23.1). As a result, pip attempts to build wandb from source. However, the build fails because the image does not have Go installed, which is required for building wandb from source. Then the error appears.
Deprecated API arguments, 'interpolation', when numpy >v1.22.0. And it was removed since v2.4.0.

Solution

For the CUDA 11.1.1 build, install the required build tools (e.g., Go, Rust, and Cargo) needed for wandb.
Update the NumPy API usage to support NumPy 1.22.0+ while maintaining backward compatibility with earlier versions.

codecov · 2026-01-06T23:42:26Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.70%. Comparing base (c99380b) to head (8e8d286).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #771   +/-   ##
=======================================
  Coverage   85.69%   85.70%           
=======================================
  Files         102      102           
  Lines        7699     7703    +4     
=======================================
+ Hits         6598     6602    +4     
  Misses       1101     1101

Flag	Coverage Δ
cpu-python3.10-unit-test	`70.96% <60.00%> (-0.02%)`	⬇️
cpu-python3.12-unit-test	`70.96% <60.00%> (+0.01%)`	⬆️
cpu-python3.7-unit-test	`70.43% <100.00%> (+0.01%)`	⬆️
cuda-unit-test	`83.59% <60.00%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

guoshzhao · 2026-01-09T20:08:37Z

superbench/benchmarks/base.py

-                    np.percentile(result, float(percentile), interpolation='nearest'), reduce_type
-                )
+                try:
+                    val = np.percentile(result, float(percentile), method='nearest')


Why do we need update this? Have you met some failure?

Yes, I met it after the changes for wandb. In PR, this also appeared.

'interpolation' was deprecated in v1.22.0 and removed on 2.4.0. In the python3.12 unit test pipeline, v2.4.1 was installed.

abuccts

pip (Python 3.8) cannot find a pre-built wheel for the latest wandb release (v0.23.1)

According to wandb v0.23.1, it's compatible with py3.8. Maybe you need to upgrade pip/wheel?

However, the build fails because the image does not have Go installed, which is required for building wandb from source.

Why not just add an apt-get install -y golang?

polarG · 2026-01-12T22:27:18Z

pip (Python 3.8) cannot find a pre-built wheel for the latest wandb release (v0.23.1)

According to wandb v0.23.1, it's compatible with py3.8. Maybe you need to upgrade pip/wheel?

However, the build fails because the image does not have Go installed, which is required for building wandb from source.

Why not just add an apt-get install -y golang?

Added build-essential packages to the CUDA 11.1.1 Dockerfile to enable building wandb.

This reverts commit 059daa9. modified: .github/workflows/build-image.yml new file: dockerfile/cuda11.1.1.dockerfile

This reverts commit 4f23173. modified: dockerfile/cuda11.1.1.dockerfile

…using version control.

abuccts · 2026-01-14T23:27:15Z

dockerfile/cuda11.1.1.dockerfile

+RUN add-apt-repository -y ppa:longsleep/golang-backports && \
+    apt-get update && \
+    apt-get install -y golang-1.24-go


merge it with existing apt-get command

The 'add-apt-repository' is installed with 'software-properties-common' in the first RUN. So, we cannot combine it with the first apt-get.

abuccts · 2026-01-14T23:28:08Z

dockerfile/cuda11.1.1.dockerfile

+    rustup update stable && \
+    rustup default stable
+
+ENV PATH="/usr/lib/go-1.24/bin:/root/.cargo/bin:${PATH}"


merge with existing env command

The previous environment lacks Go and Cargo installations, and the subsequent environment is configured specifically for Docker workflows. Given this separation of concerns, I believe we should maintain the current configuration. Please let me know if you'd like any modifications.

dockerfile/cuda11.1.1.dockerfile

abuccts · 2026-01-14T23:30:14Z

superbench/benchmarks/base.py

+                try:
+                    val = np.percentile(result, float(percentile), method='nearest')


can you open another pr for this?

If we move this to another PR, the python-unit-test will fail in this PR.

Copilot

Pull request overview

This PR fixes CI/CD issues affecting the CUDA 11.1.1 Docker image build and Python unit test pipelines by addressing two main problems: missing build dependencies for wandb installation and deprecated NumPy API usage.

Changes:

Added Go and Rust build tools to the CUDA 11.1.1 Dockerfile to support wandb v0.23.1 source builds
Updated NumPy percentile API usage to support NumPy 2.x while maintaining backward compatibility with 1.x

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
superbench/benchmarks/base.py	Updated `np.percentile` API to use try-except for backward compatibility, preferring `method='nearest'` for NumPy 2.x and falling back to `interpolation='nearest'` for older versions
dockerfile/cuda11.1.1.dockerfile	Added Go 1.24, Rust, and Cargo installations with required dependencies to enable wandb source builds

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-14T23:34:02Z

superbench/benchmarks/base.py

+                try:
+                    val = np.percentile(result, float(percentile), method='nearest')
+                except TypeError:
+                    val = np.percentile(result, float(percentile), interpolation='nearest')


The try-except approach is too broad for version detection. Since TypeError could occur for other reasons, consider checking NumPy version explicitly at module level or using a more specific exception check. For example, use hasattr(np.percentile, '__wrapped__') or check np.__version__ to determine which parameter to use.

polarG · 2026-01-14T23:57:37Z

pip (Python 3.8) cannot find a pre-built wheel for the latest wandb release (v0.23.1)

According to wandb v0.23.1, it's compatible with py3.8. Maybe you need to upgrade pip/wheel?

However, the build fails because the image does not have Go installed, which is required for building wandb from source.

Why not just add an apt-get install -y golang?

The default go version is not enough for wandb-0.23.1.

Anchor wandb version when python is 3.8.

05a89a6

polarG self-assigned this Jan 6, 2026

polarG added the enhancement New feature or request label Jan 6, 2026

polarG requested a review from a team as a code owner January 6, 2026 22:55

polarG added CI/CD Continuous integration or deployment dependencies Pull requests that update a dependency file labels Jan 6, 2026

Hongtao Zhang added 3 commits January 7, 2026 01:57

Adopt new api.

af676fc

Fix lint issues.

5a4d01e

Fix lint issues.

22fd5f5

polarG changed the title ~~CI/CD - Fix Image build for cuda11.1.1~~ CI/CD - Fix Image build for cuda11.1.1 and update numpy API Jan 7, 2026

polarG requested review from abuccts and guoshzhao January 7, 2026 18:33

guoshzhao reviewed Jan 9, 2026

View reviewed changes

abuccts reviewed Jan 9, 2026

View reviewed changes

Hongtao Zhang added 2 commits January 12, 2026 21:25

Revert wandb changes.

0080d92

Remove cuda11.1.1.

059daa9

polarG changed the title ~~CI/CD - Fix Image build for cuda11.1.1 and update numpy API~~ CI/CD - Remove Image build for cuda11.1.1 and update numpy API Jan 12, 2026

polarG enabled auto-merge (squash) January 12, 2026 22:25

Hongtao Zhang added 6 commits January 13, 2026 18:08

Revert "Remove cuda11.1.1."

7f6de0c

This reverts commit 059daa9. modified: .github/workflows/build-image.yml new file: dockerfile/cuda11.1.1.dockerfile

Install go in cuda11.1.1 dockerfile.

4f23173

Revert "Install go in cuda11.1.1 dockerfile."

5bf14d4

This reverts commit 4f23173. modified: dockerfile/cuda11.1.1.dockerfile

Anchor wandb to 0.22.3 when python < 3.12.

0687eec

Add wandb version control for python >= 3.12.

ae05143

Add build dependencies of wandb in 11.1.1 to build wandb, instead of …

1ecbcda

…using version control.

polarG changed the title ~~CI/CD - Remove Image build for cuda11.1.1 and update numpy API~~ CI/CD - Fix Image build for cuda11.1.1 and unit test pipelines Jan 14, 2026

polarG changed the title ~~CI/CD - Fix Image build for cuda11.1.1 and unit test pipelines~~ CI/CD - Fix Image build for cuda11.1.1 and py unit test pipelines Jan 14, 2026

abuccts reviewed Jan 14, 2026

View reviewed changes

abuccts requested a review from Copilot January 14, 2026 23:30

Copilot started reviewing on behalf of abuccts January 14, 2026 23:31 View session

Copilot AI reviewed Jan 14, 2026

View reviewed changes

Fix comments.

8e8d286

		try:
		val = np.percentile(result, float(percentile), method='nearest')

CI/CD - Fix Image build for cuda11.1.1 and py unit test pipelines #771

Are you sure you want to change the base?

CI/CD - Fix Image build for cuda11.1.1 and py unit test pipelines #771

Conversation

polarG commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

guoshzhao Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

polarG Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abuccts left a comment

Choose a reason for hiding this comment

Uh oh!

polarG commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

abuccts Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

polarG Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

abuccts Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

polarG Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

abuccts Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

polarG Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

polarG commented Jan 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

polarG commented Jan 6, 2026 •

edited

Loading

codecov bot commented Jan 6, 2026 •

edited

Loading

polarG Jan 9, 2026 •

edited

Loading

polarG commented Jan 12, 2026 •

edited

Loading