Skip to content

ci: install python3-pip before awscurl/tox on minimal runners #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andybrown668 wants to merge 4 commits into
base: main
Choose a base branch
from
Open

ci: install python3-pip before awscurl/tox on minimal runners #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c94e485
ci: install python3-pip before awscurl/tox on minimal runners
andybrown668 Apr 9, 2026
3dfb87e
fix(ci): pin s3s-e2e to revision that builds with cargo install
andybrown668 Apr 9, 2026
889e079
ci: use pip --break-system-packages for awscurl/tox on PEP 668
andybrown668 Apr 14, 2026
4a6b21a
ci: install cmake toolchain for s3s-e2e on e2e runners
andybrown668 Apr 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,12 +195,22 @@ jobs:
- name: Setup Rust toolchain for s3s-e2e installation
uses: dtolnay/rust-toolchain@stable

# s3s-e2e depends on aws-lc-sys (CMake) and other native crates; minimal runners may omit these.
- name: Install build dependencies for s3s-e2e
run: |
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
build-essential \
Comment on lines +200 to +203
cmake \
pkg-config \
libssl-dev

- name: Install s3s-e2e test tool
uses: taiki-e/cache-cargo-install-action@v2
with:
tool: s3s-e2e
git: https://github.com/s3s-project/s3s.git
rev: 4a04a670cf41274d9be9ab65dc36f4aa3f92fbad
rev: da276469212a6d57ac5a6bf831e6af41e0646785

- name: Run end-to-end tests
run: |
Expand Down Expand Up @@ -234,6 +244,17 @@ jobs:
- name: Make binary executable
run: chmod +x ./target/debug/rustfs

# Same as e2e-s3tests.yml: provision awscurl (admin API SigV4) and tox before run.sh.
# Minimal images may ship python3 without pip (No module named pip).
# Ubuntu/Debian PEP 668 (externally-managed-environment) rejects plain --user installs; use
# --break-system-packages for CI-only user-site installs (same fallback as scripts/s3-tests/run.sh).
- name: Install Python tools
run: |
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python3-pip
python3 -m pip install --user --break-system-packages --upgrade pip awscurl tox
Comment on lines +249 to +255
echo "$HOME/.local/bin" >> "$GITHUB_PATH"

- name: Run implemented s3-tests
run: |
DEPLOY_MODE=binary \
Expand Down
8 changes: 6 additions & 2 deletions .github/workflows/e2e-s3tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,9 @@ jobs:

- name: Install Python tools
run: |
python3 -m pip install --user --upgrade pip awscurl tox
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python3-pip
python3 -m pip install --user --break-system-packages --upgrade pip awscurl tox
echo "$HOME/.local/bin" >> "$GITHUB_PATH"

- name: Enable buildx
Expand Down Expand Up @@ -240,7 +242,9 @@ jobs:

- name: Install Python tools
run: |
python3 -m pip install --user --upgrade pip awscurl tox
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python3-pip
python3 -m pip install --user --break-system-packages --upgrade pip awscurl tox
echo "$HOME/.local/bin" >> "$GITHUB_PATH"

- name: Enable buildx
Expand Down
26 changes: 26 additions & 0 deletions scripts/s3-tests/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -723,6 +723,32 @@ install_python_package() {
local package=$1
local error_output

# Debian/Ubuntu minimal images often have python3 but no pip (No module named pip).
if ! python3 -m pip --version >/dev/null 2>&1; then
if command -v apt-get >/dev/null 2>&1; then
Comment on lines +726 to +728
Copy link

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The python3 -m pip --version check will also fail when python3 itself is missing, but the warning/error messages assume Python is present (“python3 has no pip”). Consider checking command -v python3 first (or distinguish exit codes) so the log guidance is accurate (e.g., install python3 vs install python3-pip).

Suggested change
# Debian/Ubuntu minimal images often have python3 but no pip (No module named pip).
if ! python3 -m pip --version >/dev/null 2>&1; then
if command -v apt-get >/dev/null 2>&1; then
# Distinguish between missing python3 and missing pip so remediation is accurate.
if ! command -v python3 >/dev/null 2>&1; then
if command -v apt-get >/dev/null 2>&1; then
log_warn "python3 is missing; installing python3 and python3-pip via apt..."
if [ "$(id -u)" -eq 0 ]; then
apt-get update -qq
DEBIAN_FRONTEND=noninteractive apt-get install -y python3 python3-pip || {
log_error "Failed to install python3 and python3-pip via apt-get"
return 1
}
elif command -v sudo >/dev/null 2>&1; then
sudo apt-get update -qq
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python3 python3-pip || {
log_error "Failed to install python3 and python3-pip via apt-get"
return 1
}
else
log_error "python3 is missing. Install python3 and python3-pip, then retry."
return 1
fi
else
log_error "python3 is missing. Install python3 and pip (e.g. python3-pip on Debian/Ubuntu), then retry."
return 1
fi
elif ! python3 -m pip --version >/dev/null 2>&1; then
if command -v apt-get >/dev/null 2>&1; then

Copilot uses AI. Check for mistakes.
log_warn "python3 has no pip; installing python3-pip via apt..."
if [ "$(id -u)" -eq 0 ]; then
apt-get update -qq
DEBIAN_FRONTEND=noninteractive apt-get install -y python3-pip || {
log_error "Failed to install python3-pip via apt-get"
return 1
}
elif command -v sudo >/dev/null 2>&1; then
sudo apt-get update -qq
Comment on lines +731 to +737
Copy link

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

apt-get update -qq is executed under set -e, so if it fails the whole script will exit without emitting a log_error, which makes failures harder to diagnose. Consider wrapping apt-get update with an explicit || { log_error ...; return 1; } (or similar) so update failures produce a clear message like the install failure path does.

Suggested change
apt-get update -qq
DEBIAN_FRONTEND=noninteractive apt-get install -y python3-pip || {
log_error "Failed to install python3-pip via apt-get"
return 1
}
elif command -v sudo >/dev/null 2>&1; then
sudo apt-get update -qq
apt-get update -qq || {
log_error "Failed to update apt package index"
return 1
}
DEBIAN_FRONTEND=noninteractive apt-get install -y python3-pip || {
log_error "Failed to install python3-pip via apt-get"
return 1
}
elif command -v sudo >/dev/null 2>&1; then
sudo apt-get update -qq || {
log_error "Failed to update apt package index"
return 1
}

Copilot uses AI. Check for mistakes.
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python3-pip || {
log_error "Failed to install python3-pip via apt-get"
return 1
}
else
log_error "python3 has no pip. Install python3-pip (or ensure pip) and retry."
return 1
fi
else
log_error "python3 has no pip. Install pip (e.g. python3-pip on Debian/Ubuntu) and retry."
return 1
fi
fi

# Try --user first (works on most Linux systems)
error_output=$(python3 -m pip install --user --upgrade pip "${package}" 2>&1)
if [ $? -eq 0 ]; then
Expand Down
Loading