fix(#5742): pin podman to 5.8.4 (matching the image) with a verified static bundle - #5743
Conversation
The apt preferences pin from #5738 installs podman 4.9.3 to /usr/bin/podman, but GitHub-hosted Ubuntu 24.04 runner images ship a static podman 5.8.4 bundle under /usr/local/ that wins on PATH. The version assertion then fails before sandbox creation begins. Replace the apt-based approach with a pinned podman 4.x static bundle from mgoltzsche/podman-static (v4.9.5). The tarball extracts directly into /usr/local/, overlaying the runner image's 5.x binary. This mirrors how runner-images itself installs podman and avoids apt downgrade semantics. The bundle is self-contained (podman + crun + conmon + netavark), so the system crun version no longer matters. Closes #5742
|
🤖 Finished Review · ✅ Success · Started 10:50 PM UTC · Completed 11:03 PM UTC |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
ReviewFindingsMedium
Labels: PR modifies CI script for podman installation used in sandbox setup |
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
fullsend-ai-review
Bot
added
requires-manual-review
|
/fs-fix |
|
🤖 Finished Fix · ❌ Failure · Started 11:11 PM UTC · Completed 11:22 PM UTC |
Add supply-chain integrity check for the downloaded podman-static tarball. Pinned per-architecture SHA-256 checksums are verified after download and before root extraction. The script fails fast with an actionable error if a checksum is a placeholder or does not match. Checksums are currently placeholders — compute and hardcode the real values before merging: curl -fsSL '<url>/podman-linux-amd64.tar.gz' | sha256sum curl -fsSL '<url>/podman-linux-arm64.tar.gz' | sha256sum Addresses review feedback on #5743
|
The fix agent completed, but the post-fix script failed before finishing. Workflow run: https://github.com/fullsend-ai/.fullsend/actions/runs/30498652397 Details: |
|
Investigated this while chasing #5742 down. Two things worth fixing here, independent of each other: 1. Checksums are still placeholders
2. Even with real checksums, this will still fail — missing AppArmor profileConfirmed by testing an equivalent script end-to-end via This is a known limitation of the if [[ "$(cat /proc/sys/kernel/apparmor_restrict_unprivileged_userns 2>/dev/null)" == "1" ]]; then
sudo tee /etc/apparmor.d/podman >/dev/null <<'EOF'
abi <abi/4.0>,
include <tunables/global>
profile podman /usr/{bin,local/bin}/podman flags=(unconfined) {
userns,
include if exists <local/podman>
}
EOF
sudo apparmor_parser -r -W /etc/apparmor.d/podman
fiAdd this right after the Worth reconsidering: pin to 5.8.4 instead of downgrading to 4.9.5With both fixes above applied (real checksums + AppArmor profile), I validated podman 5.8.4 — the same version the runner image already ships — end-to-end via Pinning to 5.8.4 means matching what the image already provides (no fighting against it, no future re-downgrade needed as the ecosystem moves off 4.x) rather than deliberately regressing to an older major version whose compatibility with fullsend's sandbox is unverified beyond "the crun version matches." Both approaches need the identical install mechanism (static bundle + AppArmor profile) either way — the only difference is which Happy to push these fixes to this branch directly if useful, or hand off the exact checksums for 4.9.5 if you'd rather stay on the downgrade path. |
waynesun09
changed the title
32a08d5 to
2cce6b6
Compare
…pArmor Two problems with the previous state of this branch: 1. The SHA-256 checksums were placeholders, never computed, so the script's own safety check failed the job before podman was even installed. 2. Even with real checksums, installing the mgoltzsche/podman-static bundle directly fails rootless config with "failed to reexec: Permission denied" on Ubuntu 24.04's default hardened AppArmor policy (apparmor_restrict_unprivileged_userns=1) -- the static binary ships no AppArmor profile, unlike the distro-packaged podman. actions/runner-images' own install script -- the reference this approach is modeled on -- handles exactly this by writing a custom profile and reloading AppArmor; this script didn't. Also switch the pinned version from 4.9.5 to 5.8.4, matching what the runner image already ships, rather than downgrading. No podman-4.x- specific requirement exists in fullsend's own code (internal/sandbox only does generic 'podman ps'/'podman logs' CLI calls), and pinning to the version already present means never fighting the image's own install again. Checksums are independently verified and match the values actions/runner-images itself pins for the same tag. Only the usr/ subtree of the bundle is extracted, not etc/: the bundled config defaults are generic and Fedora-oriented (including a deprecated v1-format registries.conf), and would silently override whatever the runner image already has correctly configured for its own static bundle. That etc/ exclusion has a consequence: without the bundled containers.conf, podman falls back to the base OS default at /usr/share/containers/containers.conf, which on runner images with a distro-packaged podman points at the distro's own crun (/usr/bin/crun) -- not our freshly-extracted one. Confirmed via a real e2e run on the actual affected runner image (20260726.x): podman/conmon/netavark/ aardvark-dns/pasta all correctly resolved to the new bundle, but podman info still reported ociRuntime path /usr/bin/crun at the old distro version (1.14.1), reproducing the exact original crun error from a live dispatched sandbox creation. Fixed by forcing /usr/bin/crun to be a symlink to our verified /usr/local/bin/crun, with a fail-fast check that the symlink resolves correctly -- this sidesteps podman's config-driven runtime search order entirely rather than depending on it. Validated end-to-end via workflow_dispatch on functional-tests.yml: install succeeds, rootless config succeeds, and all 4 real triage- agent sandbox-creation eval cases pass with zero regressions. Note: an earlier green functional-tests run (before the crun-symlink fix) turned out to have landed on the older, already-unaffected runner image (20260720.247.2) and was not a real test of the fix -- the crun regression was only caught by e2e landing on the actual affected image and reproducing the original failure signature. Assisted-by: Claude (fix), Grok (review) Signed-off-by: Wayne Sun <gsun@redhat.com>
2cce6b6 to
17c8d78
Compare
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Reviewed via 3-agent review-squad, all findings addressed and verified. Most importantly, confirmed working end-to-end on the actual affected runner image (ubuntu24/20260726.254.1) via a live e2e run: real sandbox creation + real triage agent execution succeed where the original bug reproduced. See PR description for the full validation trail, including a false-positive validation that was caught and corrected rather than left in place.
|
🤖 Finished Retro · ✅ Success · Started 2:22 AM UTC · Completed 2:38 AM UTC |
Retro: PR #5743 — pin podman to 5.8.4 with a verified static bundleTimeline
What went well
What went wrong
Existing issues with new evidence
Autonomy assessmentThe human reviewer added substantial value that the review agent could not: platform-specific operational knowledge (AppArmor, crun resolution), strategic version choice, and hands-on e2e verification on the actual affected runner image. The No new proposalsAll identified improvement opportunities are already tracked by existing open issues (listed above). The fix agent post-script failure is the most impactful blocking issue and is tracked at |
2 participants
Summary
The runner image's static podman bundle at
/usr/local/bin/wins onPATHover anything apt installs at/usr/bin/, so the apt-based 4.x pin from #5738 never actually took effect —podman --versionkept reporting 5.8.4 regardless. This replaces the apt pin with a static bundle install from mgoltzsche/podman-static (the same upstream sourceactions/runner-imagesitself uses for this exact bundle), pinned to 5.8.4 — matching what the image already ships — instead of downgrading to 4.x.Related Issue
Fixes #5742 (follow-up to #5733 / #5738)
Changes
.github/scripts/install-podman.sh: downloads the pinnedmgoltzsche/podman-staticv5.8.4 tarball, verifies it against a real (independently computed) SHA-256 checksum per architecture, and extracts only theusr/subtree over/usr/local/(notetc/, which ships generic Fedora-oriented config defaults that would silently override the runner image's own correctly-configured settings)./usr/bin/crunto symlink to the freshly-extracted/usr/local/bin/crun, with a fail-fast check. Without the bundledcontainers.conf(intentionally excluded above), podman falls back to the base OS default config, which on these runner images points at the distro's own (incompatible) crun — confirmed via a live e2e run on the actual affected runner image: podman/conmon/netavark/aardvark-dns/pasta all correctly resolved to the new bundle, butpodman infostill reported the old distro crun, reproducing the original crun error end-to-end. The symlink sidesteps podman's config-driven runtime search order entirely.actions/runner-images' own install script applies for this exact bundle: on Ubuntu ≥ 23.10 withapparmor_restrict_unprivileged_userns=1(24.04's default), the static binary has no AppArmor profile (unlike distro-packaged podman), so rootless podman fails withfailed to reexec: Permission deniedwithout it.Why 5.8.4 instead of downgrading to 4.x
No podman-4.x-specific requirement exists in fullsend's own code —
internal/sandbox/sandbox.goonly shells out to genericpodman ps/podman logs. Pinning to the version the image already ships means never fighting the image's own install again, versus deliberately regressing to an older major version.Testing
make lintpasses (shellcheck, pinact, GitHub Actions workflow linter)ubuntu24/20260726.254.1, the same build that originally reproduced the crun bug) via a realworkflow_dispatchof the full e2e suite:admin install→ real triage workflow dispatch → sandbox creation → real triage agent run → comment posted → labels applied, allstatus=completed conclusion=success. This is the same path that previously reproducedcrun: unknown version specifiedbefore the crun-symlink fix was added.functional-tests.ymlworkflow_dispatch: install succeeds, rootless config succeeds, 4/4 real triage-agent sandbox-creation eval cases pass with zero regressions (note: an earlier pass of this same check had landed on the older, already-unaffected runner image and wasn't a real test of the fix — only the e2e run above, on the confirmed-affected image, is decisive).Checklist
fix(#5742))