Skip to content

Add managed pipx application profile and exposure - #74

Merged
AviBackToBlack merged 10 commits into
mainfrom
codex/add-pipx-profile
Sep 23, 2026
Merged

AviBackToBlack merged 10 commits into
mainfrom
codex/add-pipx-profile

Conversation

@AviBackToBlack

Copy link
Copy Markdown
Owner

Summary

  • add a separate stateful pipx profile pinned to pipx==1.17.4, isolated from Python and uv tool state
  • make the pipx application store eligible for deterministic explicit and store-wide cb expose
  • normalize only known pipx-owned absolute links so the single managed state volume remains safe to back up and restore
  • document pip, pipx, uv-tool, lock, index-trust, and release-validation boundaries

Validation

  • go vet ./...
  • go test -race ./...
  • release-style Windows build with injected v0.0.0-citest version
  • Windows + Docker Desktop E2E: install/upgrade cowsay==6.1, cached offline pipx launch, explicit expose, store-wide expose, shim invocation, unexpose
  • checksummed state backup, volume removal, dry-run/apply restore, shim reconciliation, and restored app invocation

Advances the RM-26 pipx work in #2. It does not expose plain pip project environments.

@CherylSnowVeil CherylSnowVeil left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial review of the managed pipx profile (3fb7bb6). I verified the design against uv's cache layout and pipx's backend selection, and simulated the normalization script against a mock state tree.

What checks out

  • launcher-cache/archive-v0 is the right whitelist: uv's CachedEnvironment persists uvx environments into the content-addressed archive-v0 bucket (the environments-v2 entry is just a link into it), so archive-v0/<id>/bin/python is exactly len-5 as the script requires.
  • pipx auto-resolves to the uv backend (uv is on PATH), and uv venv makes bin/python the absolute link while python3/python3.13 stay relative — consistent with the ("bin", "python") suffix check. UV_LINK_MODE=copy plus UV_PYTHON_DOWNLOADS=never and PIPX_DEFAULT_PYTHON keep the link surface narrow and the target pinned.
  • Simulated normalization behaves as documented: app-store links become relative (bin/cowsay -> ../home/venvs/...), the environments-v2 indirection link is relativized in place, and interpreter links become real copies — so cb-pipx117-py313-state stays portable through the checksummed backup/restore path, and statearchive link validation is not weakened.
  • cb expose pipx/unexpose reuse the existing deterministic store machinery (/cb/pipx + bin), discovery mounts the volume read-only with --network none, and the managed-recognition tests cover the wrong-mount/wrong-bin-dir rejections.
  • The embedded script round-trips through the TOML subset parser; migration tests cover pre-RM-11 registries gaining the section.

Findings

🟡 [important] Dangling absolute symlink permanently breaks every pipx command — internal/registry/registry.go (pipxCommand, link.resolve(strict=True))

link.resolve(strict=True) raises an unhandled FileNotFoundError for any absolute symlink whose target no longer exists. Consequence: once a dangling absolute link persists in cb-pipx117-py313-state — e.g. a bin/ or man-page link left behind by an interrupted pipx uninstall, or a stale environments-v2 indirection — every subsequent successful pipx command exits non-zero with a raw traceback, and the dead link is never cleaned, so the failure is self-perpetuating until the user does manual volume surgery.

Reproduced locally: a single stale link aborts the scan mid-iteration (leaving the volume half-normalized) with an opaque FileNotFoundError rather than the deliberate RuntimeError message.

Remediation direction: wrap the resolve in try/except OSError and re-raise the same RuntimeError ("pipx produced unsupported absolute symlink"), or explicitly unlink/report dead links under the managed bin/man dirs. Failing closed is right; failing opaquely and permanently is not.

🟢 [nit] python3 resolves through PATH, where /cb/pipx/bin is first — internal/registry/registry.go (pipxCommand)

The wrapper calls bare python3 -c .... env_set puts PIPX_BIN_DIR first on PATH, so if any installed app ever exposes a python3 executable, normalization would invoke that script instead of the interpreter. Using /usr/local/bin/python3 — already the pinned PIPX_DEFAULT_PYTHON/allowed_python — removes the dependence entirely at zero cost.

💡 [suggestion] pipx run environments are not persisted

PIPX_CACHE_DIR is unset, so pipx run builds its environment under the disposable container's ~/.cache/pipx on every invocation — each run re-resolves and needs index access. That's probably an acceptable scope cut for the install/expose workflow this profile targets (pointing it into the state volume would also need another whitelist entry), but it may be worth a one-line README note so pipx run's always-online behavior isn't surprising.

🟢 [nit] Roadmap tracking docs still list RM-26 as remaining work

docs/roadmap-implementation-requirements.md (readiness table + "remaining work" section) and docs/roadmap-decisions.md (implementation queue item 1) still describe RM-26 as implementation-ready/pending. If the convention is to update the snapshot when an item ships, these should be adjusted here or in a follow-up.

💡 [suggestion] No automated coverage of the normalization script itself

The most failure-sensitive logic in the PR (the shell/Python link whitelist) is only asserted via strings.Contains. Given how tightly it is coupled to uv/pipx layout internals, a fixture-based test that executes the script against a mock tree would catch regressions if the pinned pipx/uv versions change the store shape. Understood if this is intentionally covered only by the manual Windows + Docker Desktop E2E.

Summary

Well-scoped implementation: the profile stays store-scoped (no project markers, no forced path policy), the env allowlist deliberately excludes the controlled variables, exposure is deterministic and inherits the locked profile, and the security-model/README docs accurately describe the new trust boundary (image lock vs package-index bootstrap). The important finding above is a robustness gap rather than a design problem; the rest are minor.

@AviBackToBlack

Copy link
Copy Markdown
Owner Author

Addressed the review in signed commit c0927aa:\n\n- The post-command normalizer now catches dangling/unresolvable absolute symlinks and raises the same explicit fail-closed RuntimeError with the offending link and target.\n- It invokes /usr/local/bin/python3 directly, so an installed pipx application cannot shadow the normalizer interpreter through the managed bin directory.\n- README now distinguishes the persistent uvx launcher cache from intentionally ephemeral pipx run environments and states the latter may require package-index access on every run.\n- I left RM26 unchecked in the roadmap docs intentionally: per the roadmap contract, an open PR is not completion; the item becomes complete only after this PR is merged.\n- The Go fixture now asserts the absolute interpreter and dangling-link exception contract. It remains a structural unit test because the repository's Go test runtime does not provide the profile's Python filesystem; real interpreter/link behavior remains part of Windows + Docker end-to-end acceptance rather than being claimed by this unit suite.\n\nValidation: go test -race ./..., go vet ./..., and git diff --check all pass locally.

@CherylSnowVeil CherylSnowVeil left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of c0927aa (the fix: harden pipx state normalization follow-up). I validated the round-1 findings against the current head and then re-reviewed the whole PR from scratch, including pipx 1.17.4 backend behavior from source.

Round-1 finding validation

  • 🟡 Dangling absolute symlink → unhandled FileNotFoundError — FIXED. link.resolve(strict=True) is now wrapped in try/except OSError and re-raised as the same explicit RuntimeError (internal/registry/registry.go, pipxCommand). FileNotFoundError, ELOOP, and permission errors are all OSError subclasses, and raw_target is captured before resolution so the message reports the dead target. The failure remains fail-closed and self-perpetuating until the link is removed, but it is now deliberate and diagnosable — matching the offered remediation.
  • 🟢 python3 resolved through PATH — FIXED. The normalizer invokes /usr/local/bin/python3 directly, and TestPipxProfile asserts both the absolute interpreter and the absence of a PATH-resolved python3 -c.
  • 💡 pipx run ephemeral environments — FIXED. README now states PIPX_CACHE_DIR is not on the managed volume and each pipx run may need package-index access.
  • 🟢 RM-26 still listed as pending in roadmap docs — intentionally unchanged. The author's position (an open PR is not completion; the item closes on merge) is consistent with the roadmap contract; those files are not part of this diff anyway.
  • 💡 No execution coverage of the normalization script — partially addressed. The fixture now asserts the absolute-interpreter and dangling-link contract fragments, but it remains structural (strings.Contains); the author's rationale (no Python filesystem in the Go test runtime; real behavior covered by Windows + Docker E2E) is reasonable. Carrying forward as an open suggestion only.

Fresh findings on c0927aa

🟡 [important] The pip backend permanently poisons the profile — internal/registry/registry.go (pipxCommand whitelist)

The normalizer only whitelists the uv-venv interpreter layout: absolute bin/python links at home/venvs/<app>/ and launcher-cache/archive-v0/<id>/. But pipx 1.17.4's pip backend creates venvs with stdlib python -m venv, whose absolute interpreter link is bin/python3.13 (the versioned env_exe), not bin/python, and whose shared-libs venv lives at home/shared (depth 4). I verified this against the pinned tag: backends/uv.py → needs_shared_libs() = False, backends/pip.py → True, and shared_libs.py builds paths.ctx.shared_libs = PIPX_HOME/shared via get_default_python() -m venv --clear.

Two reachable triggers:

  • pipx install pip — commands/install.py forces install_backend = backend or PIP for the pip package, a natural thing for a Python user to try.
  • pipx install --backend pip <pkg> / pipx reinstall --backend pip <pkg> — PIPX_DEFAULT_BACKEND is correctly excluded from env_names, but the --backend flag cannot be.

Consequence: pipx succeeds, then the normalizer hits home/shared/bin/python3.13 (and home/venvs/<app>/bin/python3.13) → RuntimeError → exit 1. Worse, home/shared is never removed by pipx — pipx uninstall moves the app venv to home/.trash but the shared venv persists — so every subsequent pipx command fails until manual surgery inside cb-pipx117-py313-state. This reproduces exactly the "self-perpetuating failure requiring volume surgery" property round 1 called out, reached through a legitimate pipx operation rather than an interrupted one.

Remediation direction: either extend the whitelist to cover the stdlib layout (home/venvs/*/bin/python3* and home/shared/bin/python3* — the target != allowed_python check already constrains them to the image interpreter), or document that only the auto-selected uv backend is supported and pip-backend installs will fail closed (the current RuntimeError message doesn't tell the user the backend is the cause).

🟡 [important] /cb/pipx/bin-first PATH still lets installed apps hijack the wrapper — internal/registry/registry.go (env_set, command)

The round-1 nit was fixed for the normalizer's python3, but the same reasoning was not applied to the two remaining PATH-resolved binaries: command = ["sh", "-c", ...] (resolved by Docker through the injected PATH, where /cb/pipx/bin precedes /usr/local/bin) and uvx --from pipx==1.17.4 pipx on line 1 of the script.

Concrete failure mode: a package that ships a console script named sh, uv, or uvx lands in /cb/pipx/bin on pipx install; the next pipx invocation then executes that script as the wrapper's shell/launcher. Unlike the fixed python3 case — where pipx itself still ran, so pipx uninstall could remove the offending app — shadowing sh or uvx prevents the shim from executing at all, so pipx uninstall cannot recover it in-tool; recovery requires manual volume surgery. Trigger probability is low (a colliding script name is required), but the consequence is an unrecoverable brick with planted code running in the wrapper context.

Zero-cost remediation: use "/bin/sh" as command[0], invoke /usr/local/bin/uvx in the script, and/or move /cb/pipx/bin after the system directories in env_set PATH (it only needs to be present for exposed-app subprocess resolution, not first).

🟢 [nit] Concurrent normalizers can collide on .cb-link/.cb-copy temp names

Two overlapping pipx shim invocations both run the rglob pass outside pipx's own file locks. If both find the same absolute link, the temporary.exists() check → symlink_to sequence can race: one normalizer's os.replace can hit FileExistsError/FileNotFoundError against the other's temp, aborting that command with a traceback. Transient — a retry succeeds — but a unique temp suffix (e.g. pid) or a FileExistsError retry would close it cheaply.

Summary

The round-1 fixes are correctly implemented and CI is green. The design remains sound for the default path: pinned pipx==1.17.4 via uvx, auto-selected uv backend (needs_shared_libs() = False, so no stdlib shared venv is created), deterministic exposure through the existing store machinery, read-only --network none discovery, and docs that accurately describe the image-lock vs index-trust boundary. The two 🟡 findings are edge paths rather than flaws in that core path, but both end in the same self-perpetuating-failure mode the round-1 fix was meant to eliminate; the pip-backend gap is the more reachable of the two via pipx install pip.

Copilot AI lite review requested due to automatic review settings September 22, 2026 00:28
@AviBackToBlack

Copy link
Copy Markdown
Owner Author

Addressed the complete second-round review in signed commit 7e6b207. The pip backend no longer poisons managed state: the normalizer permits only versioned interpreter links in home/venvs/*/bin and home/shared/bin, still requires their resolved target to equal /usr/local/bin/python3.13, and continues to reject every other external target. Wrapper dispatch is no longer shadowable by managed applications: the command uses /bin/sh, the launcher uses /usr/local/bin/uvx, the normalizer uses /usr/local/bin/python3, and system directories precede /cb/pipx/bin on PATH. Normalization temp names now include PID plus a random token and are cleaned in finally, avoiding concurrent .cb-link/.cb-copy collisions. README documents explicit pip-backend support. Validation passed: go test -race ./..., go vet ./..., git diff --check, and a networkless read-only smoke check proving all three absolute executables exist in the pinned image.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Devin Review

Comment thread internal/registry/registry.go Outdated
Comment thread internal/registry/registry.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate findings remain in symlink validation, failure-path normalization, and pipx volume creation.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 High severity · 1 Low severity

Open (2)
What changed in this PR

Adds an isolated, pinned pipx profile with managed state, application exposure, link normalization, tests, and documentation.

Changes:

  • Adds pipx==1.17.4 state and normalization support.
  • Extends cb expose for explicit and store-wide pipx applications.
  • Adds migration tests and security, architecture, and release documentation.
File Review findings
README.md Nit (2 votes): Documents the wrong pipx cache variable; use PIPX_VENV_CACHEDIR or describe the default location.
internal/​registry/​registry.go Critical (1 vote): Relative symlinks can bypass validation. Moderate (1 vote): Normalization is skipped after failed pipx commands.
internal/​registry/​registry_test.go No final findings.
internal/​registry/​file_test.go No final findings.
internal/​cli/​cli.go Moderate (1 vote): Exposure before the first pipx command can create an unmanaged empty volume.
internal/​cli/​cli_test.go No final findings.
docs/​security-model.md No final findings.
docs/​release-matrix.md No final findings.
docs/​architecture.md No final findings.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread internal/registry/registry.go Outdated
Comment thread README.md Outdated
Copilot AI review requested due to automatic review settings September 23, 2026 18:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate correctness issues remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 High severity

Open (1)
Resolved since last review (2)

Comment thread internal/registry/registry.go Outdated
Copilot AI review requested due to automatic review settings September 23, 2026 19:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Add real wrapper/E2E coverage and ensure link normalization runs after interruptions.

Review effort: Lite
Findings: None

Resolved since last review (3)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Add end-to-end tests for the pipx wrapper

internal/​registry/​registry.go:75

The new wrapper is only exercised by string-fragment assertions in registry_test.go; CI runs Go tests but never executes this embedded Python against the pinned image and a real pipx state volume. That leaves the actual uvx bootstrap, exit-status propagation, venv/cache symlink layouts, and fail-closed normalization untested, so a syntax or layout regression could pass all required checks while breaking every pipx command. Add an automated container fixture (or check in the claimed E2E harness) that installs/updates an app and verifies both successful and failed commands plus the normalized links.

Copilot AI review requested due to automatic review settings September 23, 2026 19:21
@AviBackToBlack

Copy link
Copy Markdown
Owner Author

Addressed the final non-inline re-review concerns in signed commit 1c5b647. The wrapper now lives in an embedded, directly executable Python source file; normalization runs from finally after success, command failure, or KeyboardInterrupt; negative signal return codes use the shell 128+signal convention. Five Python fixtures cover failed-command cleanup, interrupted cleanup with exit 130, negative signal mapping, relative escape rejection, and python3 interpreter copying. Linux CI now runs those fixtures, while the full Go race suite and vet remain green locally.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved symlink validation and discovery serialization issues remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 High severity · 1 Low severity

Open (2)

Comment thread internal/cli/cli.go
Comment thread docs/security-model.md Outdated
Copilot AI review requested due to automatic review settings September 23, 2026 19:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Address the two outstanding moderate review comments before approval.

Review effort: Lite
Findings: 1 High severity · 1 Low severity

Open (2)

Copilot AI review requested due to automatic review settings September 23, 2026 19:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Existing supported stores may remain unlabeled and later fail strict validation.

Review effort: Lite
Findings: None

Resolved since last review (2)

Copilot AI review requested due to automatic review settings September 23, 2026 23:20
@AviBackToBlack

Copy link
Copy Markdown
Owner Author

Followed up on the current-head closer-look note in signed commit dcb4412. Docker cannot retrofit labels onto an existing named volume, so established npm/Go/Cargo/uv/.NET/Ruby stores created before managed-volume labels remain usable. Newly created stores still receive labels, existing labeled stores still require exact ownership, and the new pipx store remains strict because it has no legitimate pre-label installed base. Added focused tests for legacy acceptance, pipx rejection, and exact-label acceptance; go test -race ./... and go vet ./... pass locally.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Pipx discovery must synchronize with .cb-pipx.lock to avoid scanning partially updated state.

Review effort: Lite
Findings: None

Copilot AI review requested due to automatic review settings September 23, 2026 23:39
@AviBackToBlack

Copy link
Copy Markdown
Owner Author

Addressed the current-head concurrency note in signed commit 64798af. Pipx exposure now runs an embedded scanner that opens the same .cb-pipx.lock read-only and holds a shared flock for the entire scan; pipx commands retain the exclusive lock. If the lock does not exist, discovery returns no apps instead of scanning or creating state, so the discovery volume remains read-only. The scanner also preserves the resolved-target boundary check and rejects symlinked lock files. CI now executes 11 Python fixtures, including proof that discovery blocks behind an exclusive wrapper lock. Locally: both Python suites pass, go test -race ./... passes, go vet ./... passes, and the pinned uv image passed a no-network Python/fcntl smoke check.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The only finding is a non-blocking documentation clarification, with no approval-blocking issues.

Review effort: Lite
Findings: None

@AviBackToBlack
AviBackToBlack merged commit c5f4d4a into main Sep 23, 2026
14 checks passed
@AviBackToBlack
AviBackToBlack deleted the codex/add-pipx-profile branch September 23, 2026 23:49
AviBackToBlack added a commit that referenced this pull request Sep 25, 2026
## Summary

- reconcile the roadmap status snapshot with merged PRs #74–#78
- mark RM-26 complete only because #74 is merged
- distinguish merged foundations from remaining overlay,
signed-registry, image-trust, self-update, WSL, and ARM64 qualification
work
- remove completed foundations from the recommended implementation queue
while preserving their shipped contracts
- keep unmerged PR coverage explicitly non-authoritative for completion

The authoritative issue #2 was updated in parallel with the same
merged-state facts.

## Validation

- `gofmt -l .`
- `go vet ./...`
- `go test -race ./...`
- `python -m unittest -v internal/registry/pipx_wrapper_test.py
internal/cli/pipx_discovery_test.py`
- `git diff --check`
<!-- devin-review-badge-begin -->

---

<a
href="https://app.devin.ai/review/avibacktoblack/container-bin/pull/85"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants