fix: authenticate private base images before remote build

[johnyaku]

Problem

A remote build using the new --base flag with a private GHCR base image fails at the FROM step:

#2 ERROR: failed to authorize: failed to fetch anonymous token: ... 401 Unauthorized
 > [internal] load metadata for ghcr.io/swarbricklab/gpu-base:20260616:
Dockerfile:3
   3 | >>> FROM ghcr.io/swarbricklab/gpu-base:20260616

Reproduced with:

absconda publish --file .../flash_scope.yaml --base ghcr.io/swarbricklab/gpu-base:20260616

Root cause

The remote builder ran docker login only after docker build, and only on the push path (remote.py). The base image is pulled during docker build (the FROM step), so a private base was pulled anonymously and the registry returned 401/403.

The builder VM's boot-time docker login (in its startup script) is the only thing that could have authenticated the pull, but it races with absconda's fixed sleep(30) start wait — the observed run started the build ~32s after boot, before the startup script finished logging in. So the pull was effectively unauthenticated.

This only surfaced now because --base is the first feature to pull a private base image; normal builds use public conda base images.

Fix

Authenticate the base image's registry before the build when it's on GHCR (the registry the configured Secret Manager credentials serve), reusing the existing credentials. The push-target login is folded into the same pre-build step and de-duplicated (base and push are usually the same registry). Credentials are logged out at the end as before.

Generated remote command now:

docker login ghcr.io  &&  docker build ...  &&  docker push ...  &&  docker logout ghcr.io

Images on other/public registries (a public nvidia/cuda base, registry-less refs like python:3.11-slim) are left untouched via _ghcr_registry(), so builds that need no auth are unaffected. This makes the fix independent of the VM's boot-state race entirely.

Changes

• remote.py: thread base_image through build_remote_image → execute_build → _run_build; authenticate GHCR before build; add _ghcr_registry() / _docker_login_command() helpers; broaden the auth-failure hint to mention read:packages (pull) as well as write:packages (push).
• cli.py: pass base_override into the remote build and record it in the manifest.
• tests/test_remote.py: assert login precedes build for a private base; unit-test _ghcr_registry() matching.

Testing

• pytest: 111 passed, 2 skipped (Docker-CLI smoke tests, no Docker on host). ruff check + ruff format --check clean.
• Verified the generated remote payload orders docker login ghcr.io before docker build.
• Not yet validated end-to-end against the live gcp-builder (requires deploying this build to NCI).

Follow-up (not in this PR)

The sleep(30) start wait in start_remote_builder is still a latent race for SSH/Docker readiness; polling /var/lib/absconda/ready would be more robust. This fix makes base-image auth independent of it, so it's no longer on the critical path.

🤖 Generated with Claude Code