[image-spec]: push nix images from remote builders

[jld-adriano]

Tracking issue

Related to exa-labs/monorepo#29166

Why are the changes needed?

ImageSpec(nix=True) can use remote Nix builders for derivation builds, but the final nix2container push previously ran from the local orchestrator. Cold image builds then copied the layer closure back to the local runner before uploading to ECR.

What changes were proposed in this pull request?

• Discover configured SSH Nix builders from NIX_CONFIG, nix.conf machine-file references, or Flytekit-specific overrides.
• For pushed Nix images, select a builder matching the target Nix system, build packages.$system.push-to-ecr into that remote store with --store ssh-ng://... --builders "", then SSH to the same builder and run the push there.
• Support Nix machine entries whose system column lists multiple comma-separated systems.
• Remap embedded ssh-key= values to local key paths before invoking nix build --store.
• Keep the existing local docker.copyTo fallback when no matching remote builder is configured or FLYTEKIT_NIX_REMOTE_PUSH=0 is set.
• Redact ECR credentials in logs.

How was this patch tested?

uv run --frozen --with ruff ruff check --fix flytekit/image_spec/default_builder.py tests/flytekit/unit/core/image_spec/test_default_builder.py
uv run --frozen --with pytest --with hypothesis pytest tests/flytekit/unit/core/image_spec/test_default_builder.py -q -k 'nix_remote or parse_nix or configured_nix or store_uri or remote_nix or select_nix'

Setup process

No extra setup.

Screenshots

N/A

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

• exa-labs/monorepo#29166

Docs link

N/A

Link to Devin session: https://app.devin.ai/sessions/1fe549f1391845c0b09626d9be9223be
Requested by: @jld-adriano

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[devin-ai-integration]

Tested the remote Nix push flow end-to-end from shell against the monorepo integration path. Full report: https://app.devin.ai/attachments/61e7f83b-021e-403d-a67a-266642815b9a/test-report.md

Escalation: cleanup of temporary ECR tag devin-remote-push-1777614783 is still blocked by missing ecr:BatchDeleteImage; scoped permission request ecr-delete-devin-test-image is pending. This does not affect build/push proof.

Remote builder behavior

• Passed: Flytekit targeted tests: 32 passed, 1 skipped.
• Passed: Flytekit ruff: All checks passed!.
• Passed: builder harness selected _NixRemoteBuilder(... ssh_host='root@54.214.44.122' ... ssh_key='/home/ubuntu/nix-runner-key').
• Passed: builder harness made remote_calls 1 and local_run_calls 0.
• Passed: remote build used --eval-store auto --store ssh-ng://root@54.214.44.122?ssh-key=/home/ubuntu/nix-runner-key --builders "" --builders-use-substitutes and returned /nix/store/...-push-to-ecr.
• Passed: SSH push ran on hostname=nix-runner-x86-1 and used /nix/store/...-copy-to.
• Passed: ECR returned digest sha256:a8145f005f9072961e85c9eaf5a6a5cc8d3a5db42962c628a09c120df300eab2, size 17117428, for tag devin-remote-push-1777614783.

Integration checks

• Passed: exa_ml Frodo/checkpoint targeted tests: 7 passed.
• Passed: exa_ml ruff: All checks passed!.
• Passed: affected monorepo pins/locks reference Flytekit bd1193acf821ef60546215392ac82abe80ef7b97; stale intermediate SHAs absent.
• Current checks snapshot: monorepo 80 passed / 0 failed / 12 pending; flytekit 1 passed / 0 failed / 0 pending.

Notes: shell-only testing, so no recording. Session: https://app.devin.ai/sessions/1fe549f1391845c0b09626d9be9223be

[devin-ai-integration]

Kitchen Sink copy of the test report: https://kitchen-sink.exa.ai/content-host/view/remote-nix-push-test-report-b367ac01