Skip to content

[flytekit]: Nix-based ImageSpec builder wrapping makePythonProject#56

Open
Carlos-Marques wants to merge 1 commit intomasterfrom
devin/1778281261-nix-uv-imagespec-builder
Open

[flytekit]: Nix-based ImageSpec builder wrapping makePythonProject#56
Carlos-Marques wants to merge 1 commit intomasterfrom
devin/1778281261-nix-uv-imagespec-builder

Conversation

@Carlos-Marques
Copy link
Copy Markdown

Why are the changes needed?

Building Flyte task container images currently requires Docker or depot. Projects that already use Nix and makePythonProject (via uv2nix + nix2container) have no way to leverage that infrastructure through Flyte's ImageSpec abstraction — they must maintain a parallel build path.

This PR adds a NixImageSpecBuilder so that any project with a uv.lock can build OCI images through Nix, using the same ImageSpec interface that Flyte tasks already consume.

What changes were proposed in this pull request?

New file: flytekit/image_spec/nix_builder.py

  • NixImageSpecBuilder — an ImageSpecBuilder subclass that:

    • Validates the ImageSpec only contains parameters compatible with a Nix build (rejects packages, apt_packages, conda, cuda, etc.)
    • Reuses _copy_lock_files_into_context from default_builder to handle local-package rewriting
    • If the project already has a flake.nix, uses it as-is
    • If not, generates a minimal flake.nix that wraps makePythonProject (configured via FLYTEKIT_NIX_PYTHON_FLAKE env var)
    • Calls nix build (local) or nix run … copyTo (push to ECR) with cross-compilation support
    • Supports Python 3.9–3.12 and linux/amd64 + linux/arm64
  • nix_image_spec(requirements, **kwargs) — convenience factory that returns an ImageSpec pre-configured with builder="nix" and nix=True

Modified: flytekit/image_spec/__init__.py

  • Registers the nix builder at priority 0 (lower than default's priority 1) so it's only used when explicitly requested via builder="nix"

Usage:

from flytekit.image_spec import nix_image_spec

image = nix_image_spec(
    "uv.lock",
    name="my-service",
    python_version="3.12",
    registry="472386928882.dkr.ecr.us-west-2.amazonaws.com/exa",
)

@task(container_image=image)
def my_task(): ...

How was this patch tested?

Unit tests in tests/flytekit/unit/core/image_spec/test_nix_builder.py (14 tests):

  • Validation: rejects non-uv.lock requirements, unsupported parameters, bad python versions, bad platforms
  • Build: mocked nix build for local builds, mocked nix run … copyTo for ECR push, verifies correct command construction
  • Flake generation: verifies generated flake.nix contains correct makePythonProject call, python version, env vars, entrypoint
  • Error handling: missing nix binary, build failures, missing FLYTEKIT_NIX_PYTHON_FLAKE env var
  • Factory: nix_image_spec sets correct defaults
14 passed in 0.20s

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Link to Devin session: https://app.devin.ai/sessions/ccc5697639074693932b0b7aed586a19
Requested by: @Carlos-Marques

Co-Authored-By: carlos <carlosmarques.personal@gmail.com>
@devin-ai-integration
Copy link
Copy Markdown

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@devin-ai-integration
Copy link
Copy Markdown

Superseding this with an Exa-specific implementation in python/shared/exa_flyte, since the builder depends on Exa's makePythonProject, ECR conventions, and monorepo flake layout. I validated the moved builder with a Flyte smoke workflow on Hephaestus (devin-kk1uwul02n, nix-image-builder-smoke:42).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant