feat: cross compile - #1434
Draft
xangcastle wants to merge 130 commits into
Draft
feat: cross compile#1434xangcastle wants to merge 130 commits into
xangcastle wants to merge 130 commits into
Conversation
…s package do not have py3-none-any
(cherry picked from commit 4869b33)
pycross-setuptools' native_wheels matrix duplicates e2e/crossbuild/zstandard (same package, and crossbuild goes further: QEMU execution plus a byte-for-byte diff against the official wheel). Its marginal value — the cross path under e2e/cases' own CC toolchain and shared-hub setup — lands as its own small PR after this one. Recoverable from commit 2273078.
geohash was the workspace's first case and kept its pyproject/uv.lock at the root under a bare pypi_crossbuild hub while every later case adopted the one-dir-one-hub pattern. Move the pair into geohash/, rename the hub to pypi_crossbuild_geohash, and align the project name and dep_group with the crossbuild_<pkg> convention.
Brings xangcastle/pep517-rust-toolchain's current state onto the spike: rust_toolchain as a label down to sdist_build, inferred setuptools-rust detection, RUST_SYSROOT derived and passed to rustc explicitly, the cargo cross env keyed off the wired toolchain, and the pycross-rust-rs case on rules_rs 0.0.110. The helper's own setuptools-rust sniffing and its tests go: Starlark decides, the helper obeys.
…line Cargo fetched its dependencies inside the build action, with whatever network the sandbox allowed: not hermetic, not reproducible beyond the lock the sdist happened to ship, and dead under remote execution. The C-family backends never had this problem: their build deps come from the uv lock and their toolchains are action inputs. sdist_configure now reads the sdist's Cargo.lock (anywhere in the tree, shallowest first: setuptools-rust keeps its crate under src/) and reports the crates it pins. For a Rust build, sdist_build downloads each crates.io crate through Bazel's downloader with the lock's checksum into a cargo vendor directory, and the generated pep517_native_whl passes it to the helper, which writes the source replacement into the sandbox CARGO_HOME and sets CARGO_NET_OFFLINE. Locks pinning git or path sources are rejected outright. Sdists without a Cargo.lock (tiktoken) build online with a warning, or take one from uv.override_package(cargo_lock = ...): vendored the same way and placed next to the crate's Cargo.toml before the build. e2e/crossbuild now runs every sdist build with sandbox network blocked; all five Rust cases (maturin on rules_rust and rules_rs, setuptools-rust, with and without a shipped lock) pass offline.
…t-macos # Conflicts: # e2e/cases/uv-deps-650/crossbuild/snapshots/app_amd64_layers_listing.yaml # e2e/cases/uv-deps-650/crossbuild/snapshots/app_arm64_layers_listing.yaml
Rust sdists (maturin, setuptools-rust) no longer need a `rust_toolchain` attribute nor a rules_rust/rules_rs dependency: the `rust_tools` module extension downloads rustc + cargo + rust-std for the host and rust-std for every cross target from static.rust-lang.org, pinned by sha256, and registers one `rust_pep517_toolchain` per host x target pair. - uv/private/rust: versions (pins), repository rules, toolchain rule, extension; toolchain_type wired into a new `pep517_rust_whl` rule (factory over pep517_native_whl) that exports CARGO/RUSTC/sysroot/ target std roots to the build helper. - sdist_build emits `pep517_rust_whl` for Rust builds; setuptools-rust is only inferred under a setuptools backend (numpy ships meson's .rs test cases and must stay a meson build). - build_helper merges host sysroot + target rust-std the way rustup lays them out, for both native and cross cargo builds. - rust_layer.bzl, the `rust_toolchain` tag attr and the rules_rs e2e case are removed; e2e/crossbuild builds every Rust case offline.
The root and e2e/cases suites already run on Linux for every PR, and the crossbuild cases are exercised by the darwin cross-build pipeline. The job tripled macOS minutes without covering anything new.
…t-macos # Conflicts: # .github/workflows/ci-workflows.yaml # e2e/README.md # e2e/crossbuild/MODULE.bazel # e2e/crossbuild/pycross-setuptools/BUILD.bazel
This reverts commit 1258def.
numpy's sdist vendors meson's test suite, .rs files included, so the configure tool inferred setuptools-rust for it and sdist_build treated a meson build as Rust: it demanded a rust_toolchain the project never needed. setuptools-rust only ever rides on a setuptools backend (or a bare setup.py).
… with Without a toolchain, cargo or maturin failed deep inside the build action with no hint of the cause. sdist_build already knows at repository time that the sdist builds Rust; it now fails there, naming uv.project(rust_toolchain) and the reason (maturin backend, or setuptools-rust requirement). Hand-wired uv.override_package(toolchains = [...]) entries are trusted as the escape hatch. Docs say why the attribute exists at all: rules_py cannot resolve a Rust toolchain itself without making rules_rust a dependency of every consumer.
The inferred setuptools-rust requirement is a guess from stray .rs files: zstandard ships an optional rust-ext it never builds by default, and the fail-early check refused to generate its repository. A guess still wires the project's toolchain when there is one, but never demands one; the build proceeds unwired, as before, and fails only if the backend needs rustc.
… toolchain State the design plainly: fetching rustc and cargo is the Rust rulesets' job and rules_py depends on neither, so the project points uv.project at the toolchain it registers. Give the rules_rust label and the rules_rs wiring the e2e uses, and note that only declared Rust builds demand a toolchain.
pycross-rust, pycross-rust-rs and pycross-tiktoken get the pycross_wheel_export from #1507, so CI installs and runs their cross-built wheels on native amd64 and arm64 runners like every other crossbuild case. The rules_rs case's test file takes the bundle's name so the runner finds it.
…tically `uv.project(rust_toolchain = "@rules_rust//rust/toolchain:current_rust_toolchain")` declares the project's Rust toolchain once. sdist_build already knows each sdist's build backend and declared build requirements from the configure tool; when the backend is maturin, or setuptools-rust is among the requirements, the generated pep517_native_whl gets the toolchain plus a local rust_host_sysroot layer in `toolchains`. pep517_native_whl derives CARGO/RUSTC/RULES_PY_RUST_HOST_SYSROOT from their make-variables, so no per-package uv.override_package is needed for Rust. rust_host_sysroot moves from e2e/crossbuild/tools into rules_py: it re-resolves the toolchain in the exec configuration, because under a cross transition current_rust_toolchain returns the target-targeting toolchain, whose sysroot lacks the exec-platform std that build scripts and proc-macros need. It reads platform_common.ToolchainInfo only, so rules_py takes no dependency on rules_rust. Requirement names are normalized from their PEP 508 spelling (extras, specifiers and markers dropped): the configure tool reports build requirements as declared, e.g. `setuptools-rust>=1.7`. Inferred setuptools-rust counts too: the configure tool adds it for sdists that ship .rs files without declaring it, and injects it into the build venv, so the build needs the toolchain just the same. The inference is limited to setuptools backends (or a bare setup.py); numpy vendors meson's test suite, .rs files included, and stays a meson build. The helper no longer re-derives "is this a Rust build" from pyproject.toml, which would miss inferred builds: CARGO wired by the rule is the signal for the cargo cross environment. rust_toolchain travels as a label, not a string, so a relative or apparent-repo spelling resolves where the user wrote it. Tests: unittests for the detection and for the BUILD wiring it emits; an analysis test with a fake toolchain proves rust_host_sysroot exports the exec-configured sysroot; pycross-rust is the end-to-end consumer, now with no override at all.
…ot explicitly New e2e/crossbuild case pycross-rust-rs builds pydantic-core with rules_rs toolchains: a native import and linux/amd64 + linux/arm64 cross builds through uv.project(rust_toolchain = ...), mirroring pycross-rust. rules_rs declares its toolchains with rules_rust's rust_toolchain rule inside the patched rules_rust repository it provisions, so the same current_rust_toolchain label and providers apply, and both rulesets coexist in one workspace because each carries its own toolchain type. The case pins that the wiring is a provider contract, not a dependency on one ruleset. It also exposed a layout assumption: the helper ran rustc bare, which infers its sysroot from its own location, and rules_rs fetches rustc and rust-std into separate repositories (E0463: can't find crate for `std`). The rule now derives RUST_SYSROOT, the sysroot rules_rust's toolchain assembles, and the helper passes it: natively through a rustc wrapper (--sysroot), under cross as the target side of the merged sysroot instead of dirname(rustc). Tests: unit tests for the native cargo env and for the merge with a rules_rs-style layout; an analysis test with a fake toolchain exporting CARGO/RUSTC/RUST_SYSROOT asserts the derived env keys and that the make-variable names do not leak.
…line Cargo fetched its dependencies inside the build action, with whatever network the sandbox allowed: not hermetic, not reproducible beyond the lock the sdist happened to ship, and dead under remote execution. The C-family backends never had this problem: their build deps come from the uv lock and their toolchains are action inputs. sdist_configure now reads the sdist's Cargo.lock (anywhere in the tree, shallowest first: setuptools-rust keeps its crate under src/) and reports the crates it pins. For a Rust build, sdist_build downloads each crates.io crate through Bazel's downloader with the lock's checksum into a cargo vendor directory, and the generated pep517_native_whl passes it to the helper, which writes the source replacement into the sandbox CARGO_HOME and sets CARGO_NET_OFFLINE. Locks pinning git or path sources are rejected outright. Sdists without a Cargo.lock build online with a warning, or take one from uv.override_package(cargo_lock = ...): vendored the same way and placed next to the crate's Cargo.toml before the build. pycross-tiktoken (setuptools-rust, no shipped lock) is that consumer. e2e/crossbuild now runs every sdist build with sandbox network blocked; the whole suite passes offline, Rust cases included.
Without a toolchain, cargo or maturin fail deep inside the build action with no hint of the cause. sdist_build now fails while the repository is generated, naming the attribute to set, when the sdist declares a Rust build: a maturin backend, or setuptools-rust among its declared build requirements. Explicit uv.override_package(toolchains = [...]) entries are trusted as the by-hand escape hatch. Inferred setuptools-rust (stray .rs files) is only a guess and never demands a toolchain: zstandard ships an optional rust-ext it does not build by default. A guess still wires the project's toolchain when there is one; without one the build proceeds unwired, as before, and fails only if the backend really needs rustc.
pycross-rust, pycross-rust-rs and pycross-tiktoken get the pycross_wheel_export from #1507, so CI installs and runs their cross-built wheels on native amd64 and arm64 runners like every other crossbuild case. The rules_rs case's test file takes the bundle's name so the runner finds it. The crossbuild README lists the Rust cases alongside the other backends.
… toolchain State the design plainly: fetching rustc and cargo is the Rust rulesets' job and rules_py depends on neither, so the project points uv.project at the toolchain it registers. Give the rules_rust label and the rules_rs wiring the e2e uses, and note that only declared Rust builds demand a toolchain.
…n uv.project()
`uv.project()` goes back to hub, lock and pyproject. The toolchains the
module's sdists build with move to their own tag on the same extension,
resolved in _parse_projects next to override_package and annotate_packages
and producing the same sdist_build repositories:
uv.package_toolchains(rust_toolchain = "@rules_rust//rust/toolchain:current_rust_toolchain")
Without `lock` the declaration covers every uv.project() of the module, one
line per MODULE.bazel instead of one per lock. With `lock` it applies to that
project only and wins over the module-wide one, which is how a workspace
builds one project on rules_rust and another on rules_rs. Two declarations
with the same scope, a lock without a uv.project(), and a tag naming no
toolchain fail with their own messages.
The attribute stays `rust_toolchain` rather than a generic list because
sdist_build has to know which toolchain is the Rust one to wire it only into
Rust builds with the exec-sysroot layer; other project-level toolchains get
their own attribute when needed. `cargo_lock` remains per package on
override_package: it is data for the offline crate vendoring, not a
toolchain.
e2e/crossbuild exercises the three scopes: pycross-rust declares the
module-wide rules_rust toolchain, pycross-rust-rs overrides it for its lock
with the rules_rs one, pycross-tiktoken inherits.
maturin 1.8+ fetches rustc through puccinialin when it finds no cargo, so a Rust sdist with no toolchain wired, or one the setuptools-rust inference left unwired, would build with a toolchain downloaded inside the action. Every sdist build now carries MATURIN_NO_INSTALL_RUST=1; a package env entry still wins.
…dists The rustc wrapper only injected --sysroot, so a Rust wheel carried the sandbox and execroot paths of the host that built it (debug info, panic locations) and LLVM's per-host module ids in symbol names: the wheel action itself hits the remote cache, but its bytes differ per host, so every downstream action (whl_install, venvs) misses. The wrapper now remaps the sandbox root and the execroot away and compiles target crates as a single codegen unit; build scripts and proc-macros, which never reach the wheel, keep cargo's codegen. Native builds treat every crate as target. Crates with C or C++ sources (ring, zstd-sys) build them through cc-rs, which looks up CC_<triple>, CXX_<triple>, AR_<triple> and RANLIB_<triple> before falling back to the PATH; only CARGO_TARGET_<TRIPLE>_LINKER was set, so those objects came from whatever compiler the runner had. Cross builds now export the wired C toolchain under both spellings, with a ranlib wrapper over `ar s` for toolchains that ship none. The linker stays the C driver: linking with c++ would add an implicit libstdc++ dependency to every Rust extension, not only the ones with C++ sources. Tests run the generated wrapper around an argv-echoing rustc and check the cc-rs variables end to end through the cross env.
The CycloneDX SBOM maturin writes into the wheel records every crate as path+file:///<sandbox>/..., and the sandbox id differs per action, so two builds of one sdist never produced the same wheel even with the rustc wrapper remapping paths. maturin 1.15 has no flag or variable for it, only the [tool.maturin.sbom] table, appended to the extracted pyproject.toml unless the sdist configures it itself.
…toolchains Two review asks on uv.package_toolchains(). A declaration can now name a package, like uv.override_package(): `name` alone covers that package in every project of the module, `name` with `lock` covers it in one project. And the tag carries `toolchains`, the generic list uv.override_package() forwards per package, so a JDK or Ant reach every source build of a module or project with one declaration. Each attribute resolves from the most specific declaration that sets it: package in lock, package everywhere, lock, module. `rust_toolchain` stays its own attribute because sdist_build wires it only into maturin and setuptools-rust builds with the exec-sysroot layer, while `toolchains` is forwarded as-is. A `name` that matches no source build fails, like an uv.override_package() that matches nothing; two declarations with the same scope fail. pycross-jdk moves its JDK and Ant from the per-package override to a lock-scoped declaration.
Cargo's `-C metadata=` hash, which rustc mangles into every symbol and uses as the crate's stable id, mixes in `rustc -vV` (host triple included) and the paths of host-compiled build scripts, so a wheel's `.so` differed by build host even with paths remapped. The rustc wrapper now gives target crates a hash derived from host-independent identity: the toolchain's `rustc --version` string, `CARGO_PKG_NAME` and `CARGO_PKG_VERSION` from the environment cargo hands rustc, the crate name, its types, cfgs, target and the codegen options its profile sets. Two versions of one package hash apart, and so do the two compilations a native build makes of one crate under differing `build-override` and normal profiles. Path-carrying options (`linker`, `link-arg`) are excluded; `-C extra-filename` stays cargo's so artifact names never collide. Host crates keep cargo's hash: they never reach the wheel. The wrapper is exercised around an argv-echoing fake rustc: same hash from two sandboxes, different across versions, features and toolchains, host crates untouched, both `-C metadata=` spellings.
crossbuild/test.sh becomes the same aggregator cases/test.sh is: it runs every `<case>/test.sh`, so a case that needs a real `bazel run`, the host's identity or network the action sandbox denies owns its own script instead of growing the workspace-level one. The macOS amd64 cross build of pycross-geohash moves to pycross-geohash/test.sh.
Every sdist_build repository of a Rust sdist now carries a `cargo_lock_generator` target. `bazel run` extracts the sdist, runs `cargo generate-lockfile` on its shallowest manifest with the project's own `rust_toolchain` (cargo and rustc from the toolchain's runfiles) and copies the lock into the workspace: `<name>-<version>.Cargo.lock` at the root, the first argument, or in place when `uv.override_package(cargo_lock = ...)` already declares one. It prints the override to add. The warning for an sdist that ships no Cargo.lock names the target. The generated script is checked in an analysis test around a fake toolchain exposing cargo and rustc as Files; the BUILD rendering and the lock path derivation have unit tests; pycross-tiktoken/test.sh runs the real target against crates.io and checks the result is a lock for that crate. Regenerating tiktoken's declared lock reproduces the checked-in file byte for byte. docs: the generator, the symbol-hash normalization, and why rules_py writes no PYO3_CONFIG_FILE (maturin writes its own in a cross build; setuptools-rust derives the target from PYO3_CROSS_PYTHON_VERSION).
The Rust toolchain gets its own tag, `uv.rust_toolchain(toolchain = ..., lock = ...)`, scoped to the module or to one project's lock; a lock-scoped declaration wins for its project. Detection makes the wide scope precise: sdist_build wires the toolchain only into builds whose backend is maturin or setuptools-rust, so nothing else sees it. Toolchains a single package needs beyond Rust (a JDK, Ant) stay where they were before bc44642, on `uv.override_package(toolchains = ...)`. Their files become action inputs of that package's build alone, instead of every sdist built from source in the scope. The `name`-scoped variant and the "matches no source build" check go away with the generic list; pycross-jdk returns to its per-package override. `cargo_lock` remains an override_package attribute: a lock pins one sdist version's crates, a build input like pre_build_patches, not a toolchain.
…-sdist-macos Brings #1526's final state onto the spike: uv.rust_toolchain(toolchain, lock) replaces the spike's uv.project(rust_toolchain = ...), generic toolchains stay per package on uv.override_package, offline crate vendoring with the :cargo_lock generator, the reproducible rustc wrapper (remapped paths, one codegen unit, host-independent -C metadata), cc-rs environment, maturin SBOM off, and the per-case test.sh aggregator. The spike's own helper work is replayed on top: pkg-config's python3 pointed at the interpreter the wheel is built for, meson-python's build dir kept so meson-log.txt survives a failed build, and static libstdc++ for GNU C++ links. pycross-bcrypt and pycross-rpds_py move to the module-wide uv.rust_toolchain() declared in pycross-rust.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes are visible to end-users: yes/no
Test plan