feat(uv): build Rust sdists with the project's rules_rust or rules_rs toolchain - #1526
Open
xangcastle wants to merge 19 commits into
Open
xangcastle wants to merge 19 commits into
xangcastle wants to merge 19 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
py_binary startup benchmark
sys.path quality
Bazel analysis benchmark
py_image_layer benchmark
|
xangcastle
force-pushed
the
xangcastle/pep517-rust-toolchain
branch
from
September 8, 2026 12:07
227775d to
c303a1b
Compare
Aspect Workflows Tasks📅 Fri Sep 18 02:03:11 UTC 2026 Task Results
⏱ Last updated Fri Sep 18 02:08:25 UTC 2026 · 📊 GitHub API quota 469/7,700 (6% used, resets in 54m) |
xangcastle
force-pushed
the
xangcastle/pep517-rust-toolchain
branch
from
September 8, 2026 12:36
c303a1b to
7ad6e75
Compare
xangcastle
marked this pull request as ready for review
September 8, 2026 13:01
Member
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7ad6e75644
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
jbedard
reviewed
Sep 8, 2026
jbedard
reviewed
Sep 8, 2026
jbedard
reviewed
Sep 8, 2026
xangcastle
force-pushed
the
xangcastle/pep517-rust-toolchain
branch
2 times, most recently
from
September 8, 2026 21:57
32680ac to
8954140
Compare
xangcastle
force-pushed
the
xangcastle/pep517-rust-toolchain
branch
2 times, most recently
from
September 9, 2026 13:19
454282e to
b2164a9
Compare
xangcastle
changed the title
jbedard
pushed a commit
that referenced
this pull request
Sep 9, 2026
CI and e2e only: no change under `uv/`; the public API is unchanged.
**Native-arch verification** (`.github/workflows/ci-workflows.yaml`):
each crossbuild case exports `<case>_{amd64,arm64}_bundle` tarballs
(wheel plus its standalone `test_<case>.py`,
`e2e/crossbuild/tools/wheel_export.bzl`). Three pipelines cross-build on
one runner, upload the bundles, and `pip install` + run them on a native
runner of the target architecture, with no containers and no emulation
in the verdict:
- darwin → linux/amd64 + linux/arm64 (gated to `main` and `*macos*`
branches),
- ubuntu/amd64 → ubuntu/arm64,
- ubuntu/arm64 → ubuntu/amd64.
`test-all` gates on the two Linux pipelines and accepts `skipped` for
the macOS one outside the gated branches. The bundle-building steps pass
`--remote_download_outputs=toplevel`: `--config=ci` builds without the
bytes, and with a warm cache the tarballs this job uploads were not
materialized (the first run of this branch passed only because its cache
was cold).
**New cases**, all built for linux/amd64 and linux/arm64 with
`collect_wheels` (Tag metadata, ELF arch of every bundled `.so`) plus a
native import test:
- `pycross-geohash`: C++ extension with no upstream wheel. On a macOS
host, `test.sh` also cross-builds it for macOS amd64 (a manual
`collect_wheels` with `check_elf = False`; the platform transition
always resolves to os:macos, so `target_compatible_with` cannot tell
hosts apart).
- `pycross-msgpack`: Cython-generated C extension via setuptools.
- `pycross-numpy`: meson-python with `config_settings = {"setup-args":
["-Dblas=none", "-Dlapack=none"]}` (#1524); the hermetic sandbox has no
BLAS.
- `pycross-psutil`: plain setuptools C extension.
- `pycross-compile_pyc`: regression guard for cross-arch `whl_install`
pyc precompilation.
The Rust cases stay out of this PR on purpose: `pycross-rust` keeps
building as on `main` and gets no export, so the pipelines never depend
on the Rust toolchain work in #1526. Once that lands, the Rust cases
join the native runs with a one-line `pycross_wheel_export` each.
The macOS `smoke` job is removed: the root and `e2e/cases` suites
already run on Linux for every PR, and the crossbuild cases are
exercised by the darwin cross-build pipeline above, so it tripled macOS
minutes without covering anything new.
`collect_wheels` gains `tags` and `check_elf` for the macOS collection;
`pycross-setuptools` shares its setproctitle wheel label between
`collect_wheels` and the export instead of repeating it.
---
### Changes are visible to end-users: no
### Test plan
- New test cases added
- `cd e2e/crossbuild && bazel test //... && bash test.sh` passes
locally.
- CI: the three build/run pipelines plus `test-all`.
xangcastle
force-pushed
the
xangcastle/pep517-rust-toolchain
branch
from
September 10, 2026 12:07
0ea8797 to
1b7bec1
Compare
xangcastle
changed the title
jbedard
reviewed
Sep 10, 2026
…tically `uv.project(rust_toolchain = "@rules_rust//rust/toolchain:current_rust_toolchain")` declares the project's Rust toolchain once. sdist_build already knows each sdist's build backend and declared build requirements from the configure tool; when the backend is maturin, or setuptools-rust is among the requirements, the generated pep517_native_whl gets the toolchain plus a local rust_host_sysroot layer in `toolchains`. pep517_native_whl derives CARGO/RUSTC/RULES_PY_RUST_HOST_SYSROOT from their make-variables, so no per-package uv.override_package is needed for Rust. rust_host_sysroot moves from e2e/crossbuild/tools into rules_py: it re-resolves the toolchain in the exec configuration, because under a cross transition current_rust_toolchain returns the target-targeting toolchain, whose sysroot lacks the exec-platform std that build scripts and proc-macros need. It reads platform_common.ToolchainInfo only, so rules_py takes no dependency on rules_rust. Requirement names are normalized from their PEP 508 spelling (extras, specifiers and markers dropped): the configure tool reports build requirements as declared, e.g. `setuptools-rust>=1.7`. Inferred setuptools-rust counts too: the configure tool adds it for sdists that ship .rs files without declaring it, and injects it into the build venv, so the build needs the toolchain just the same. The inference is limited to setuptools backends (or a bare setup.py); numpy vendors meson's test suite, .rs files included, and stays a meson build. The helper no longer re-derives "is this a Rust build" from pyproject.toml, which would miss inferred builds: CARGO wired by the rule is the signal for the cargo cross environment. rust_toolchain travels as a label, not a string, so a relative or apparent-repo spelling resolves where the user wrote it. Tests: unittests for the detection and for the BUILD wiring it emits; an analysis test with a fake toolchain proves rust_host_sysroot exports the exec-configured sysroot; pycross-rust is the end-to-end consumer, now with no override at all.
…ot explicitly New e2e/crossbuild case pycross-rust-rs builds pydantic-core with rules_rs toolchains: a native import and linux/amd64 + linux/arm64 cross builds through uv.project(rust_toolchain = ...), mirroring pycross-rust. rules_rs declares its toolchains with rules_rust's rust_toolchain rule inside the patched rules_rust repository it provisions, so the same current_rust_toolchain label and providers apply, and both rulesets coexist in one workspace because each carries its own toolchain type. The case pins that the wiring is a provider contract, not a dependency on one ruleset. It also exposed a layout assumption: the helper ran rustc bare, which infers its sysroot from its own location, and rules_rs fetches rustc and rust-std into separate repositories (E0463: can't find crate for `std`). The rule now derives RUST_SYSROOT, the sysroot rules_rust's toolchain assembles, and the helper passes it: natively through a rustc wrapper (--sysroot), under cross as the target side of the merged sysroot instead of dirname(rustc). Tests: unit tests for the native cargo env and for the merge with a rules_rs-style layout; an analysis test with a fake toolchain exporting CARGO/RUSTC/RUST_SYSROOT asserts the derived env keys and that the make-variable names do not leak.
…line Cargo fetched its dependencies inside the build action, with whatever network the sandbox allowed: not hermetic, not reproducible beyond the lock the sdist happened to ship, and dead under remote execution. The C-family backends never had this problem: their build deps come from the uv lock and their toolchains are action inputs. sdist_configure now reads the sdist's Cargo.lock (anywhere in the tree, shallowest first: setuptools-rust keeps its crate under src/) and reports the crates it pins. For a Rust build, sdist_build downloads each crates.io crate through Bazel's downloader with the lock's checksum into a cargo vendor directory, and the generated pep517_native_whl passes it to the helper, which writes the source replacement into the sandbox CARGO_HOME and sets CARGO_NET_OFFLINE. Locks pinning git or path sources are rejected outright. Sdists without a Cargo.lock build online with a warning, or take one from uv.override_package(cargo_lock = ...): vendored the same way and placed next to the crate's Cargo.toml before the build. pycross-tiktoken (setuptools-rust, no shipped lock) is that consumer. e2e/crossbuild now runs every sdist build with sandbox network blocked; the whole suite passes offline, Rust cases included.
Without a toolchain, cargo or maturin fail deep inside the build action with no hint of the cause. sdist_build now fails while the repository is generated, naming the attribute to set, when the sdist declares a Rust build: a maturin backend, or setuptools-rust among its declared build requirements. Explicit uv.override_package(toolchains = [...]) entries are trusted as the by-hand escape hatch. Inferred setuptools-rust (stray .rs files) is only a guess and never demands a toolchain: zstandard ships an optional rust-ext it does not build by default. A guess still wires the project's toolchain when there is one; without one the build proceeds unwired, as before, and fails only if the backend really needs rustc.
pycross-rust, pycross-rust-rs and pycross-tiktoken get the pycross_wheel_export from #1507, so CI installs and runs their cross-built wheels on native amd64 and arm64 runners like every other crossbuild case. The rules_rs case's test file takes the bundle's name so the runner finds it. The crossbuild README lists the Rust cases alongside the other backends.
… toolchain State the design plainly: fetching rustc and cargo is the Rust rulesets' job and rules_py depends on neither, so the project points uv.project at the toolchain it registers. Give the rules_rust label and the rules_rs wiring the e2e uses, and note that only declared Rust builds demand a toolchain.
xangcastle
force-pushed
the
xangcastle/pep517-rust-toolchain
branch
from
September 10, 2026 19:30
1b7bec1 to
1f4c6cd
Compare
…n uv.project()
`uv.project()` goes back to hub, lock and pyproject. The toolchains the
module's sdists build with move to their own tag on the same extension,
resolved in _parse_projects next to override_package and annotate_packages
and producing the same sdist_build repositories:
uv.package_toolchains(rust_toolchain = "@rules_rust//rust/toolchain:current_rust_toolchain")
Without `lock` the declaration covers every uv.project() of the module, one
line per MODULE.bazel instead of one per lock. With `lock` it applies to that
project only and wins over the module-wide one, which is how a workspace
builds one project on rules_rust and another on rules_rs. Two declarations
with the same scope, a lock without a uv.project(), and a tag naming no
toolchain fail with their own messages.
The attribute stays `rust_toolchain` rather than a generic list because
sdist_build has to know which toolchain is the Rust one to wire it only into
Rust builds with the exec-sysroot layer; other project-level toolchains get
their own attribute when needed. `cargo_lock` remains per package on
override_package: it is data for the offline crate vendoring, not a
toolchain.
e2e/crossbuild exercises the three scopes: pycross-rust declares the
module-wide rules_rust toolchain, pycross-rust-rs overrides it for its lock
with the rules_rs one, pycross-tiktoken inherits.
maturin 1.8+ fetches rustc through puccinialin when it finds no cargo, so a Rust sdist with no toolchain wired, or one the setuptools-rust inference left unwired, would build with a toolchain downloaded inside the action. Every sdist build now carries MATURIN_NO_INSTALL_RUST=1; a package env entry still wins.
jbedard
reviewed
Sep 10, 2026
jbedard
reviewed
Sep 10, 2026
…dists The rustc wrapper only injected --sysroot, so a Rust wheel carried the sandbox and execroot paths of the host that built it (debug info, panic locations) and LLVM's per-host module ids in symbol names: the wheel action itself hits the remote cache, but its bytes differ per host, so every downstream action (whl_install, venvs) misses. The wrapper now remaps the sandbox root and the execroot away and compiles target crates as a single codegen unit; build scripts and proc-macros, which never reach the wheel, keep cargo's codegen. Native builds treat every crate as target. Crates with C or C++ sources (ring, zstd-sys) build them through cc-rs, which looks up CC_<triple>, CXX_<triple>, AR_<triple> and RANLIB_<triple> before falling back to the PATH; only CARGO_TARGET_<TRIPLE>_LINKER was set, so those objects came from whatever compiler the runner had. Cross builds now export the wired C toolchain under both spellings, with a ranlib wrapper over `ar s` for toolchains that ship none. The linker stays the C driver: linking with c++ would add an implicit libstdc++ dependency to every Rust extension, not only the ones with C++ sources. Tests run the generated wrapper around an argv-echoing rustc and check the cc-rs variables end to end through the cross env.
The CycloneDX SBOM maturin writes into the wheel records every crate as path+file:///<sandbox>/..., and the sandbox id differs per action, so two builds of one sdist never produced the same wheel even with the rustc wrapper remapping paths. maturin 1.15 has no flag or variable for it, only the [tool.maturin.sbom] table, appended to the extracted pyproject.toml unless the sdist configures it itself.
…toolchains Two review asks on uv.package_toolchains(). A declaration can now name a package, like uv.override_package(): `name` alone covers that package in every project of the module, `name` with `lock` covers it in one project. And the tag carries `toolchains`, the generic list uv.override_package() forwards per package, so a JDK or Ant reach every source build of a module or project with one declaration. Each attribute resolves from the most specific declaration that sets it: package in lock, package everywhere, lock, module. `rust_toolchain` stays its own attribute because sdist_build wires it only into maturin and setuptools-rust builds with the exec-sysroot layer, while `toolchains` is forwarded as-is. A `name` that matches no source build fails, like an uv.override_package() that matches nothing; two declarations with the same scope fail. pycross-jdk moves its JDK and Ant from the per-package override to a lock-scoped declaration.
jbedard
reviewed
Sep 11, 2026
jbedard
reviewed
Sep 11, 2026
Member
|
@tamird please review the primary rust sdists API here. Also review the API proposals made in PR comments as well as any other sdists and crossbuild related issues you find |
Cargo's `-C metadata=` hash, which rustc mangles into every symbol and uses as the crate's stable id, mixes in `rustc -vV` (host triple included) and the paths of host-compiled build scripts, so a wheel's `.so` differed by build host even with paths remapped. The rustc wrapper now gives target crates a hash derived from host-independent identity: the toolchain's `rustc --version` string, `CARGO_PKG_NAME` and `CARGO_PKG_VERSION` from the environment cargo hands rustc, the crate name, its types, cfgs, target and the codegen options its profile sets. Two versions of one package hash apart, and so do the two compilations a native build makes of one crate under differing `build-override` and normal profiles. Path-carrying options (`linker`, `link-arg`) are excluded; `-C extra-filename` stays cargo's so artifact names never collide. Host crates keep cargo's hash: they never reach the wheel. The wrapper is exercised around an argv-echoing fake rustc: same hash from two sandboxes, different across versions, features and toolchains, host crates untouched, both `-C metadata=` spellings.
crossbuild/test.sh becomes the same aggregator cases/test.sh is: it runs every `<case>/test.sh`, so a case that needs a real `bazel run`, the host's identity or network the action sandbox denies owns its own script instead of growing the workspace-level one. The macOS amd64 cross build of pycross-geohash moves to pycross-geohash/test.sh.
Every sdist_build repository of a Rust sdist now carries a `cargo_lock_generator` target. `bazel run` extracts the sdist, runs `cargo generate-lockfile` on its shallowest manifest with the project's own `rust_toolchain` (cargo and rustc from the toolchain's runfiles) and copies the lock into the workspace: `<name>-<version>.Cargo.lock` at the root, the first argument, or in place when `uv.override_package(cargo_lock = ...)` already declares one. It prints the override to add. The warning for an sdist that ships no Cargo.lock names the target. The generated script is checked in an analysis test around a fake toolchain exposing cargo and rustc as Files; the BUILD rendering and the lock path derivation have unit tests; pycross-tiktoken/test.sh runs the real target against crates.io and checks the result is a lock for that crate. Regenerating tiktoken's declared lock reproduces the checked-in file byte for byte. docs: the generator, the symbol-hash normalization, and why rules_py writes no PYO3_CONFIG_FILE (maturin writes its own in a cross build; setuptools-rust derives the target from PYO3_CROSS_PYTHON_VERSION).
The Rust toolchain gets its own tag, `uv.rust_toolchain(toolchain = ..., lock = ...)`, scoped to the module or to one project's lock; a lock-scoped declaration wins for its project. Detection makes the wide scope precise: sdist_build wires the toolchain only into builds whose backend is maturin or setuptools-rust, so nothing else sees it. Toolchains a single package needs beyond Rust (a JDK, Ant) stay where they were before bc44642, on `uv.override_package(toolchains = ...)`. Their files become action inputs of that package's build alone, instead of every sdist built from source in the scope. The `name`-scoped variant and the "matches no source build" check go away with the generic list; pycross-jdk returns to its per-package override. `cargo_lock` remains an override_package attribute: a lock pins one sdist version's crates, a build input like pre_build_patches, not a toolchain.
xangcastle
added a commit
that referenced
this pull request
Sep 15, 2026
…-sdist-macos Brings #1526's final state onto the spike: uv.rust_toolchain(toolchain, lock) replaces the spike's uv.project(rust_toolchain = ...), generic toolchains stay per package on uv.override_package, offline crate vendoring with the :cargo_lock generator, the reproducible rustc wrapper (remapped paths, one codegen unit, host-independent -C metadata), cc-rs environment, maturin SBOM off, and the per-case test.sh aggregator. The spike's own helper work is replayed on top: pkg-config's python3 pointed at the interpreter the wheel is built for, meson-python's build dir kept so meson-log.txt survives a failed build, and static libstdc++ for GNU C++ links. pycross-bcrypt and pycross-rpds_py move to the module-wide uv.rust_toolchain() declared in pycross-rust.
jbedard
reviewed
Sep 15, 2026
The configure tool infers setuptools-rust from stray .rs files so the package lands in the build venv. That guess also drove the toolchain wiring: with a module-wide uv.rust_toolchain(), zstandard (an optional Rust extension its setup.py never builds by default) got the toolchain and the exec sysroot layer as action inputs, a :cargo_lock target, and 47 crates vendored at repository generation. Wiring now requires a declared Rust build, the same predicate the fail-early check already used: a maturin backend, or setuptools-rust in [build-system].requires, setup.cfg or setup.py setup_requires. Inference keeps injecting setuptools-rust into the venv and nothing else.
crossbuild pins the generated BUILD.bazel of its 16 sdist_build repositories under snapshots/, one write_source_files diff test each, so the toolchains, Rust sysroot layer, vendored crates and build deps every sdist receives are a reviewable diff. A toolchain reaching a package whose backend does not need it now fails the suite instead of silently growing its action inputs. `bazel run //:sdist_build_snapshots` updates them.
xangcastle
added a commit
that referenced
this pull request
Sep 18, 2026
…-sdist-macos Syncs the spike with #1526's Rust wiring fix (declared Rust builds only, so zstandard's optional extension no longer pulls the toolchain or vendors crates) and the sdist_build BUILD snapshots. The snapshot list covers the spike's 18 repositories, bcrypt and rpds_py included, and setproctitle at the 1.3.7 the spike pins.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rust sdists (maturin, setuptools-rust) build with the Rust toolchain the project registers. rules_py depends on neither rules_rust nor rules_rs and works with both.
API
uv.rust_toolchain(toolchain = "@rules_rust//rust/toolchain:current_rust_toolchain", lock = ...): module-wide or per lock, the lock wins. sdist_build wires it, plus an exec-configuredrust_host_sysrootlayer, into every sdist whose backend is maturin, or setuptools with setuptools-rust declared in its build requirements. Such an sdist with no toolchain fails at repository generation naming the declaration to add. Stray.rsfiles (zstandard's optional extension) neither wire nor demand one.uv.override_package(cargo_lock = ...): a Cargo.lock for an sdist that ships none. Crates are vendored from the lock at repository generation and cargo builds offline; git and path sources are rejected.bazel run @sdist_build__<project>__<pkg>__<ver>//:cargo_lockwrites that lock with the project's toolchain.Build
CC_<triple>,CXX_<triple>,AR_<triple>andRANLIB_<triple>.--remap-path-prefixfor sandbox and execroot,-C codegen-units=1and a host-independent-C metadata=for target crates, maturin's SBOM off.MATURIN_NO_INSTALL_RUST=1on every build.e2e/crossbuild: pycross-rust (rules_rust), pycross-rust-rs (rules_rs, lock-scoped), pycross-tiktoken (setuptools-rust,
cargo_lock); their wheels run on native amd64 and arm64 runners.test.shaggregates per-case scripts; pycross-tiktoken's runs:cargo_lock. The BUILD every sdist_build repository generates is snapshotted undersnapshots/, so which sdists get toolchains, the Rust layer or vendored crates is a reviewable diff.Changes are visible to end-users: yes
New
uv.rust_toolchain(),uv.override_package(cargo_lock = ...)and:cargo_locktargets. Rust sdists no longer need a per-packageuv.override_package(toolchains = [...]).Test plan
cd e2e/crossbuild && bazel test //... && bash test.shpasses locally with the sandbox network blocked, cross-building pydantic-core (rules_rust and rules_rs) and tiktoken for linux/amd64 and linux/arm64 from a macOS host.docs/uv.mdupdated.