Skip to content

feat(uv): reproducible rustc wrapper and cc-rs environment for Rust sdists - #1537

Closed
xangcastle wants to merge 2 commits into
xangcastle/pep517-rust-toolchainfrom
xangcastle/pep517-rust-reproducible
Closed

xangcastle wants to merge 2 commits into
xangcastle/pep517-rust-toolchainfrom
xangcastle/pep517-rust-reproducible

Conversation

@xangcastle

Copy link
Copy Markdown
Member

Stacked on #1526. Helper-only: the rustc wrapper and the cargo environment that PR introduces.

Reproducible rustc wrapper. The wrapper only injected --sysroot, so a Rust wheel carried the sandbox and execroot paths of the host that built it (debug info, panic locations) and LLVM's per-host module ids in symbol names. The wheel action itself hits the remote cache, but its bytes differ per host, so every downstream action (whl_install, venvs) misses. The wrapper now passes --remap-path-prefix for the sandbox root and the execroot and -C codegen-units=1 for target crates; build scripts and proc-macros, which never reach the wheel, keep cargo's codegen. Native builds treat every crate as target. maturin's CycloneDX SBOM is turned off in the extracted pyproject.toml unless the sdist configures [tool.maturin.sbom] itself: it records every crate as path+file:///<sandbox>/..., and the sandbox id differs per action, so no two builds agreed while it was on (maturin 1.15 has no flag or variable for it). Rewriting cargo's -C metadata= hash is left for a follow-up: it needs a host-independent toolchain identity, and the changes here already make two checkouts produce identical wheels (see test plan).

cc-rs environment. Crates with C or C++ sources (ring, zstd-sys) build them through cc-rs, which looks up CC_<triple>, CXX_<triple>, AR_<triple> and RANLIB_<triple> (dashed and underscored spellings) before falling back to the PATH; only CARGO_TARGET_<TRIPLE>_LINKER was set. Cross builds now export the wired C toolchain under those names, with a ranlib wrapper over ar s for toolchains that ship none (shared with the CMake toolchain file). The cargo linker stays the C driver: linking with c++ would give every Rust extension an implicit libstdc++ dependency, not only those with C++ sources.

Tests: the generated wrapper is run around an argv-echoing fake rustc (remap flags, codegen-units gating on --target, native mode, cargo's arguments untouched); the cc-rs variables are checked directly and through the cross env.


Changes are visible to end-users: no

Test plan

  • New test cases added (see above).

  • cd e2e/crossbuild && bazel test //pycross-rust/... //pycross-rust-rs/... //pycross-tiktoken/... passes offline (native, linux/amd64 and linux/arm64 from a macOS host).

  • Reproducibility: the same commit checked out at two paths, hence two output bases and sandbox ids, builds byte-identical cross pydantic-core wheels:

    wheel sha256 (both checkouts)
    pydantic_core-2.47.0-cp312-cp312-linux_x86_64.whl 0c5d3027e6be866c0005247c277263560450fe6a57c2d01aa0665d718217a723
    pydantic_core-2.47.0-cp312-cp312-linux_aarch64.whl b9c9af30a4faae3ebc6781a304307722320627cb1c84de9b554ae9c94df20729

    Before this PR the .so differed by sandbox paths and, with the wrapper alone, only the SBOM and its RECORD line still differed.

…dists

The rustc wrapper only injected --sysroot, so a Rust wheel carried the
sandbox and execroot paths of the host that built it (debug info, panic
locations) and LLVM's per-host module ids in symbol names: the wheel action
itself hits the remote cache, but its bytes differ per host, so every
downstream action (whl_install, venvs) misses. The wrapper now remaps the
sandbox root and the execroot away and compiles target crates as a single
codegen unit; build scripts and proc-macros, which never reach the wheel,
keep cargo's codegen. Native builds treat every crate as target.

Crates with C or C++ sources (ring, zstd-sys) build them through cc-rs,
which looks up CC_<triple>, CXX_<triple>, AR_<triple> and RANLIB_<triple>
before falling back to the PATH; only CARGO_TARGET_<TRIPLE>_LINKER was set,
so those objects came from whatever compiler the runner had. Cross builds
now export the wired C toolchain under both spellings, with a ranlib
wrapper over `ar s` for toolchains that ship none. The linker stays the C
driver: linking with c++ would add an implicit libstdc++ dependency to
every Rust extension, not only the ones with C++ sources.

Tests run the generated wrapper around an argv-echoing rustc and check the
cc-rs variables end to end through the cross env.
The CycloneDX SBOM maturin writes into the wheel records every crate as
path+file:///<sandbox>/..., and the sandbox id differs per action, so two
builds of one sdist never produced the same wheel even with the rustc
wrapper remapping paths. maturin 1.15 has no flag or variable for it, only
the [tool.maturin.sbom] table, appended to the extracted pyproject.toml
unless the sdist configures it itself.
@coderabbitai

coderabbitai Bot commented Sep 10, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 501f211e-e9b3-4d2f-9939-79d14bf01031

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@xangcastle

Copy link
Copy Markdown
Member Author

Folded into #1526 as commits 0c8eae1 and 5e93608; same tree, same evidence.

@xangcastle xangcastle closed this Sep 10, 2026
@github-actions

Copy link
Copy Markdown

py_binary startup benchmark

Version Time (ms) vs BCR vs main Build (s)
BCR 1.11.7 167.2/166.7 ±2.4 1.64
main 58.5/58.3 ±0.6 -65.0% 1.77
PR 58.2/58.1 ±0.6 -65.2% -0.4% 1.52

Time = mean/median ±stddev.
Measured with hyperfine --warmup 5 --runs 50 --shell=none on Linux
Gate: PR vs HEAD main median (threshold: 10%, and must exceed the 2×SE noise floor, here 0.4%). BCR is shown only as a historical baseline.
Build time: cold bazel build //:bench with isolated output base, no disk cache; external repos prefetched so network is excluded.

sys.path quality

Version entries sp roots dupes
BCR 1.11.7 6 1 0
main 7 2 0
PR 7 2 0

sys.path quality measured by bench_syspath inside the assembled venv: sys.path entries, distinct site-packages roots, duplicate realpaths. Duplicates indicate symlink redundancy; many roots suggest an inefficient venv layout.

Bazel analysis benchmark

Version Time (ms) vs BCR vs main Targets Actions Configured (ext) Cfg/target Actions/cfg
BCR 2.0.0-alpha.6 1462/1430 ±261 404 20613 7380 (6675) 18.3 2.79
main 1381/1351 ±262 -5.5% 404 18363 7379 (6674) 18.3 2.49
PR 1450/1361 ±319 -4.8% +0.7% 404 18363 7379 (6674) 18.3 2.49

Measured with hyperfine --warmup 1 --runs 10 on Linux
Time = mean/median ±stddev. Cfg = configured targets; (ext) = the count in external repos (the @pypi hub machinery).
Gate: PR vs HEAD main median (threshold: 10%, and must exceed the 2×SE noise floor, here 19.3%). BCR is shown only as a historical baseline.
Command: warm-server bazel build --nobuild //workspace/..., analysis cache discarded each run via a fresh --action_env value; no disk cache.
Workload: 100 py_test targets — 2 test files/package in file generation mode (one venv per test file); a single dep_group.

py_image_layer benchmark

Scenario Version Time (s) Actions vs BCR vs main
analysis BCR 2.0.0-alpha.6 0.77/0.76 ±0.15 747
analysis main 0.73/0.71 ±0.14 668 -5.8%
analysis PR 0.74/0.68 ±0.15 668 -10.2% -4.8%
1p source BCR 2.0.0-alpha.6 0.74/0.74 ±0.05 4
1p source main 0.76/0.76 ±0.05 5 +2.8%
1p source PR 0.75/0.75 ±0.06 5 +1.0% -1.7%
3p wheel BCR 2.0.0-alpha.6 0.86/0.85 ±0.03 5
3p wheel main 0.93/0.92 ±0.05 6 +7.6%
3p wheel PR 0.92/0.93 ±0.04 6 +8.5% +0.8%

Time = mean/median ±stddev.
Measured with hyperfine on Linux, building //workspace:image_layers (10 binaries, ~30-wheel dep pool, grouped first-party/pip/interpreter tier) with isolated output base, no disk cache.
Scenarios: analysis = warm-server bazel build --nobuild, re-analyzed each run via a fresh --action_env value; incrementals run against a built state with warm analysis: source = append to the last package's lib.py, wheel = rewrite click post_install_patches content.
Actions: for Analysis, the total action count behind the image target from aquery deps(...); for incrementals, actions re-executed for the mutation, from a single instrumented run's BEP build metrics (deterministic; per-mnemonic breakdown in the *-actions.json artifacts). Informational only, not gated.
Gate: PR vs HEAD main median per scenario (threshold: 10%, and must exceed the 2×SE noise floor). BCR is shown only as a historical baseline.

@xangcastle
xangcastle deleted the xangcastle/pep517-rust-reproducible branch September 10, 2026 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant