feat: Add cross-compilation support for sdist packages with native extensions - #1363
xangcastle wants to merge 57 commits into
Conversation
✨ Aspect Workflows Tasks📅 Fri Aug 14 08:43:19 UTC 2026 ✅ 44 successful tasks
⏱ Last updated Fri Aug 14 08:54:55 UTC 2026 · 📊 GitHub API quota 1,677/15,000 (11% used, resets in 45m) |
py_binary startup benchmark
sys.path quality
Bazel analysis benchmark
|
5e9c4f1 to
cd6dfa7
Compare
xangcastle
changed the title
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
Reviewing this draft at the author's explicit request.
rules_pycross already provides the native CC layer, separate execution and target Python interpreters, PEP 517 cross-build environment, and target-aware sysconfig that this change reimplements. Its v2 alpha also exposes those primitives through a public backend integration API. [0] [1] [2] [3]
The proposed implementation instead infers cross-compilation from an absent optional native toolchain, omits target native dependency and runtime closures, hard-codes macOS deployment tags, and rejects valid Windows wheels. The Linux-only fixture checks the ELF architecture and filename without proving that a cross-built native extension actually loads.
Please first evaluate integration with rules_pycross, including the v2 alpha's transitive-dependency compatibility. If that integration is not viable, document the actual blocker and propose the smallest reusable upstream interface before introducing a parallel cross-compilation implementation. [4]
[0] https://github.com/jvolkman/rules_pycross/blob/v2.0.0-alpha.2/pycross/backend.bzl#L63-L83
[1] https://github.com/jvolkman/rules_pycross/blob/74ee87c7d8eab76a673c07df3ec9e55d1e629e49/pycross/toolchain.bzl#L33-L55
[2] https://github.com/jvolkman/rules_pycross/blob/74ee87c7d8eab76a673c07df3ec9e55d1e629e49/pycross/private/build/actions/cc_layer.bzl#L126-L182
[3] https://github.com/jvolkman/rules_pycross/blob/74ee87c7d8eab76a673c07df3ec9e55d1e629e49/pycross/private/build/tools/utils/sysconfig_utils.py#L97-L169
[4] https://github.com/bazelbuild/bazel-central-registry/blob/main/modules/rules_pycross/2.0.0-alpha.2/MODULE.bazel
— tamirdex
|
|
||
| target_os, target_cpu = get_target_platform(ctx) | ||
|
|
||
| return struct( |
There was a problem hiding this comment.
This returns compiler tools and flags, but not the CcInfo headers, dependent static/shared libraries, or target C++ runtime that a real source-built extension requires. The wheel action cannot materialize that dependency closure; inspecting the toy geohash ELF cannot establish that the resulting extension links or loads. The existing pycross layer explicitly gathers all three. Please compose with that layer or explain and test the real alternative. [0]
— tamirdex
| sysconfig_file = _find_sysconfigdata(runtime) | ||
| if sysconfig_file: | ||
| extra_inputs.append(depset([sysconfig_file])) | ||
| env["RULES_PY_TARGET_SYSCONFIGDATA"] = sysconfig_file.path |
There was a problem hiding this comment.
Copying one target _sysconfigdata file into a build still executed by the host interpreter does not create a target Python environment. Build backends continue to observe host interpreter identity, packaging tags, and platform behavior; the later validation checks only the final platform segment, so a host cp313 ABI can pass for a cp312 target with the same OS and CPU. pycross carries both interpreters and builds the appropriate cross environment instead. [0]
— tamirdex
| return "linux-" + _PYTHON_CPU_MAP.get(target_cpu, target_cpu) | ||
| if target_os == "darwin": | ||
| cpu = "arm64" if target_cpu == "aarch64" else target_cpu | ||
| return "macosx-11.0-" + cpu |
There was a problem hiding this comment.
The macOS deployment version is a property of the target interpreter/SDK, not always 11.0. Hard-coding it produces an incorrect compatibility tag for targets requiring another deployment version. Derive the platform and deployment target from target sysconfig, as pycross already does, and add an actual Darwin cross-build regression. [0]
— tamirdex
| ) | ||
| exit(1) | ||
|
|
||
| if expected_cpu not in platform_tag: |
There was a problem hiding this comment.
A valid Windows x86-64 wheel has platform tag win_amd64, but _expected_cpu_in_tag returns x86_64 here. Consequently every such wheel fails this validation despite being correct; Windows x86 has the analogous win32/i686 mismatch. Conversely unrecognized target OS/CPU values silently skip validation above. Derive supported wheel tags from actual target metadata and fail closed for unsupported targets.
— tamirdex
c1e9648 to
4a95d04
Compare
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
Re-reviewing because Jason explicitly requested review of 9f3b372.
The new execution transition still selects host_platform and repository-time host libc, rather than the wheel action’s actual execution platform. The associated frontend test removes its execution-platform assertion. This cannot build correctly under heterogeneous remote execution; the new documentation itself confirms that the design assumes exec == host.
The five existing review threads remain current and unanswered. The replacement CC layer does not stage transitive CcInfo headers, native libraries or runtime dependencies; the PEP 517 frontend still runs the host interpreter with only a target sysconfig file; macOS wheel tags still hard-code deployment target 11.0; and wheel validation still rejects valid win_amd64 tags. The Linux fixture checks one patched extension’s ELF and filename but never executes the cross-built artifact or verifies a native dependency, target Python ABI, Darwin or remote worker.
rules_pycross already publicly exports target-aware CC extraction, PEP 517 actions, wheel repair and execution-platform transitions. Please evaluate composing those existing primitives before maintaining a second incomplete cross builder. If a dependency or compatibility issue prevents reuse, identify it and add real target-runtime and remote-execution coverage before calling this general cross-compilation.
https://github.com/jvolkman/rules_pycross/blob/v2.0.0-alpha.2/pycross/backend.bzl#L28-L39
https://github.com/jvolkman/rules_pycross/blob/v2.0.0-alpha.2/pycross/backend.bzl#L63-L83
— tamirdex
01fdffa to
9061079
Compare
Ports four cases from rules_pycross's e2e suite, exercising the **existing** sdist→wheel pipeline (host builds only — no cross-compilation involved). Split out of #1363, where these cases don't depend on the cross feature; every test passes against current `main` unchanged. ## Cases - **pycross-setuptools** (`build_setuptools`): three setuptools C-extension sdists — PyYAML (Cython-generated extension, with an observable `PYYAML_FORCE_LIBYAML=0` env override), setproctitle (plain C, `pre_build_patches`), zstandard (vendored libzstd, `resource_set`). - **pycross-patches** (`patches_and_hooks`): pre-build and post-install patch phases stacked on setproctitle — the post-install hunk carries the pre-build patch's output as context, so an ordering regression fails the build itself. - **pycross-pure-python** (`build_pure_python`): hatchling and flit-core backends, runtime imports, site-packages placement hygiene, plus a `collect_wheels` matrix asserting anyarch wheels stay `-none-any` under non-host platform transitions (exec-platform resolution of the build tooling, no native toolchain needed). - **pycross-distutils-probe**: build-action env hygiene — a fresh child interpreter spawned by the backend must resolve `distutils` on Python 3.12+, guarding `rule.bzl`'s `_INHERITED_PYTHON_ENV` filter. Fails only in the child, so it's invisible to a plain "does it build" check. Shared `tools/`: `collect_wheels` macro (adds the wheel-tag assertion the rules_pycross original lacks) and `check_wheel_tags.py`. ## Not included `pycross-setuptools`' cross matrix (`native_wheels*`, `check_wheel_native.py`): building a C extension for a non-host platform needs `native_build_toolchain_type` resolution for that platform, which lands with #1363. The second commit documents that scope cut. --- ### Changes are visible to end-users: no ### Test plan - New test cases added --------- Co-authored-by: Jason Bedard <jason+github@jbedard.ca>
324b8e6 to
2273078
Compare
e29dda8 to
9762970
Compare
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
…rkspace (#1431) Groundwork for the upcoming cross-compilation support, split out to slim down #1363. The rules_pycross ports (`pycross-distutils-probe`, `pycross-patches`, `pycross-pure-python`, `pycross-setuptools`) landed in `e2e/cases`, but they are sdist-crossbuild suites: their hubs carry package-specific overrides (`default_build_dependencies`, pre/post-install patches, `resource_set`) that shouldn't share a module with unrelated cases. This PR moves them into their own `e2e/crossbuild` workspace — a minimal `MODULE.bazel` (rules_py via `local_path_override`, `bazel_lib`, the LLVM toolchain, PBS interpreters 3.12/3.13, uv) — which is where the cross-build test matrix from #1363 will land next. Everything moves as pure renames; the wheel-collection tooling (`collect_wheels`, `check_wheel_tags`) comes along since `pycross-pure-python` was its last consumer under `e2e/cases`. CI gets the new workspace in the test matrix and the macOS smoke job, which previously covered these suites through `e2e/cases`. No rule code changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The four rules_pycross ports moved from e2e/cases into the crossbuild workspace, where the cross toolchains they exercise actually live: - restore the pycross-setuptools native_wheels matrix deferred out of #1413 (zstandard rebuilt for amd64/arm64 with ELF and wheel-tag assertions) — under this workspace's registered cross CC toolchain it runs where it couldn't on main - carry the reviewed refinements from #1413 over the original port - pin setproctitle to -std=gnu17: gcc_toolchain's GCC defaults to C23, where `bool` is a keyword and 1.3.2's `typedef char bool` breaks — coverage the e2e/cases host toolchain never gave us - drop the cases, their MODULE includes and the collect_wheels tooling from e2e/cases
uv-deps-650/crossbuild's test_crossbuild_pyc.sh guarded whl_install's compileall picking the exec-platform interpreter under a cross transition; its removal left that regression uncovered. The restored guard is stronger than the original "it builds" check: - the platform pins //uv/private/pyc:precompile=True, so the coverage survives a future default flip instead of silently draining - the sh_test asserts .pyc files actually exist in the cross-built install tree, catching exec_matches_target silently skipping compileall — verified to fail when precompile is off Asserted over the binary's runfiles rather than an image layer: py_image_layer deliberately drops __pycache__ from layer mtrees.
The committed lock carried lockFileVersion 28 (a bazel 9 artifact), which --lockfile_mode=error rejects under the workspace's .bazelversion.
Behavior-neutral cleanup, verified by a full from-scratch rebuild of the e2e/crossbuild suite (every action key changed with the tool): - _get_wrapper_flags no longer reimplements the sysroot absolutization that _absolutize_sysroot_flags already does; it absolutizes first and only filters. The execroot rationale now lives in one place. - A NoReturn _die() replaces nine print-to-stderr-then-exit blocks. The build-command-detection branch keeps an explicit raise because ty does not narrow NoReturn in module-level flow. - _expected_cpu_in_tag drops the identity entries from its maps. - Comments and docstrings compress to their load-bearing rationale; the module docstring now describes what the script actually is. Prose that restated the code, duplicated an in-template comment, or carried reference URLs is gone.
9762970 to
1e4e576
Compare
Split rule.bzl following the one-rule-per-file convention py/private already uses: pep517_whl.bzl (anyarch), pep517_native_whl.bzl (native + cross), common.bzl (shared helpers and base attrs), with defs.bzl as the package's single public surface. cc_layer.bzl and exec_transition.bzl were already factored by concern and stay put. The runtime tools move to tools/: build_helper.py (the main of every generated build_tool py_binary) and memory_monitor.py, each still following the package's constraints (single-file helper, conditional py_library). Mechanical fallout: the sdist_build BUILD template loads defs.bzl and points at the tools/ labels (generated-repo snapshots regenerated in the root and e2e/cases workspaces), test data labels and the dirname/TEST_SRCDIR-walking path assumptions follow the move, and the CI typecheck file lists pick up the new paths.
Two builder jobs produce docker-loadable OCI tarballs (new oci_load + tarball targets in pep517_cross_case, tagged manual) and native runner jobs docker-load and run them — no QEMU in the execution loop, so emulation can neither mask nor introduce failures: - crossbuild-images (linux/amd64 builder) uploads the arm64 tarballs; crossbuild-run-arm64 executes them on ubuntu-24.04-arm. Native amd64 execution already happens in the e2e-crossbuild test job. - crossbuild-images-darwin (gated like smoke, 10x-billed) cross-builds psutil/contourpy/rpds_py for both arches; crossbuild-run-darwin-built executes them on native amd64 + arm64. The macOS smoke job filters -requires-docker, so this is the first execution coverage the darwin-to-linux path gets. The reverse direction (arm64 builder -> amd64 execution) is documented as blocked in the workflow comment: gcc_toolchain registers exec=x86_64 only and //tools:rust_host_sysroot pins the x86_64 rust toolchain.
The native-runner jobs remove QEMU from the execution loop, but the builder still needs it during the build itself: meson's compiler sanity check and cc.run() probes execute target-arch binaries through the exe_wrapper, which relies on binfmt_misc (contourpy failed with "Executables created by cpp compiler ... are not runnable").
The smoke job and the darwin image builder invoke bazel with --config=ci in every workspace; this one lost the definition when the branch's .bazelrc superseded main's in the merge, failing with "Config value 'ci' is not defined in any .rc file".
…licitly The BCR llvm toolchain supplies its C++/unwind runtime (libc++.a, libc++abi.a, libunwind.a) as toolchain inputs via static_runtime_lib — never as link-action flags — and ships empty stub libraries on the search path, so the wrapper's name-based "-lc++ -lc++abi" (and rustc's "-lgcc_s" fold to "-lunwind") resolved against stubs. The resulting .so's linked cleanly (undefined symbols are legal in shared links) and only exploded at dlopen on the target: contourpy with 45 undefined std::__1 symbols, rpds_py with undefined _Unwind_*. First caught by the darwin-built native-execution CI jobs; gcc_toolchain never hits this because its self-contained driver resolves its own runtime. cc_layer now extracts static_runtime_lib behind the -nostdlib++ marker (so the gcc path is untouched), the rule forwards the archives as action inputs plus RULES_PY_CXX_STATIC_RUNTIME, and the cross wrapper links them as absolute paths in dependency order on every ELF link, dropping -lgcc_s. Archive semantics keep this inert for pure-C links.
Temporary instrumentation: the darwin builder's contourpy build fails in meson's sysconfig python lookup with a bare UnicodeDecodeError one-liner; MESON_FORCE_BACKTRACE names the file and line doing the read.
Both wrappers classified anything without "-c" as a link. meson probes the compiler through the wrapper with "-E -v -", "-print-search-dirs", and "--version"; the link-only additions were mostly inert there until the static runtime archives became positional inputs — "-E" preprocesses positional inputs, so libc++.a turned into megabytes of binary on stdout that meson strictly utf-8 decodes (its sysconfig python lookup died with UnicodeDecodeError on darwin hosts, where the absence of pkg-config is what routes meson into that code path). is_link now excludes -c/-E/-S/-fsyntax-only and the introspection queries in both the native and cross wrappers.
An ubuntu-24.04-arm builder cross-compiles the per-backend subset to amd64 and a native amd64 runner executes the artifacts — the reverse of the existing direction. gcc_toolchain has no arm64-exec variant, so this is also the first Linux-hosted exercise of the llvm toolchain fallback (the same path darwin hosts take). rust and the CC layer needed no changes: rust_host_sysroot re-resolves via exec_transition on any host, and rules_rust already registers both linux target triples.
The passthrough exe_wrapper assumed Linux hosts can always run target-arch ELFs via binfmt, but a dynamically-linked probe also needs QEMU_LD_PREFIX to resolve the target's ld.so — and only gcc_toolchain's layout provides that sysroot. Under the llvm fallback (empty sysroot, as exercised by the arm64 builder) meson's sanity check died with "not runnable". Gate the wrapper on RULES_PY_TARGET_GCC_SYSROOT: elsewhere meson gets needs_exe_wrapper=true with no wrapper, the same honest skip-or-explicit-error semantics darwin hosts already had.
…wnloads --config=ci turns on --remote_download_outputs=minimal, so on cache hits the tarball this job uploads never lands in bazel-bin and the cp fails; toplevel keeps the byte savings for everything else.
The builder fleet is now three-way (amd64, arm64, darwin); the unqualified name predates the other two.
Six flat jobs rendered as dependency spaghetti in the Actions graph.
Each builder direction is now one workflow_call node
(crossbuild-{amd64,arm64,darwin}) that expands to its own images->run
chain, parameterized over the four real differences: builder runner,
setup flavor, case list, and native-executor matrix. The build command
is now uniform too — every builder gets --config=ci with toplevel
output materialization and the meson traceback instrumentation.
…skip The previous needs list referenced the pre-refactor job names (twice), which is a workflow startup error — the whole CI run failed to launch. test-all now needs the three reusable-workflow nodes and requires success from everything unconditional; crossbuild-darwin may be skipped (it only runs on main and '*macos*' branches) but must pass when it ran.
Drop every container_structure_test from e2e/crossbuild: they compiled
and executed on the same host through a docker+QEMU sandwich, and the
crossbuild-verify pipelines have superseded them — each case's OCI
tarball is uploaded and `docker run` on native amd64 and arm64 runners,
which is where execution verification now lives exclusively. geohash and
zstandard gain the same {amd64,arm64}_load/_tarball targets the macro
declares, so the builder's tarball query picks them up automatically;
the amd64 pipeline now ships both arch sets (in-suite amd64 command
tests were the only native amd64 execution before).
`bazel test //...` in the workspace is structural-only (ELF arch, ABI
tags, byte-diffs) and no longer needs docker at all. The
container_structure_test dep is gone; uv_bin pin restored to 0.11.21,
the only version pinned in versions.bzl.
test-all's needs pointed at the reusable-workflow job names that the flat-jobs rework removed — a workflow startup error. It now gates on the three crossbuild-run jobs, tolerating the skip of the gated darwin one. With the in-suite container tests gone, the amd64 builder was uploading arm64 tarballs only, leaving the macro cases without native amd64 execution — it now ships both arch sets (with toplevel output materialization) and its run job fans out to native amd64 and arm64 runners, mirroring the darwin pipeline's shape.
crossbuild-images-amd64 builds only the amd64 tarballs (native builds: no cross machinery, no binfmt, no QEMU step) and a single crossbuild-run-amd64-built job executes the artifact on a separate native amd64 machine. arm64-target coverage from this host moves entirely to the other pipeline directions.
The point of this direction is empirical cross-compilation validation: build every case's arm64 image on the amd64 host (binfmt present only for meson's build-time probes) and execute the artifact on a native arm64 runner — one builder job, one runner job.
Same build-without-the-bytes failure the arm64 builder hit: with --config=ci, disk-cache hits leave the tarball unmaterialized and the cp fails. toplevel downloads fix it, as on the other builders.
b25e5ec to
25d3b44
Compare
3 participants
Enable pep517_native_whl to cross-compile Python sdists containing C or C++ extensions when the target platform differs from the exec host.
The native_build_toolchain sentinel is now optional, so a missing resolution signals cross-compilation mode and falls through to a user-registered cross CC toolchain (such as
toolchains_llvm) instead of hard-failing.A new
cc_layer.bzlmodule extracts compiler paths,CFLAGS, andLDFLAGSfrom the resolved cross CC toolchain at analysis time, and build_helper.py generates compiler wrappers that re-inject -target/--sysroot identity flags, filter incompatible linker flags from the exec host, and override LDSHARED to use the cross CC toolchain's link flags. The target interpreter's_sysconfigdatais loaded via_PYTHON_SYSCONFIGDATA_NAMEso setuptools produces correctEXT_SUFFIXandSOABItags, and a post-build check validates the wheel's platform tag against the target before accepting the output.Changes are visible to end-users: yes/no
Test plan