Skip to content

pyspark cross-platform build is broken due to target python toolchain selection #138

Description

@azryve

Hi Michael.

I encountered an issue with cross-platform builds seemingly introduced in #123.

But since it was the exactly a point of the MR I am not sure if cross-platform builds were supposed to be supported in first place.

Still I decided to share still, if you would not consider it a regression feel free to close it.

Environment

bazel: 9.1.1
ofiuco: 0.9.2
python: 3.14
host platform: linux-amd64
target platform: linux-aarch64

Behaviour

I encountered this specifically with pyspark build so I used it for a repro.

Building for a native platform works fine

$ bazel build @poetry//:pyspark --platforms=@zig_sdk//platform:linux_amd64
INFO: Analyzed target @@ofiuco++parse+poetry//:pyspark (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target @@ofiuco++parse+poetry//:pyspark up-to-date:
  bazel-bin/external/ofiuco++parse+poetry/3.14/x86_64-unknown-linux-gnu/pyspark
  bazel-bin/external/ofiuco++parse+poetry/3.14/x86_64-unknown-linux-gnu/.dist-info/pyspark/entry_points.txt
INFO: Elapsed time: 2.025s, Critical Path: 0.09s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

Building for a foreign platform breaks:

$ bazel build @poetry//:pyspark --platforms=@zig_sdk//platform:linux_aarch64
WARNING: Build option --platforms has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: Analyzed target @@ofiuco++parse+poetry//:pyspark (0 packages loaded, 1169 targets configured).
ERROR: /home/azryve/.cache/bazel/_bazel_azryve/e97c2d0c9dd84381e200fb8ac0da62c9/external/ofiuco++parse+poetry/BUILD.bazel:16:8: Installing package pyspark for Python 3.14 aarch64-unknown-linux-gnu failed: (Exit 1): python3 failed: error executing InstallWheel command (from package rule target @@ofiuco++parse+poetry//:pyspark) external/rules_python++python+python_3_14_aarch64-unknown-linux-gnu/bin/python3 -B external/ofiuco+/python/private/package_deps.py install external/ofiuco++parse+pyspark-4.1.2 ... (remaining 14 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src/main/tools/linux-sandbox-pid1.cc:542: "execvp(external/rules_python++python+python_3_14_aarch64-unknown-linux-gnu/bin/python3, 0x47f550)": No such file or directory
Target @@ofiuco++parse+poetry//:pyspark failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 4.963s, Critical Path: 1.59s
INFO: 2 processes: 4 action cache hit, 2 internal.
ERROR: Build did NOT complete successfull

Expected behaviour

Reverting the change fixes the build:

 bazel build @poetry//:pyspark --platforms=@zig_sdk//platform:linux_aarch64
INFO: Analyzed target @@ofiuco++parse+poetry//:pyspark (3 packages loaded, 1893 targets configured).
INFO: Found 1 target...
Target @@ofiuco++parse+poetry//:pyspark up-to-date:
  bazel-bin/external/ofiuco++parse+poetry/3.14/aarch64-unknown-linux-gnu/pyspark
  bazel-bin/external/ofiuco++parse+poetry/3.14/aarch64-unknown-linux-gnu/.dist-info/pyspark/entry_points.txt
INFO: Elapsed time: 44.107s, Critical Path: 38.24s
INFO: 2 processes: 2 action cache hit, 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions