x build fails copying files into stage0-sysroot using local rust

[pmur]

Summary

When building rust with a local toolchain, boostrap fails with errors like:

thread 'main' panicked at src/bootstrap/src/lib.rs:1729:17:
failed to copy `/usr/lib/cups/backend/gutenprint53+usb` to `/home/paumurph/git/rust-dev/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/cups/backend/gutenprint53+usb`: Permission denied (os error 13)

Command used

./configure --local-rust-root=/usr --llvm-root=/usr
RUST_BACKTRACE=1 ./x --stage=1 build sysroot

Expected behaviour

The x command succeeds

Actual behaviour

bootstrap attempts to copy the contents /usr/lib and fails when it encounters files with unexpected permissions.

Bootstrap configuration (bootstrap.toml)

See configure command above.

Operating system

Fedora 41

HEAD

On the beta branch, based off b5e10d8c000a153cae6a14d70372dc4b39439a3f

Additional context

Build Log

   0: __rustc::rust_begin_unwind
             at /builddir/build/BUILD/rust-1.88.0-build/rustc-1.88.0-src/library/std/src/panicking.rs:697:5
   1: core::panicking::panic_fmt
             at /builddir/build/BUILD/rust-1.88.0-build/rustc-1.88.0-src/library/core/src/panicking.rs:75:14
   2: bootstrap::Build::copy_link_internal
             at ./src/bootstrap/src/lib.rs:1729:17
   3: bootstrap::Build::copy_link
             at ./src/bootstrap/src/lib.rs:1682:9
   4: bootstrap::Build::cp_link_r
             at ./src/bootstrap/src/lib.rs:1757:17
   5: bootstrap::Build::cp_link_r
             at ./src/bootstrap/src/lib.rs:1755:17
   6: bootstrap::Build::cp_link_r
             at ./src/bootstrap/src/lib.rs:1755:17
   7: <bootstrap::core::build_steps::compile::StdLink as bootstrap::core::builder::Step>::run
             at ./src/bootstrap/src/core/build_steps/compile.rs:814:13
   8: bootstrap::core::builder::Builder::ensure
             at ./src/bootstrap/src/core/builder/mod.rs:1596:23
   9: <bootstrap::core::build_steps::compile::Std as bootstrap::core::builder::Step>::run
             at ./src/bootstrap/src/core/build_steps/compile.rs:156:13
  10: bootstrap::core::builder::Builder::ensure
             at ./src/bootstrap/src/core/builder/mod.rs:1596:23
  11: <bootstrap::core::build_steps::compile::Rustc as bootstrap::core::builder::Step>::run
             at ./src/bootstrap/src/core/build_steps/compile.rs:1074:9
  12: bootstrap::core::builder::Builder::ensure
             at ./src/bootstrap/src/core/builder/mod.rs:1596:23
  13: <bootstrap::core::build_steps::compile::Assemble as bootstrap::core::builder::Step>::run
             at ./src/bootstrap/src/core/build_steps/compile.rs:2155:28
  14: bootstrap::core::builder::Builder::ensure
             at ./src/bootstrap/src/core/builder/mod.rs:1596:23
  15: bootstrap::core::builder::Builder::compiler
             at ./src/bootstrap/src/core/builder/mod.rs:1298:9
  16: <bootstrap::core::build_steps::compile::Std as bootstrap::core::builder::Step>::make_run
             at ./src/bootstrap/src/core/build_steps/compile.rs:123:23
  17: bootstrap::core::builder::StepDescription::maybe_run
             at ./src/bootstrap/src/core/builder/mod.rs:475:13
  18: bootstrap::core::builder::StepDescription::run
             at ./src/bootstrap/src/core/builder/mod.rs:607:17
  19: bootstrap::core::builder::Builder::run_step_descriptions
             at ./src/bootstrap/src/core/builder/mod.rs:1271:9
  20: bootstrap::core::builder::Builder::execute_cli
             at ./src/bootstrap/src/core/builder/mod.rs:1251:9
  21: bootstrap::Build::build
             at ./src/bootstrap/src/lib.rs:705:17
  22: bootstrap::main
             at ./src/bootstrap/src/bin/main.rs:99:5
  23: core::ops::function::FnOnce::call_once
             at /builddir/build/BUILD/rust-1.88.0-build/rustc-1.88.0-src/library/core/src/ops/function.rs:250:5

My first attempt at working around was the oneliner, though I am not sure this is correct:

--- a/src/bootstrap/src/core/build_steps/compile.rs
+++ b/src/bootstrap/src/core/build_steps/compile.rs
@@ -811,7 +811,7 @@ fn run(self, builder: &Builder<'_>) {
                 let _ = fs::remove_dir_all(sysroot.join("lib/rustlib/src/rust"));
             }
 
-            builder.cp_link_r(&builder.initial_sysroot.join("lib"), &sysroot.join("lib"));
+            builder.cp_link_r(&builder.initial_sysroot.join("lib/rustlib"), &sysroot.join("lib/rustlib"));
         } else {
             if builder.download_rustc() {
                 // Ensure there are no CI-rustc std artifacts.

[Kobzol]

Hi, what does /usr/bin/rustc --print sysroot output?

[pmur]

Hi, what does /usr/bin/rustc --print sysroot output?

$ /usr/bin/rustc --print sysroot
/usr

[cuviper]

ISTR bootstrap used to forgo all copying for externally-provided stage0 like this.

[Kobzol]

We copy the stage0 sysroot to the stage0-sysroot directory, because during the build, we actually need to copy stuff (rlibs/rmetas) into that directory. Normally, the sysroot is a self-contained directory with Rust-specific files, but in your case it's essentially just /usr. We don't really know how to identify all the relevant files in the sysroot, so we just copy everything, which is a problem for your use-case. If we didn't do this copying, then we would either start putting things into /usr (which is even more problematic), or it won't be possible to do a check/build of the stage1 compiler.

We could maybe hotfix this by ignoring files with bad permissions, but I'm not sure if I like that hack. I'm inclined to say that this is a wontfix, and that if you want to use a custom external rustc, it should have a self-contained sysroot, same as what you get with rustup.

[cuviper]

We copy the stage0 sysroot to the stage0-sysroot directory, because during the build, we actually need to copy stuff (rlibs/rmetas) into that directory.

I assume that's copying newly built stuff, but what exactly? Why is this different than ever before?

if you want to use a custom external rustc, it should have a self-contained sysroot

That's more difficult to arrange in a distro context, so I'd like to figure out how we can continue supporting what used to work fine. (And of course, we can contribute patches to that end, once I understand what's going on...)

[Kobzol]

I assume that's copying newly built stuff, but what exactly?

When you check/build a stage 1 rustc using stage 0 rustc, the rmetas/rlibs of the stage 1 rustc get copied into the stage 0 sysroot. This is required in order to check/build stage 1 rustc tools (miri, rust-analyzer, rustfmt, clippy, etc.). Currently it happens unconditionally, so even if you don't build these tools, these artifacts will be copied into the stage 0 sysroot.

I can't say that I love this behavior, but apparently rustc can't/doesn't support multiple sysroots, so it currently kind of has to work like this, at least from the rustc side.

There are probably some ways to improve this situation though:

• Only copy rlibs/rmetas when we have to build rustc tools and not unconditionally. This is something that I want to eventually do, but it's likely a very non-trivial refactoring of bootstrap (which is currently heavily under-maintained and generally in a pretty bad shape overall). This wouldn't help if you want to build e.g. miri/clippy/... ofc.
• Don't perform the copy if the stage0 rustc is externally provided and not beta. Then of course building rustc tools like miri/clippy won't work, but at least rustc/std hopefully should.
• Figure out a way of copying only the relevant stuff from the stage 0 sysroot to the stage0-sysroot directory. Maybe we can just hardcode something like "copy lib/rustlib, bin/<looks like librustc_driver/looks like LLVM>". Sounds quite hacky...

Why is this different than ever before?

It should have behaved like this since forever, I think? It's possible that bootstrap had some bugs that hid this. Also the stage 0 redesign might have changed behavior here, the build/check staging was quite wild, I'm now slowly trying to consolidate it.

[cuviper]

Why is this different than ever before?

It should have behaved like this since forever, I think? It's possible that bootstrap had some bugs that hid this. Also the stage 0 redesign might have changed behavior here, the build/check staging was quite wild, I'm now slowly trying to consolidate it.

I believe it's new from the redesign, added here:
5dc0d89#diff-5a1e05f2688d271039171a547d407d0c8a96715ee64d35562fc76b4c9a874303R797

Before, it only copied stuff if "not set to a custom path".

• Only copy rlibs/rmetas when we have to build rustc tools and not unconditionally. This is something that I want to eventually do, but it's likely a very non-trivial refactoring of bootstrap (which is currently heavily under-maintained and generally in a pretty bad shape overall). This wouldn't help if you want to build e.g. miri/clippy/... ofc.

• Don't perform the copy if the stage0 rustc is externally provided and not beta. Then of course building rustc tools like miri/clippy won't work, but at least rustc/std hopefully should.

We don't need miri since that's a nightly-only thing, but we do ship most other tools including clippy. Even then, I think those should be stage2 builds for dist, so they probably still don't need compiler libs in stage0-sysroot, right?

• Figure out a way of copying only the relevant stuff from the stage 0 sysroot to the stage0-sysroot directory. Maybe we can just hardcode something like "copy lib/rustlib, bin/<looks like librustc_driver/looks like LLVM>". Sounds quite hacky...

I suspect lib/rustlib might suffice -- it's only copying lib right now, not any bin at all. For Linux, that would include lib/{librustc_driver,libLLVM} too, but we have those elsewhere in /usr/lib64 anyway. I imagine most distros are similar, or else have rpaths to find them relative to the uncopied bin anyway.

[Kobzol]

We don't need miri since that's a nightly-only thing, but we do ship most other tools including clippy. Even then, I think those should be stage2 builds for dist, so they probably still don't need compiler libs in stage0-sysroot, right?

If you do a stage 2 rustfmt build, only the stage1 rlibs/rmetas have to be available, so we don't need the compiler libs in stage0-sysroot, indeed. I guess that this could be the simplest fix, that if you do a stage N tool build, then only copy the rlibs to the N-1 rustc sysroot, and don't do this for N-2. (I am excited to see what other use-cases this will break =D)

[cuviper]

I can't say that I love this behavior, but apparently rustc can't/doesn't support multiple sysroots, so it currently kind of has to work like this, at least from the rustc side.

On second thought, why do the compiler libs need to be in the sysroot? Couldn't we provide them with -L?
(That is a way to get "multiple sysroots" in some sense.)

[Kobzol]

Let's ask someone who knows more about this :D (https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Proposal.20to.20cleanup.20stages.20and.20steps.20after.20the.20redesign/near/527359940)

[heftig]

I had to solve another problem while working around this:

The compiler prefers lib64 over lib, so if a symlink /usr/lib64 -> lib exists, lib64 also shows up in rustc --print target-libdir.

Bootstrap will create an empty lib64/rustlib/<target>/lib/ in the sysroot but copy the crates to lib/rustlib/<target>/lib/.

The compiler then finds the former dir and doesn't look for the latter, so it can't find the crates.

#t-infra/bootstrap > ✔ 1.89.0 bootstrap attempts to copy entire /usr/lib