Rewrite init in Rust

[mtjhrc]

Currently our custom init is written in C, I believe it it would ease further development to rewrite it in Rust.

Unanswered questions we need to investigate:

• do we still keep statically linking it? - probably yes
  • is musl libc good enough for us?
  • alternatively is there really no way to link against glibc statically?
  • alternatively can we just go no-std, no-libc route via https://github.com/bytecodealliance/rustix
    • similar to nix crate but can make raw syscalls
    • this should also shrink the init binary size, not sure if we care that much though
• how is Rust support for FreeBSD guests?
  • will we make sure upstream libkrun init supports FreeBSD?

[mtjhrc]

Assigned to @jakecorrenti since I believe he started working on this

[nohajc]

I can answer the FreeBSD part. The aarch64 variant is a tier 3 target which makes it more difficult to cross-compile.

https://doc.rust-lang.org/nightly/rustc/platform-support.html#tier-3

Currently, you need Rust nightly to make it somewhat reasonable.

Example: https://github.com/nohajc/anylinuxfs/blob/main/build-app.sh#L39-L45

[nohajc]

Just want to mention there is a language with arguably better cross-compilation story, excellent C interop and producing lean binaries. Zig.

Not sure if the goal here is to just replace C with something easier to write or have everything in one language.

Anyway, I was actually considering Zig for the guest part of FreeBSD tests so I can avoid the Rust nightly issue. Although to be fair, Zig hasn't reached 1.0 yet, so it might not be considered stable either.

[mtjhrc]

Not sure if the goal here is to just replace C with something easier to write or have everything in one language.

For me "something easier to write" is the main goal, "have everything in one language" is also nice, but more like a secondary goal. Using something like Zig introduces a huge dependency for us, which I'd like to avoid. I've never used Zig, but it looked cool (less safety oriented than Rust though...)

[nohajc]

It's somewhere between C and Rust safety-wise. I don't have much experience with it myself. It just crossed my mind.

It's probably still suitable for early adopters only. If you're looking for stable language editions there's no such thing currently.

Although there are projects which use it as a C cross-toolchain instead of gcc or llvm. It can serve as a really smart wrapper for clang/llvm. It has some pretty nice tricks up its sleeve like bundling all existing libcs for any platform you can think of. So you can do simpler builds without any sysroot for example.

[nohajc]

I can answer the FreeBSD part. The aarch64 variant is a tier 3 target which makes it more difficult to cross-compile.

https://doc.rust-lang.org/nightly/rustc/platform-support.html#tier-3

Oh, actually it seems this is about to change soon for the better:
rust-lang/compiler-team#961

[d-e-s-o]

alternatively is there really no way to link against glibc statically?

You can via something like RUSTFLAGS='-C target-feature=+crt-static' cargo build --target x86_64-unknown-linux-gnu. However, the general guidance on linking glibc statically applies (but I guess you may already do that so nothing should change in this respect?).

FWIW, for my own project I started off with a Rust init binary, but ultimately sadly was forced to scratch it. The problem I had was two fold:

1. in order to make the init binary available on the file system I needed to build it from a build script. If you just invoke cc (as you do now) or rustc that's fine. But the moment you have a Rust dependency you need a Cargo project and Cargo doesn't like recursive invocations (it may just deadlock). That may or may not be solved with upstream output directory locking changes, but I am not sure it's a supported workflow regardless (and in any event those changes are not yet that widespread, I think).
   I somehow couldn't come up with a clean solution that circumvents build script usage (and, hence, this problem), but I may just have missed something.
2. the resulting Rust binary is comparatively huge. I don't have concrete numbers at hand, but even with a good chunk of the usual tips and tricks applied, I still recall it being significantly larger, to the degree that I felt uncomfortable with the size and stuck it out with C instead (though 1. was the bigger sticking point, I believe). It's sad...

[nohajc]

@d-e-s-o What's your project? :)

[d-e-s-o]

@d-e-s-o What's your project? :)

https://github.com/d-e-s-o/vmsh

[d-e-s-o]

in order to make the init binary available on the file system I needed to build it from a build script.

I just checked and I don't think the new build-dir layout changes anything -- still seeing a deadlock in my testing. An alternative path would be https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#artifact-dependencies, but it's unstable and so a no-go from my point of view.

Edit: Of course you can also always fudge a Makefile on top and funnel everything through there, but for me that's a hard pass.

Edit2: Figured out the deadlock thing. It was due to a shared build-dir configuration I had set. It's not standard and so wouldn't affect everybody, but still not great and there may be more dragons.

[jakecorrenti]

alternatively is there really no way to link against glibc statically?

You can via something like RUSTFLAGS='-C target-feature=+crt-static' cargo build --target x86_64-unknown-linux-gnu. However, the general guidance on linking glibc statically applies (but I guess you may already do that so nothing should change in this respect?).

FWIW, for my own project I started off with a Rust init binary, but ultimately sadly was forced to scratch it. The problem I had was two fold:

1. in order to make the init binary available on the file system I needed to build it from a build script. If you just [invoke `cc`](https://github.com/containers/libkrun/blob/a3b7ae213195c9f871a17c72f0d020e46ed90584/src/devices/build.rs#L32) (as you do now) or `rustc` that's fine. But the moment you have a Rust dependency you need a Cargo project and  Cargo doesn't like recursive invocations (it may just deadlock). That may or may not be solved with upstream output directory locking changes, but I am not sure it's a supported workflow regardless (and in any event those changes are not yet that widespread, I think).
   I somehow couldn't come up with a clean solution that circumvents build script usage (and, hence, this problem), but I may just have missed something.

2. the resulting Rust binary is comparatively _huge_. I don't have concrete numbers at hand, but even with a good chunk of [the usual tips and tricks](https://github.com/johnthagen/min-sized-rust) applied, I still recall it being significantly larger, to the degree that I felt uncomfortable with the size and stuck it out with C instead (though 1. was the bigger sticking point, I believe). It's sad...

I'm still in the very early stages of development, so this might be too early to say for sure... However, I have an init binary written in rust that works with the basic BLK=1 NET=1 functionality. The binary is 363k versus the 1M of the C version.

[nohajc]

1M? When I cross-compile on macOS, it's 734K. Still surprised the Rust version is smaller. Nice.