Clean up stale split debuginfo in incremental builds

[orlp]

Fixes #161824.

LLM disclosure: I heavily used LLMs to navigate the codebase and suggest prototype solutions, but every line of code in the end result is written by me.

For the reviewer: there is one snag I don't fully grasp. This PR changes the WorkProduct struct which I believe is serialized / deserialized. Is some sort of migration necessary so old WorkProducts deserialize correctly?

[rustbot]

r? @mu001999

rustbot has assigned @mu001999.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: compiler, incremental
• compiler, incremental expanded to 76 candidates
• Random selection from 22 candidates

[orlp]

r? @bjorn3

(Since you responded on the issue.)

[bjorn3]

Is some sort of migration necessary so old WorkProducts deserialize correctly?

Rustc never reads incr comp caches from different rustc versions.

[bjorn3]

I feel like it would be better if we instead put all debuginfo in a new crate specific directory inside the output directory and then just clear out this directory right before codegen (or after codegen remove all files not listed as dwarf_object in the CompiledModules). That way stale debuginfo can also be removed for non-incremental builds when the amount/names of codegen units changes and we don't need to persistently track this info. If anything I would like to remove WorkProduct entirely to ensure we can't accidentally forget to track a work product in the incr comp session. Instead just build a new incr comp session dir from scratch every time (#159287) and copy the latest version of a work product into it at a location derived from the cgu name.

[orlp]

@bjorn3 That sounds like it could work as well. Perhaps that PR needs a new reviewer or a gentle ping? I don't think it has seen any review work since its conception in July.

[bjorn3]

That PR is blocked on me resolving some review comments that @nnethercote passed me in private after asking if was fine with an LLM review. I resolved most already, but one is a bit more tricky. (fcntl(F_SETLK) was a mistake. Unix should have used flock() from the start.) I just need to find some time to finish it.

That PR shouldn't be necessary for moving unpacked debuginfo into a separate directory.

[orlp]

That PR shouldn't be necessary for moving unpacked debuginfo into a separate directory.

I don't understand this. As far as I understand the debuginfo files need to be right next to the object files otherwise the debugger can't find them. So either you move everything or nothing.

You could probably move them and update the paths with some kind of arguments to the linker, but that's also more involved.

[bjorn3]

As far as I understand the debuginfo files need to be right next to the object files otherwise the debugger can't find them.

You could also move the object files into this dir if necessary. They are removed after linking anyway (unless -Csave-temps is used, but that is only for debugging anyway). In any case my understanding is that the object files embed a path to the separate debuginfo files which can be either relative or absolute. So I think just adjusting the path where LLVM should write the dwo file is enough to make it include the correct relative path in the object file. Adjusting temp_path_dwo_for_cgu (+ creating the debuginfo directory first) may be enough.

You could probably move them and update the paths with some kind of arguments to the linker, but that's also more involved.

The linker knows nothing about debuginfo. It just copies .debug_* sections into the output executable by default.