Improve Zig cache handling

[aherrmann]

As of #253 the Zig cache is stored in an absolute path that is not managed by Bazel and is persisted between build actions and Bazel builds. This requires dedicated Bazel configuration (--sandbox_add_mount_pair) to pass the path through the sandbox. This is not strictly hermetic as the previous state of the cache can impact a following build, and as the cache may be persisted across Zig SDK updates or other actions that may trigger build failures due to incompatible changes in the cache treatment.

Consider alternative options to manage the Zig cache, e.g. uber/hermetic_cc_toolchain#83, or a persistent worker as described in #87. Take local builds, CI builds, and remote execution builds, as well as Zig SDK, Bazel, and rule set version updates into account.

[cerisier]

FYI, I spent quite some time trying to circumvent the problem using normal means but failed everytime because of sandboxing. No exception for the option depicted by uber/hermetic_cc_toolchain#83 repo

I'm currently exploring a launcher and a persistent worker. I have cycles on that if you want to exchange ideas.

With a persistent worker, my understanding is that we would only be able to have a shared cache for invocations done by this worker. I have also tried resolving the execroot workspace folder and using bazel-out to have a shared worked cache dir. It works successfully because worker sandbox is symlink forest.

So my findings is that is that worker allows either:

• Sharing zig-cache per worker
• Sharing zig-cache across workers with a bit of a hack

The good points are:

• zig-cache becomes per project and not global to the system (I found that a huge /tmp/zig-cache has bad implications on build times)
• zig-cache can become suffixed by toolchain version to avoid collisions
• zig-cache is bazel clean-able

The bad points are:

• We are writing a worker whose only perk is to allow non global, non mounted shared zig-cache