Fix conda builds for packages requiring __cuda virtual package

[pditommaso]

Summary

• Fixes conda-file and conda-packages builds that fail when a dependency requires the __cuda virtual package (e.g., pytorch-gpu >= 1.12.1, JAX, TensorFlow)
• Adds retry-on-failure logic to all 4 micromamba v2 templates (Docker/Singularity x file/packages): the first install attempt output is captured, and if it fails with __cuda in the error, retries with CONDA_OVERRIDE_CUDA="12" set
• Zero overhead for non-GPU builds; GPU builds incur only a fast solver failure (~4s) before the successful retry

Closes #1026

Test plan

• All 34 TemplateUtilsTest tests pass
• Verify with a real conda-file build containing pytorch-gpu>=2.0
• Verify non-GPU builds (e.g., bwa salmon) are unaffected

[pinin4fjords]

Minor suggestion: consider using a higher sentinel value than "12" for CONDA_OVERRIDE_CUDA. The value just needs to satisfy __cuda >=N constraints - the solver picks the best available CUDA build from the channels regardless. "12" works today but would fail if a future package requires __cuda >=13. A value like "99" is equally correct and future-proof:

$ CONDA_OVERRIDE_CUDA=99 micromamba create --dry-run -f env_gpu.yml
  + cuda-version  13.2    # solver still picks the best available
  + pytorch-gpu   2.10.0  cuda129_mkl_...

The value is passed through as a raw string with no validation (virtual_packages.cpp).

[munishchouhan]

Shouldn't we add it for micromamba-v1 also?

[adamrtalbot]

Does this hard code CUDA 12? CUDA has an annoying dependency matrix so hard coding feels risky (I need 11! I need 13!).

We can also introspect from the Conda packages but this is more involved: https://anaconda.org/nvidia/cuda

You can prove it works by building the Alphafold container: https://github.com/google-deepmind/alphafold/blob/main/docker/Dockerfile 😉

[pditommaso]

Uff, you are never happy folks :D

[pinin4fjords]

Does this hard code CUDA 12? CUDA has an annoying dependency matrix so hard coding feels risky (I need 11! I need 13!).

@adamrtalbot The "12" doesn't hardcode CUDA 12 in the container - it just tells the solver "assume CUDA >= 12 is available." The solver then picks the best CUDA build from the channels independently (we got cuda-version 12.9 with pytorch-gpu 2.10.0). If someone's environment needs CUDA 11.8, the solver will select it as long as the packages support it.

That said, using "99" instead of "12" avoids any ambiguity and future-proofs against __cuda >=13 constraints.

[pditommaso]

Could a fair starting point? we could add a cuda specific option later to specific it

[munishchouhan]

Shouldn't we add it for micromamba-v1 also?

the issue is reported on the default template which is micromamba-v1
https://wave.seqera.io/view/builds/bd-bbf66b1b68ac0df5_1

So we also need to make changes to the default template

[pditommaso]

We are going to switch to v2 soon as default #1024

[pinin4fjords]

Could a fair starting point? we could add a cuda specific option later to specific it

Agreed, this is a great starting point. The two-pass logic is the important part - it unblocks all current GPU builds with zero impact on non-GPU builds. I don't think a user-facing --cuda flag would add much value in practice - the override only gates which builds the solver considers eligible, it doesn't pin anything in the container. If someone needs a specific CUDA version, they'd pin cuda-version=X in their environment YAML directly.

One thing worth noting: the sentinel value does affect the solve. With CONDA_OVERRIDE_CUDA=11.8, the solver excludes CUDA 12+ builds entirely and picks pytorch-gpu 2.5.1 instead of 2.10.0. Using a high value like 99 means "let the solver pick the best available," which is the right default for container builds where you don't know the runtime host's driver version.

[pinin4fjords]

We are going to switch to v2 soon as default #1024

I'd love to be able to get good builds ASAP though :-)

[munishchouhan]

We are going to switch to v2 soon as default #1024

But we are still not removing micromamba-v1, so worth to add the fix there too

[pditommaso]

@pinin4fjords so you are ok to merge or you want CONDA_OVERRIDE_CUDA=11.8 ?

[pinin4fjords]

@pinin4fjords so you are ok to merge or you want CONDA_OVERRIDE_CUDA=11.8 ?

I'd suggest CONDA_OVERRIDE_CUDA="99" rather than 11.8 or 12. It's clearly not a real CUDA version, so nobody will mistake it for a pin, and it's high enough that it'll satisfy any future __cuda >=N constraint without needing to be updated. The solver picks the best available build from the channels regardless of this value.

Also +1 to @munishchouhan's point about adding the fix to the v1 templates too - looks like they're not included in the PR yet.

[pditommaso]

Done, let's test in v2 before applying in v1 template

[pditommaso]

Can you please review and approve ?

[pinin4fjords]

Looks good. The two-pass logic is correct, 99 is the right sentinel value, and testing in v2 first before applying to v1 makes sense. Thanks for the quick turnaround on this.