Fix CI failures - #2076
Merged
Merged
Conversation
The SuperBuild nests ninja inside ninja with no job limit at either
level, so a 4-core runner ends up with ~30 concurrent compilers. On
ubuntu-24.04-arm (16 GB) this exhausted memory mid-build and the job
died with "The runner has received a shutdown signal" and exit 143,
with no compiler error in the log.
Replicated on a 4-CPU pin inside a 16 GiB + 4 GiB swap cgroup:
uncapped 583 s anon peak 15.9 GiB + 4 GiB swap, OOM-killed on the
second attempt (19 cc1plus, largest 6.3 GiB)
level 3 584 s anon peak 11.8 GiB, no swap
level 2 634 s anon peak 11.8 GiB, no swap
Wall time is unchanged at 3 because the build's tail is serial
(PoissonRecon is one translation unit; OpenSfM sets its own
--parallel from nproc). CMAKE_BUILD_PARALLEL_LEVEL is read by
cmake --build at both levels, so 3 bounds the total at 9 jobs.
setup-pixi caches the installed environments by default. The gpu environment carries the CUDA toolkit, so the cache entry is 7.4 GB against a 10 GB per-repository allowance shared with the four Native Builds caches of about 1 GB each. Over the allowance GitHub evicts the least recently used entry, so the caches evict each other and the publish job, which runs a few times a month, gains nothing from it.
With the default fail-fast, one failing platform cancels the other three, so a PR that breaks only the Windows build reports nothing about Linux, macOS or arm until the Windows failure is fixed. Each platform now runs to completion and reports on its own.
cuda-toolkit depends on cuda-nvcc, and every win-64 build of that package
on conda-forge depends on vs2019_win-64. The gpu environment therefore
carried two Visual Studio activation scripts, and each one runs vcvars and
prepends the full MSVC tree to PATH. Together with the repository's own
VsDevCmd call, the activated PATH on the windows-2022 runner exceeded the
8191 bytes cmd.exe accepts. cmd.exe then ignores PATH entirely, so when
nvcc shells out to cl.exe during CMake's CUDA compiler identification, the
OpenMVS configure step fails with:
'cl.exe' is not recognized as an internal or external command
Every Publish Windows Setup run since the pixi migration failed this way.
On win-64, depend on cuda-nvcc_win-64 and cuda-libraries-dev instead of
cuda-toolkit. cuda-nvcc_win-64 is the compiler package without the
vs2019_win-64 pin; cuda-libraries-dev is the same header and library set
cuda-toolkit provided. The profilers, nvml headers and command-line tools
that cuda-toolkit also pulled in are not referenced by the SuperBuild and
are dropped. linux-64 keeps cuda-toolkit unchanged.
The lock diff removes 19 win-64 packages from the gpu environment and adds
none. Reproduced on a Windows 11 machine under the real pixi activation:
with the base PATH padded to the runner's length the CUDA compiler check
fails on master and passes with this change.
The vs2022_win-64 package in the build environments runs vcvars64.bat from its own activation script before this file runs. The VsDevCmd.bat call here then set up the same toolset a second time, and each run prepends the full MSVC and Windows SDK tree to PATH. On the windows-2022 runner the result exceeded the 8191 bytes cmd.exe accepts, which is what made nvcc unable to find cl.exe during the OpenMVS configure step. The script now checks that the package activation set VCToolsInstallDir and fails with the same message as before if it did not. Environments that carry no compiler package (prod, gpu-prod) exit at the top; previously the vswhere lookup failed there on every activation and the error was swallowed. Verified on a Windows 11 machine with the gpu environment from this branch: the activated INCLUDE, LIB and LIBPATH lose only their duplicated SDK entries, PATH shrinks from 3003 to 2262 bytes with no distinct directory lost, the full SuperBuild completes with OpenMVS linked against CUDA, and the prod, gpu-prod and default environments activate and pass the smoke test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes three different failures in the current CI system:
Publish Windows Setup (Windows GPU)
https://github.com/OpenDroneMap/ODM/actions/runs/34355323708/job/102478410205#step:5:2153
The failure was immediately caused by the
PATHvar being too long exceeding the windows limit for 8191 bytes. The root cause was our dependencies including two versions of MSVC and adding both to the path. We also added more to the path inactivate-msvc.batwhich I cleaned up. The path should now be well under the limit. I tested this on my personal PC.As a bonus I also removed the cache here which was using up 11GB of space without providing much benefit.
Native builds (arm)
https://github.com/OpenDroneMap/ODM/actions/runs/33495602391/job/99816940025
This one is harder to see directly, but I could confirm by rerunning the build locally. The memory usage during compilation exceeded the runner limits. We can reduce the max memory usage by limiting the allowed parallelism in cmake. I tested a few values and 3 seemed the best. This capped memory at <12GB while not affecting build time.
Native builds
https://github.com/OpenDroneMap/ODM/actions/runs/33495602391
When one build failed, the rest were stopped. This was deliberately in place to reduce usage and avoid contention especially while the CI was unreliable. We can flip this flag so all builds finish and revisit later if we hit issues.