makeflow blindly starts building `nproc` independet package in parallel

[smuzaffar]

I tested bits build with --makeflow for 20 independent dummy packages (created using the auto generation of package recipe feature). As expected , makeflow just blindly start building 16 packages (on a 16 core VM) in parallel. For now these dummy packages were only doing sleep 60 but if these were actual packages (e.g. root, geant4, tensorflow, onnxruntime, pytorch, blackhat, openloops, rivet, herwig7 which are all independent packages) and each package use make/ninja -j $(nproc) then that will kill build node

[a]

1840697 15:41                  \_ /bin/bash -e /build/muz/bits-works/bits/bits build -c cms.bits --makeflow dummy-pkg-all
1840700 15:41                      \_ python3 /build/muz/bits-works/bits/bitsBuild build -c cms.bits --makeflow dummy-pkg-all
1840775 15:41                          \_ /bin/sh -c (cd /build/muz/bits-works/sw/BUILD/f400a0c097e7a1e55adcbcc77741153e1ce279c1/makeflow; makeflow --clean; makeflow)
1840776 15:41                              \_ /bin/sh -c (cd /build/muz/bits-works/sw/BUILD/f400a0c097e7a1e55adcbcc77741153e1ce279c1/makeflow; makeflow --clean; makeflow)
1840778 15:41                                  \_ makeflow
1840830 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='22825354c00524a8613cccc390444f79c4690b2a' P
1840839 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg11/1.0-local1/build.sh
1841184 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg11/1.0-local1/build.sh
1841195 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841186 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/22825354c00524a8613cccc390444f79c4690b2a/log
1840831 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='6bc84eaa07f027714791c56c65fdc430281714b0' P
1840844 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg10/1.0-local1/build.sh
1841096 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg10/1.0-local1/build.sh
1841117 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841098 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/6bc84eaa07f027714791c56c65fdc430281714b0/log
1840832 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='d7d19cbe8ee94bc289f1236a2fc04297d175b4de' P
1840842 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg14/1.0-local1/build.sh
1841133 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg14/1.0-local1/build.sh
1841158 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841135 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/d7d19cbe8ee94bc289f1236a2fc04297d175b4de/log
1840833 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='37bc2861258a7ea4a4e62ad1f785ffb1eb19d40a' P
1840846 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg3/1.0-local1/build.sh
1841153 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg3/1.0-local1/build.sh
1841175 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841154 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/37bc2861258a7ea4a4e62ad1f785ffb1eb19d40a/log
1840834 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='d5f23e384c63ba7bc07bf6880837ebe907bbe075' P
1840848 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg5/1.0-local1/build.sh
1841171 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg5/1.0-local1/build.sh
1841188 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841174 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/d5f23e384c63ba7bc07bf6880837ebe907bbe075/log
1840835 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='73c1b7e08b5987fa0c4c82061de4a126cc22a23e' P
1840849 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg12/1.0-local1/build.sh
1841131 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg12/1.0-local1/build.sh
1841160 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841132 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/73c1b7e08b5987fa0c4c82061de4a126cc22a23e/log
1840836 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='3ed9376af56f0773585627f70e36f839b48b70f4' P
1840852 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg17/1.0-local1/build.sh
1841118 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg17/1.0-local1/build.sh
1841148 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841120 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/3ed9376af56f0773585627f70e36f839b48b70f4/log
1840837 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='c943f09359d23245fbbc06d8a4aab520ea03cfb9' P
1840853 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg7/1.0-local1/build.sh
1841162 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg7/1.0-local1/build.sh
1841181 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841163 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/c943f09359d23245fbbc06d8a4aab520ea03cfb9/log
1840838 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='fe3320f4a6006cb1d80d4701ac886275fe0ce94d' P
1840854 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg13/1.0-local1/build.sh
1841145 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg13/1.0-local1/build.sh
1841167 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841147 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/fe3320f4a6006cb1d80d4701ac886275fe0ce94d/log
1840840 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='22b7e595229ea04d32df0fcb59c7662735ac7b93' P
1840855 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg6/1.0-local1/build.sh
1841140 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg6/1.0-local1/build.sh
1841164 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841141 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/22b7e595229ea04d32df0fcb59c7662735ac7b93/log
1840841 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='be5efca198be73b2842cb39804ce694efe8ea66b' P
1840856 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg18/1.0-local1/build.sh
1841123 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg18/1.0-local1/build.sh
1841151 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841125 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/be5efca198be73b2842cb39804ce694efe8ea66b/log
1840843 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='45d67da82a2f91319ca807e3d984ddbfba1fe37d' P
1840858 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg4/1.0-local1/build.sh
1841156 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg4/1.0-local1/build.sh
1841179 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841157 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/45d67da82a2f91319ca807e3d984ddbfba1fe37d/log
1840845 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='14e44efda95eb66cfcf123958e6fd5ae7ca8df02' P
1840862 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg16/1.0-local1/build.sh
1841193 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg16/1.0-local1/build.sh
1841197 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841194 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/14e44efda95eb66cfcf123958e6fd5ae7ca8df02/log
1840847 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='6169f337d481df16f328f57abf4fb3aeb67f8b62' P
1840857 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg15/1.0-local1/build.sh
1841168 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg15/1.0-local1/build.sh
1841185 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841170 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/6169f337d481df16f328f57abf4fb3aeb67f8b62/log
1840850 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='c6445466bddca2e8077fcb4433c096549382b480' P
1840859 15:41                                      |   \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg0/1.0-local1/build.sh
1841182 15:41                                      |       \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg0/1.0-local1/build.sh
1841191 15:41                                      |       |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841183 15:41                                      |       \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/c6445466bddca2e8077fcb4433c096549382b480/log
1840851 15:41                                      \_ sh -c env ARCHITECTURE='slc9_x86-64' BUILD_REQUIRES='defaults-release' CACHED_TARBALL='' CAN_DELETE='' COMMIT_HASH='0' DEPS_HASH='a95805fbcd31d34f46472ad99b7971a63a3b9705' DEVEL_HASH='' DEVEL_PREFIX='' BUILD_FAMILY='release' GIT_COMMITTER_NAME='unknown' GIT_COMMITTER_EMAIL='unknown' INCREMENTAL_BUILD_HASH='0' JOBS='16' PKGHASH='82d45ec23af449dbba5d56bb9ba7dea71852b37f' P
1840861 15:41                                          \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg8/1.0-local1/build.sh
1841177 15:41                                              \_ /bin/bash -e -x /build/muz/bits-works/sw/SPECS/slc9_x86-64/dummy-pkg8/1.0-local1/build.sh
1841192 15:41                                              |   \_ /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60
1841178 15:41                                              \_ /usr/bin/coreutils --coreutils-prog-shebang=tee /usr/bin/tee /build/muz/bits-works/sw/BUILD/82d45ec23af449dbba5d56bb9ba7dea71852b37f/log

[pbuncic]

You are right. I tried three concurrent Clang builds on my 16-core/64 GB machine, but despite being fully loaded, the computer remained responsive and nothing went wrong. I do not see how this behaviour could be easily prevented in makeflow. In the real world, we will not see this issue, and using makeflow actually offers tangible performance improvements and additional flexibility. For instance, building ROOT in alice.bits on my machine (once the sources are cached) takes 21 min, while with makeflow it drops to 16 min.

[pbuncic]

To test this, I prepared this package:

pbuncic@bits:~/Software/Bits/alice.bits$ cat test.sh
package: test
version: "1"
requires:

• ROOT
• GEANT4
• Rivet
• Openloops
• Herwig

The build with makeflow completed without any problem.

[smuzaffar]

thanks @pbuncic for looking in to it and confirming that makeflow does not monitor the tasks and build all independent packages in parallel (max up to nproc)

I have tested bits with makeflow for a dummy project and it resulted in hanging the build node. I had to reboot the node to get it in working stat. On a 16 core VM, each independent package was build with make -j 16 and bits/makeflow started 16 such builds in parallel ending up running 256 g++ processes [a] and avg load of system was above 300 [b].

I am sure that such a uncontrollable build will kill our aarch64 build machines (which are 20cores with 60GB memory each) while building PyTorch which requires nearly 4GB memory per nvcc process. Even building PyTroch alone we reduce number of parallel processes otherwise system kills it due to lack of memory.

[a]

2970440                                      |       |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy5.cpp -o heavy5.o
2970474                                      |       |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy6.cpp -o heavy6.o
2970507                                      |       |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy7.cpp -o heavy7.o
2970533                                      |       |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy8.cpp -o heavy8.o
2970548                                      |       |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy9.cpp -o heavy9.o
2970566                                      |       |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy10.cpp -o heavy10.o
2970587                                      |       |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy11.cpp -o heavy11.o
2970605                                      |       |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy12.cpp -o heavy12.o
2970635                                      |       |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy13.cpp -o heavy13.o
2970656                                      |       |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy14.cpp -o heavy14.o
2970680                                      |       |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy15.cpp -o heavy15.o
2970693                                      |       |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy16.cpp -o heavy16.o
2970268                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy1.cpp -o heavy1.o
2970278                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy2.cpp -o heavy2.o
2970285                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy3.cpp -o heavy3.o
2970305                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy4.cpp -o heavy4.o
2970308                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy5.cpp -o heavy5.o
2970351                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy6.cpp -o heavy6.o
2970376                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy7.cpp -o heavy7.o
2970377                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy8.cpp -o heavy8.o
2970437                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy9.cpp -o heavy9.o
2970472                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy10.cpp -o heavy10.o
2970511                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy11.cpp -o heavy11.o
2970541                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy12.cpp -o heavy12.o
2970572                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy13.cpp -o heavy13.o
2970601                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy14.cpp -o heavy14.o
2970629                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy15.cpp -o heavy15.o
2970649                                              |       \_ g++ -std=c++20 -O2 -Wall -Wextra -fPIC -c heavy16.cpp -o heavy16.o

> ps -u muzaffar -o pid,cmd --forest | grep 'g++ ' |wc
    257    3573   33287

[b]

top - 08:55:09 up 442 days, 15:18,  5 users,  load average: 299.77, 304.13, 252.62
Tasks: 936 total,   3 running, 932 sleeping,   0 stopped,   1 zombie
%Cpu(s):  0.1 us, 30.3 sy,  0.0 ni,  0.3 id, 68.4 wa,  0.8 hi,  0.1 si,  0.0 st
MiB Mem :  29155.6 total,    348.9 free,  28240.3 used,    566.4 buff/cache
MiB Swap:  15360.0 total,      0.0 free,  15360.0 used.    301.4 avail Mem

top - 08:44:19 up 442 days, 15:07,  5 users,  load average: 306.40, 296.71, 198.29
Tasks: 941 total,  35 running, 906 sleeping,   0 stopped,   0 zombie
%Cpu(s): 31.5 us, 50.7 sy,  0.0 ni,  0.1 id, 16.8 wa,  0.7 hi,  0.0 si,  0.0 st
MiB Mem :  29155.6 total,    340.4 free,  28204.4 used,    610.8 buff/cache
MiB Swap:  15360.0 total,      0.1 free,  15359.9 used.    302.3 avail Mem

[pbuncic]

I am puzzled, since in my tests (multiple concurrent Clang compilations) I was seeing up to 200 g++ processes, but the load never exceeded 32. Could this have something to do with the build node configuration?

[smuzaffar]

Yes, it could be due to build node itself. I ran the test on a OpenStack based 16Core , 32GB RAM VM. It has 2TB Ceph Volume (IO3 type) attached to it. This is one of our developers nodes which CMS developers use for building CMS software. From grafana monitoring I see that there was a lot of time spend in wait and system calls. Looks like all the memory was consumed

[pbuncic]

I tested building ROOT using makeflow and the newly committed code that allows up to n parallel builds. Here are the results:

time bits build --defaults o2 ROOT
real 29m4.641s
user 569m19.083s
sys 50m43.019s

time bits build --builders 1 --defaults o2 ROOT
real 28m55.371s
user 567m19.107s
sys 49m32.357s

time bits build --builders 2 --defaults o2 ROOT
real 24m52.287s
user 577m55.607s
sys 50m59.027s

time bits build --builders 3 --defaults o2 ROOT
real 24m32.519s
user 582m8.884s
sys 52m30.948s

time bits build --makeflow --defaults o2 ROO
real 24m36.582s
user 587m14.931s
sys 52m1.384s

While it is true that with the --builders option we can gain more control when the build node has insufficient memory, --makeflow already squeezes the maximum the build node can do, provided that memory and I/O are not limiting factors. On the other hand, makeflow, in principle, allows the build to be distributed across multiple nodes. We can keep both options for now but the steering in build.py is getting a bit messy....

[pbuncic]

@smuzaffar, not sure if you saw this. If you agree that there is nothing more we can do, I'd close the issue.

[smuzaffar]

@pbuncic , I am not suggesting to drop makeflow feature. It will be nice once we manage to use remote resources to build packages. For local build I will still prefer to use --builder N option.
ROOT itself does not have a complex dependency chain and most of its dependencies do not take much resources. I would like to test makeflow workflow for a bit large software stack (e.g. building all cmssw dependencies) to see how it behaves. I am 100% sure that it will crash our aarch64 hosts :-)