Johnfreeman/issue510 externals v2.3#525
Merged
jcfreeman2 merged 56 commits intodevelopfrom Feb 6, 2026
Merged
Conversation
added 30 commits
December 9, 2025 15:12
…aled by "spack location -p gcc" in last night's nightly into vendored externals directory
…rom the v1 spack-packages/ repo
…installing CMake by adding a dependency found in spack-packages
…cc 14 and newer, covered in spack/spack#47019
… the spack-0.22.0 builtin provides
…pack-packages#115 which lets it build in newer versions of gcc
… and the 2022-era "latest" tag doesn't build under gcc 15.2. Note I've backed up the repo in https://github.com/jcfreeman2/cyrus-sasl
… and this will greatly speed development of externals v2.3
…olly doesn't build against gcc 15.2 This is due to greater stringency in how gcc 15.2 handles templates; the specific error is /tmp/root/spack-stage/spack-stage-folly-2025.11.24.00-xccq7iggewxd6v4rjz3rn22pcmmupa4d/spack-src/folly/container/detail/F14SetFallback.h:262:16: error: call of overloaded 'find(const std::basic_string_view<char>&)' is ambi guous 1898 262 | return find(key) != this->end(); 1899 | ~~~~^~~~~ 1900 /tmp/root/spack-stage/spack-stage-folly-2025.11.24.00-xccq7iggewxd6v4rjz3rn22pcmmupa4d/spack-src/folly/container/detail/F14SetFallback.h:262:16: note: there are 10 candidates
Basically, with my attempt to bump CMake from 3.26.3 to 4.1.2, several packages
don't build out-of-the box due to this error:
1 error found in build log:
>> 3 CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
4 Compatibility with CMake < 3.5 has been removed from CMake.
...so in this commit, I vendor packages either to allow for a newer
version with more modern CMake code (nlohmann-json) or to add patches
to update the cmake_minimum_required calls (glog, msgpack-c,
rapidjson)
A quick summary: * folly get bumped to a recent (last couple of weeks) version * gcc got rewound to 14.3 since folly currently can't build under gcc 15.2 (see earlier commit) * cyrus-sasl gets its own repo tag since it needed to be updated to build under gcc 15.2 (not sure itf this is needed for 14.3) * cmake switched from 4.2.0 to 4.1.2; less "bleeding edge"
…o as to patch their cmake_minimum_required lines to be acceptable to contemporary CMake (v4.1.2)
This requires some vendoring / modifications, including: -Vendoring python from Spack 0.22.0 so I can add newer versions -Vendoring glib from Spack 0.22.0 so as to support newer versions -Vendoring py-cython from Spack 0.22.0 to support Python v3.12.12 -Vendoring py-numpy from Spack 0.23.1 to allow for versions which support Python v3.12.12 -Updating py-moo to depend on newer py-numpy
…et a "ModuleNotFoundError: No module named 'setuptools'" error without explicit py-setuptools dependence
…versions, this time for Pistache, libmng and libtiff
…e CMake from externals v2.2 to avoid a complex series of patches to account for deprecated-in-later-CMake-versions cmake_minimum_version calls
I may revisit this decision, but for now uhal in its current incarnation simply doesn't build in Python 3.12 with the error message being "no module named distutils" Changes include: -Un-vendoring glib, as this isn't necessary for Python 3.11 -Bumping the version of py-anyconfig used by py-moo to include a fix which removed the deprecated pkg_resources (ssato/python-anyconfig#156)
…n to use the test externals v2.3 image I've created
…d-since-2022 pypi area when creating the externals image
…ernals image is v2.3T, underscoring that it's being used just for testing purposes right now
…ly on this branch
This really involves two things: -(Re)vendoring glib so that I can add a version new enough to work with Python 3.12.12; note I also needed to add a dependency on py-packaging -Bumping up to a modern version of uhal. Note that even though some work was done back in August by the developers to support Python 3.12, I still needed to add a patch to prevent compilation of Python code in uhal when spack install is called, otherwise there's an error
… externals image
…est build workflow is checking out the daq-release repo
… and fix checkout failure
…t to try and fix checkout failure" This reverts commit e30d2da.
…en the test build workflow is checking out the daq-release repo" This reverts commit b5f9649.
…under Python 3.12.12 the integration tests give "No module named 'kafka.vendor.six.moves'" error messages
… builtin, doesn't build under gcc 14.3. Bump to highest stable v19, 19.1.7
… than Spack openssl
…ations and clean the cache when the build-ext.sh finishes successfully
…-sasl versions <= 2.1.28, as this resulted in the fddaq-v5.5.0 version of cyrus-sasl being build with gcc 11 and causing a link problem when rdkafka-based repos were built
…s v2.3 test builds
…y an incomplete attempt to remove any dependencies on a Spack installation of git; I'll probably take these out before a PR, but am leaving them in for purposes of reproducibility right now
…https://github.com/DUNE-DAQ/daq-release/wiki/Externals-Survey-v2.3#rdkafka-error, roll back cyrus-sasl to its original version with a patch which lets it build under gcc 14.x
…s v2.3 test builds
… version of cyrus-sasl now that the patch allows it to build under gcc 14.x
…ts" file used to self-consistently update the Python packages Before getting into per-package commentary, some preliminary observations: * The changes here are motivated by the code outside of our pure Python packages; those changes will come in a followup commit. In other words, I looked at the various import statements of the Python code in our hybrid C++/Python packages (like daq-buildtools or daqconf) to determine what packages were needed * The general constraint was "require a package's externals v2.3 version to be the same or newer as its externals v2.2 version * For the python package itself, I pinned it to v3.12.12 * Having said that, I may need to pin versions for other packages (like click) which I know are currently installed as spack packages (py-click, in this case) * If I added a package, then I set its minimum version to v0.0.0 * In some isolated cases, I don't account for importations in our code. E.g., the 2017-era imports in the timing repo's code. * I generally only included constraints on packages which are directly imported, as opposed to _their_ dependencies. In the spirit of Chesterton's Fence, however, I left some in - e.g., for rsa, which is needed by google-auth, which is needed by kubernetes. Specific commentary: * GitPython was added because runconftool uses it (yes, runconftools is a pure Python package, so it's a little bit of jumping-the-gun in the context of this commit) * h5py added as its used in numerous places (integrationtest, rawdatautils...) * pyipmi - perhaps this is overkill, it's in the timing repo? * PySocks added thanks to a "import socket" call in rest_api_child.py in drunc; again, perhaps jumping the gun a little. * tqdm added as this is needed in parts of the trgtools code * PyYAML since there are numerous places in daq-buildtools code which call "import yaml"
…pos' pyproject.toml files to the comprehensive pyproject.toml. Note that for packages corresponding to https://github.com/DUNE-DAQ/daq-deliverables/issues/197 I pegged the versions rather than letting them float upwards
A couple of comments: -The pyproject.toml on which I ran "poetry install" to get these package versions is in the scripts/version_surveys subdirectory -I removed references to moo since py-moo is a Spack-installed package
…or durationpy's version as 0.1, so removing it for now
…pdated rclone and openmpi versions
…o see it once, ever) be suppressed
…ing message which only needs to be seen once about possible future policy change
…is feature branch
…ma9-slim-externals:v2.3 (effectively an alias to the final, successful test externals image ghcr.io/dune-daq/alma9-slim-externals:v2.3Y)
…10_externals_v2.3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses Issue #510. The final draft of externals v2.3 has been available for testing for a while now (
EXTFD_DEV_260115_A9), and it was agreed with Wes and others - and mentioned at the Software I&T meeting this week - that at the end of this week (i.e., Friday, Feb. 6) we'll switch the nightly over to be externals v2.3 based, while aBKP-prefixed parallel nightly based on externals v2.2 will continue to be available.This is just a subset of the testing that has been undergone, but among other things:
-The integration tests pass
-About two weeks ago Kurt performed some testing with only minor changes suggested, such as DUNE-DAQ/integrationtest#141
-Since v8.11.0,
daq-buildtoolsworks both with externals v2.2 and externals v2.3 areas.As of this moment I'm marking this PR a "Draft", but plan to lift this status and merge tomorrow.
For further details, see the Wiki I dedicated to externals v2.3 work here.