Migrate CI/CD to NWChemEx org, master branch, real PyPI - #34
Merged
Conversation
Remove the 'Private :: Do Not Upload' PyPI classifier, which per PyPI's own documented behavior causes any upload (including to TestPyPI) to be permanently rejected outright -- a hard blocker for publishing. Switch pull_request and merge workflows from NWChemEx/.github + master to ryanmrichard/.github + build_overhaul. FriendZone is pure Python (no CMakeLists.txt, confirmed by an earlier de-C++-ify commit in this branch's history), so this uses a new pattern rather than the compiled-repo template: doc_target: "Sphinx" skips the nonexistent Doxygen/CMake doc build (deploy_nwx_docs.yaml/test_nwx_docs.yaml both document this literal special value), and a lean OS-only test_python job (no compiler axis -- nothing to compile) replaces the old CMake-oriented test_library job, modeled on NWXCMake's own working pure-Python CI. The release job similarly reuses build_pypi_dist's existing plain-Python path (cibw_build left unset), exactly matching NWXCMake's already-working merge.yaml. Delete stale root requirements.txt: unreferenced by any CI workflow, inconsistent with pyproject.toml (pydantic isn't a real dependency, confirmed unimported anywhere under python/friendzone), superseded by pyproject.toml's optional-dependencies as the single source of truth. Delete tests/python/unit_tests/test_friendzone.py: dead code (only an if __name__ == "__main__" block, references a src/python path that no longer exists post-move to python/friendzone). Add tests/python/conftest.py with the session-scoped RuntimeView fixture, for consistency with the rest of the ecosystem even though no current FriendZone test constructs RuntimeView directly.
…ypi_dist docs/source/conf.py's autoapi_dirs still pointed at 'src', the pre-move path from before the de-C++-ify commit relocated Python source to python/friendzone -- broke the Sphinx docs build outright. build_pypi_dist's 'Verify Wheel Installs' step needs extra_index_url to resolve nwchemex-simde (TestPyPI-only) when installing the just-built wheel; also add a PR-time dry-run build check (test_pypi_deploy, matching NWXCMake's own pull_request.yaml) to catch sdist-manifest issues an editable install wouldn't.
tests/python/unit_tests/nwx2ase/test_nwchem_via_ase.py and nwx2molssi/test_nwchem_via_molssi.py both unconditionally 'import numpy as np' at module level (unlike the ase/qcengine/etc. imports, which are properly feature-gated via find_spec rather than a hard import), so numpy is a real, always-needed test dependency that was missing from pyproject.toml.
…hed wheel target Under actions/setup-python's default '3.x' (resolves to 3.14 today), pip found no matching prebuilt wheel for nwchemex-simde/parallelzone/etc (cp312-only, per platform_matrix.yaml) and fell back to building them from source. That build succeeded (setup_nwx_dev_env provides MPI/ Boost/etc.) but produced a binary missing the rpath-bundled shared libs (e.g. libspdlog) that cibuildwheel+delocate normally bundle into the real published wheel, so it failed to import at runtime.
…A shmem probes Extends the zero-rebuild diagnostic step to confirm which nwchemex package versions were actually installed, compare the bundled libmpi hash/size/symbols against round 1 and round 3 builds, and print verbose shmem/opal component selection output plus forced shmem=posix/mmap/sysv attempts, per Part 1 Step 1 of the wheel-MPI debugging plan.
Root cause found: vendored MPI's singleton startup can't find its orted daemon executable (auditwheel/delocate only bundle shared libs, not bin/ executables). Fixed at the source in build_pypi_dist (stop vendoring MPI on Linux, require system MPI), so the diagnostic step is no longer needed.
…gnostic Pytest now dies with no traceback on the first real test (after excluding vendored MPI from wheels and switching to system MPI), right where conftest.py's session RuntimeView fixture would run. Isolate whether RuntimeView() itself is still the crash point against system MPI, or whether it's further downstream in chemist/tensorwrapper.
Confirmed the republished wheel (0.1.61) no longer bundles libmpi, yet RuntimeView() still fails with the same /usr/local/openmpi help-file error as the old vendored-MPI bug. Check whether the system apt libmpi.so.40 itself somehow has that prefix baked in, whether any residual bundled libmpi remains anywhere in site-packages, and the extension's actual RPATH/RUNPATH.
… scope to ubuntu pip show | head triggered a broken pipe under pipefail, aborting the step before reaching the real checks. Also add a direct ompi_info --path prefix call (independent of our own extension) to check what the ambient apt-installed OpenMPI actually reports, and check the CMake-installed lib/ shared libraries' own RPATH/RUNPATH, not just the wheel's own extension. Scope the whole step to ubuntu-latest only since it uses Linux-only tools (strings, readelf) and was spuriously failing macOS.
Confirmed root cause chain end to end: vendored MPI's singleton mode couldn't find its orted daemon (auditwheel never bundles bin/ executables) -> fixed by excluding MPI libs from the wheel and requiring system MPI -> the auditwheel --exclude glob patterns silently didn't work (cibuildwheel v2.22.0's pinned auditwheel 6.1.0 only supports exact soname matching, not fnmatch) -> fixed with exact sonames. Confirmed via ompi_info/readelf that the extension's baked-in RUNPATH still lists the build container's /usr/local/openmpi/lib first, but correctly falls through to the system libmpi.so.40 since that directory doesn't exist at runtime -- RuntimeView() and chemist/tensorwrapper imports all succeed now.
…m in CI load_nwchem_via_molssi_modules hardcoded its module keys to lowercase "nwchem" instead of "NWChem" (regressed in a226a7d, "Python Packaging (#30)"), diverging from this function's own docstring, the sibling ASE wrapper's "ASE(NWChem) : " convention, and both this repo's and NWChemEx's test suites, all of which expect "NWChem : SCF" etc. Since ModuleManager key lookup is exact-string, this silently broke every NWChem-via-MolSSI module registration. Also enable the new needs_nwchem install in test_python so this repo's own nwx2molssi tests (and NWChemEx's downstream compute_energy tests) actually exercise a real NWChem install in CI instead of skipping via is_nwchem_enabled(), which was masking this regression from CI in the first place.
needs_nwchem alone installs the external nwchem binary, but load_molssi_modules()/load_ase_modules() are no-ops unless qcengine/qcelemental/networkx (molssi extra) and ase (ase extra) are also installed -- confirmed the smoke-test PR's first run still skipped every nwx2molssi/nwx2ase NWChem test with 'MolSSI/ASE is not enabled!' despite the casing fix and a real nwchem on PATH.
ase's Atoms.new_array() still does `a.shape = (-1,) + shape` on an empty array, deprecated by numpy 2.5 in favor of np.reshape. Only surfaced now that the ase extra is actually installed and exercised in CI (previously silently skipped via is_ase_enabled()). Unrelated to anything on our side -- ase's own bug, filtered out rather than letting it hard-fail every ASE-backed friend test under this repo's filterwarnings = ["error", ...] pytest config.
- Repoint all ryanmrichard/.github references to NWChemEx/.github - Flip merge.yaml/pull_request.yaml triggers from build_overhaul to master - Drop TestPyPI repository-url/extra_index_url everywhere, including the repo-local test_python_build.yaml (simde is now on real PyPI, so the extra index is no longer needed to resolve it) - Pin nwchemex-simde to its published PyPI floor instead of a git URL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the ecosystem-wide build/CI migration (see NWXCMake, PluginPlay, Chemist, SimDE, ChemCache).
ryanmrichard/.githubreferences toNWChemEx/.githubmerge.yaml/pull_request.yamltriggers frombuild_overhaultomasterrepository-url/extra_index_urleverywhere, including the repo-localtest_python_build.yaml(nwchemex-simdeis now on real PyPI, so the extra index is no longer needed)nwchemex-simdeto its published PyPI floor instead of a git URLNote:
NWChemEx/FriendZone#14(grad_mods) is a separate, pre-existing PR that is currentlyCONFLICTINGagainstmaster-- not touched here, left for its author to rebase.