SNOPT Module Compilation and OpenMDAO build_pyoptsparse deprecation

[robfalck]

Description of feature

Hello all, first let me say that I'm really happy with the cyipopt interface so that we can get a working pyoptsparse with IPOPT trivially. Building the SNOPT module just seems like it could be made easier.

I've decided to take a approach of having a build_snopt_module script that will build the SNOPT module using either the source files or an existing shared library.

Potential solution

I've implemented a script that seems to work here, but I need more testers. I'd love to be able to get this working on windows using the Intel IFX compiler if possible.

https://github.com/robfalck/pyoptsparse/blob/build_snopt_module/pyoptsparse/build_snopt_module.py

This would mean we could get SNOPT up and working with a conda-installed version of pyoptsparse using a simple command:

  python -m pyoptsparse.build_snopt_module ~/Downloads/snopt-7.7/src

or, if a dynamic library already exists:

python -m pyoptsparse.build_snopt_module --snopt-lib /path/to/libsnopt7.so

Some questions

I'm currently downloading some necessary files from github in the script because, while included in the setup.py package_data, they are not made available on conda: https://github.com/robfalck/pyoptsparse/blob/498e4820d2eddd06eb092c25e8701107a90804e2/pyoptsparse/build_snopt_module.py#L127
If these were available in the conda version this would get a little more simple.

The grab-all-fortran-files script was skipping both sn27lu77.f and sn27lu90.f. I needed to include the latter to get this to work. Actually this is a case where it seems that depending on when it was obtained some people have different files in their SNOPT library. I added logic to only include the appropriate ones in the compilation.

Testing is difficult on CI because, while I'd use this on the OpenMDAO side of things, I'm not sure how your CI testing works with SNOPT.

If we get this working we'd probably be able to deprecate build_pyoptsparse. Would you be interested in accepting this script?

[marcomangano]

Hi @robfalck, I can't test the script at the moment but I skimmed through the main steps and functions. Thanks for putting it together with extensive comments, this would be definitely a welcome improvement!

I have some concerns about testing and maintaining this script, and I am sure @ewu63 and maybe @whophil will also have opinions about this. Just some quick thoughts:

• This script is just aimed at Conda builds, correct?
• I am not sure why the auxiliary files in pySNOPT/source do not show up on Conda, but I believe we could/should include them since they are part of pyOptSparse and not SNOPT. The files show up in the tarball from this repo that is used by the build scripts in conda-forge. Do you think this issue is a matter of how the libraries are stored on Conda, or something else happening during the build process?
• Testing might be a headache:
  - In this repo, as long as the PR does not come from a fork, we can run SNOPT tests on private builds - but the build process is local and does not rely on Conda.
  - On the Conda-forge side, I don't think we can setup any kind of private build, so I am not sure how to test this against proprietary code there.
  - I skimmed the OpenMDAO repo and as far as I understand you are relying on conda for pyoptsparse (at least for docker/testing). It might be doable to test the script on a private build that includes SNOPT source code on your side, but I am not sure how your CI is structured.

Overall, it might still make sense to host the script here, I am just not sure how to make it maintainable and testable in the long run.

[robfalck]

1. Yes this is mainly aimed at people who install from conda, but it also works with a cloned version of pyoptsparse.

2. I haven't really used conda for distribution, but I suspect there a some sort of file that dictates which files are included in the builds there. Maybe somewhere in the conda feedstock for pyoptsparse?

3. Yeah I'm not sure how to best test this. Since you can test with proprietary SNOPT files on github, I tested with a local build of pyoptsparse after doing python -m pip install -e pyoptsparse.git and it just worked.

I do think it makes sense for the script to be part of pyoptsparse.

I can't tell you how much I appreciate the cython IPOPT interface, it really simplified things. This was just an attempt to also get SNOPT working with minimal effort from the user.

Between this and the pixi environments we released with OpenMDAO today, a user can get a reliable MPI and SNOPT-capable OpenMDAO environment up and running in a couple of minutes.

[ewu63]

Glad to hear that cyipopt has been working well! I agree that there is a lot of improvement that can be made to the SNOPT installation process. Currently we only explicitly support building from source and only when building pyoptsparse (with copied SNOPT source files in-tree). As you point out, there are times where you may want to use a conda-provided pyoptsparse with your own SNOPT (either from source or from libsnopt7.so).

I took at a look at the script and I think there are some alternative approaches that may be more maintainable and I kind of want to throw out some ideas here for your thoughts. The goal here is to ultimately produce a shared library of SNOPT with Python bindings, either from source or linking to an existing dynamic library, and install it to the appropriate location. I see a few options here

• We create a meson/conda recipe for SNOPT in a separate repo which obviously cannot contain the actual SNOPT source code in its build steps, but which could be manually built and installed by the user by just invoking pip or similar. For bonus points, it could also produce a conda package and those artifacts can be hosted on e.g. a NASA internal conda channel. Serving pre-compiled packages via conda would be the easiest way to distribute the optimizer and avoids any user compilation at all. But the user can still point it to the source and compile it locally and install it properly as a package.
• Track the build steps within pyOptSparse, perhaps somewhere in pySNOPT/. This could be combined with the existing meson build steps to make it more flexible. However, the user still will have to clone this repo, do some configuration to build the .so, and then cp the .so in some non-pythonic way (such that pip or other build systems do not know this package exists or how to uninstall it). This is similar to the monolithic Python script you have, but I think a little bit more maintainable. Perhaps we can figure out how to build this "subpackage" via meson in a more intuitive way.

I am def leaning towards option 1, but curious for everyone's thoughts. Some questions for you @robfalck:

1. Is it necessary that you need users to perform the SNOPT compilation themselves (with multiple compilers/platforms) rather than having some ability to serve conda artifacts in a centralized and access-controlled fashion?
2. I personally find it important that if SNOPT is installed outside of pyoptsparse, it be installed by a proper Python backend and treated as a separate package rather than just copied to some $PREFIX as part of the build step. Would it make sense then to make this a separate package/repo if the user has to clone a repo to compile SNOPT on their machine regardless?

[robfalck]

1. I'm not aware of a common conda-repository available within NASA. Access to such resources between centers requires the involvement of IT. Users currently install SNOPT anyway with build_pyoptsparse, its just that this new approach is much easier.

2. Originally I thought that I wanted "cysnopt". A pre-built binary that could be downloaded, allow users to swap in their libsnopt7.so/dylib/dll, and just work. In fact I got so far as to get a working version of that with the help of claude code. In the end, I decided on this approach for a few reasons.

• There's significantly more code in the implementation of cysnopt than in this implementation, I think its less maintainable.
• Testing on CI doesn't alleviate any of the SNOPT availability issues.
• It's another package to maintain and I'm trying not to put more work on anyone right now.
• It's a binary package so we'd have to produce a lot of binary packages, and while possible I just don't have experience with that.
• In my experience, swapping libraries isn't the most trivial thing to do due to security measures put in place by macos, but there may be easier ways that I'm not aware of.

Because pyoptsparse, as installed from conda or as built with meson, allows the user to just specify the path to their SNOPT module, this approach seems a lot more simple. For maintainability sake, we could just save the appropriate meson build files rather than generating them dynamically.

I'm exploring ways in OpenMDAO to allow tests with SNOPT to be run for PRs submitted or approved by team members. That would prevent nefarious exploitation of the github actions workflows to expose the proprietary SNOPT data.

[ewu63]

Not shipping compiled binaries is fine with me since that is more of an infrastructure question than a pyOptSparse question. However, I still think the recipe for building such a binary locally should be its own python package with its own meson build files. Having an installation Python script copy the file into some $PREFIX/lib location at build time is not a scalable solution IMO, since no build tool is tracking this library and it cannot be uninstalled unless you know exactly where it is. This can easily lead to multiple snopt.so on the sys.path where it's opaque to the user where the library (or any import issue) is coming from.

I think I am somewhat converging on the following proposal:

• Create a libsnopt repo (or any other name)
• This repo contains the following
  • The meson recipe to build snopt.so from either source files or the dynamic library, the location of which can be obtained at build time via an environment variable
  • The setuptools backend to install this library (or meson-python if we fancy) as an actual Python package
• We could even extend the current pyoptsparse build process to accept a dynamic library also

The user experience would be as follows

1. Have an environment that contains pyoptsparse without SNOPT. This could be a conda environment with the conda-forge pyoptsparse package, or some other venv with a local build/install of pyoptsparse from source.
2. Clone the pyoptsparse repo locally if it is not already available
3. Navigate to this libsnopt directory, set up the environment variable to point to the SNOPT source or dynamic library location. Then run pip install .
4. The user should now have snopt available in this environment

For convenience and ease of use, this could actually be placed inside pyoptsparse instead of being a standalone repo. That way, it can share the same Fortran/Python files that are needed in the build process. The only thing that will be a little bit duplicated between pyoptsparse and this package will be the meson recipe, but I think that's OK given its simplicity. And this would be something I think the pyoptsparse maintenance team can support in development/maintaining and it would live inside this repo. We can even switch our internal docker builds to use this process to test it regularly and make sure it doesn't break. What do we think?

[ewu63]

Of course, the original question was on running a Windows build with ifx, which is something I do not have experience in. I do know that Windows compilations are possible but I have only worked on the conda-forge toolchain which uses flang. Of course, meson should at least help us somewhat in that process, but I doubt we will have a CI solution to test this scenario regularly.

[robfalck]

Having an installation Python script copy the file into some $PREFIX/lib location at build time is not a scalable solution IMO, since no build tool is tracking this library and it cannot be uninstalled unless you know exactly where it is. This can easily lead to multiple snopt.so on the sys.path where it's opaque to the user where the library (or any import issue) is coming from.

The script that I listed puts it under pyoptsparse/pySNOPT by default. That's the location that pyoptsparse assumes if the environment variable is not set. Wouldn't this just be cleaned up if pyoptsparse is uninstalled?

1. Have an environment that contains pyoptsparse without SNOPT. This could be a conda environment with the conda-forge pyoptsparse package, or some other venv with a local build/install of pyoptsparse from source.
2. Clone the pyoptsparse repo locally if it is not already available
3. Navigate to this libsnopt directory, set up the environment variable to point to the SNOPT source or dynamic library location. Then run pip install .
4. The user should now have snopt available in this environment

I'm sorry I'm not understanding, but why do you prefer this to:

1. Have an environment that contains pyoptsparse without SNOPT. This could be a conda environment with the conda-forge pyoptsparse package, or some other venv with a local build/install of pyoptsparse from source.
2. Run python -m pyoptsparse.build_snopt_module /path/to/src

The notion of having a conda-installed while also relying on a locally-cloned instance of the repo feels a bit confusing.

It just seems to me that if pyoptsparse is going to contain pySNOPT and have the user tell it where the snopt.so module is with an environment variable (or assume its within the pySNOPT directory by default), then it would be nice if it was also made trivially easy to build that module.

[robfalck]

I have a functional solution that is just implemented as part of our build_pyoptsparse script for now.

I've managed to get it to work under windows using the ifx fortran compiler. The one issue I'm having right now is that Python >= 3.8 requires that the intel fortran runtime libraries need to be made available by either

1. Copying the libraries to the same path as the compiled snopt pyd linking library
2. Calling os.add_dll_directory on windows.

The first approach works but isn't ideal because these files may change with updates to ifx and while space is cheap, having a lot of virtual environments would mean many copies of these files.

The second approach requires this logic be included in pySNOPT.py.
If we went this route, it might be good to have an environment variable like "PYSNOPT_LIB_PATH" set by the user and then have the call to os.add_dll_directory add that path if it's been set.

[ewu63]

Hi Rob, I see what you mean now, yes what you said makes sense, with the script provided by the package so the user does not have to clone anything at all. I pushed what I had to this branch. The entrypoint is not installed and the missing Fortran files are still not included, but those can be addressed quickly once #429 is merged. For now, clone and go to pyoptsparse/pySNOPT/libsnopt. You should be able to run python build_snopt.py <path-to-source> and it will install snopt for you. Essentially this has the same logic that you had in the python script, but the bulk of the logic is reimplemented in meson-python which is also invoked as an actual build backend. The only deficiency I see is that the version of the package is not passed correctly, but doing so is challenging since no tools support dynamic versioning in this manner. Anyway, feel free to test it and see what you think.

As for the Windows issue, I wonder if the use of meson-python will also help us there since they have some special import hooks. I'm not too familiar with how DLL loading works, but this page offers some explanation. Looks like we may have to fiddle with some options, e.g. tool.meson-python.allow-windows-internal-shared-libs. But meson-python might have the machinery to just make installation on Windows "work out of the box".

[robfalck]

That's good to hear, I'll take a look at this.
On the windows side, I think in the extreme case we have an environment variable like "PYOPTSPARSE_DLL_DIRS" and then in pysnopt we just add whatever directories are there via os.add_dll_directory. If meson is able to do this out of the box, that would be great.