Incompatibility with python2 and python3

[ning-y]

I could not install GIM3E on either python2 or python3. On python2, GIM3E's dependencies have since moved to python3 causing syntax errors during their installations (in particular, swiglpk uses a sep argument in its print in its setup.py). On python3, GIM3E uses several python2-specific syntaxes which are no longer valid in python3.

Thus far I have almost overcome this by automatic conversion of python2-specific syntax to syntax that is compatible with both python2 and python3 using the modernize module in combination with the autopep8 module. The former modernize does well enough except one from __future__ import ... statement must be reordered in gime3e/core/gime3e.py (see PyCQA/modernize#263). The latter autopep8 is necessary because there are mixed tabs and spaces in the GIM3E source code but modernize does not handle those.

I've also found that on Termux but not on Debian, GLPK and SWIG must be installed at a system-level (pkg install swig glpk), because pre-built wheels for swiglpk seem to be unavailable (so it builds from source, but building from source requires GLPK and SWIG as dependencies). I doubt anyone else would run into this issue, though.

For Debian,

# Start in the GIM3E repository
python3 -m venv venv
source venv/bin/activate
pip install modernize autopep8
modernize -w ez_setup.py setup.py gim3e/
autopep8 -r --aggressive --in-place gim3e
# Fix the `from __future__ import ...` not being at top of file; you can use a text editor too
printf '%s\n' 3m1 w q | ed -s gim3e/core/gim3e.py
pip install .

I will update this issue if/when I progress further.

[ning-y]

Unsurprisingly, GIM3E is not compatible with modern COBRApy (cobra-0.29.1).

python3 gim3e/test_all.py
Traceback (most recent call last):
  File "/home/ning/repos/gim3e/gim3e/test_all.py", line 7, in <module>
    from gim3e.test import test_all
  File "/home/ning/repos/gim3e/gim3e/test/__init__.py", line 12, in <module>
    from cobra.manipulation import initialize_growth_medium
ImportError: cannot import name 'initialize_growth_medium' from 'cobra.manipulation' (/home/ning/repos/gim3e/lib/python3.12/site-packages/cobra/manipulation/__init__.py)

I tried installing GIM3E with COBRApy pinned to 0.2.0, but this failed due to python2-specific syntax in COBRApy's ez_setup.py.

COBRApy version pinned to	Can install	Can pass tests	Comments
0.2.0	No	N/A	Python2-specific syntax in COBRApy ez_setup.py
0.2.1	No	N/A	Install error for COBRApy: ModuleNotFoundError: No module named 'ez_setup'
0.3.0	Yes	No	Test error: "No working solvers detected"
0.4.0	Yes	No	Test error: "No working solvers detected"
0.5.4	Yes	No	Test error: "No working solvers detected"
0.6.0	No	N/A	This COBRApy version is not on PyPI
0.8.2	No	N/A	Install error for ruamel.yaml-0.14.12: NameError: name 'Str' is not defined. Did you mean: 'str'?
0.9.0	No	N/A	Install error for ruamel.yaml-0.14.12: NameError: name 'Str' is not defined. Did you mean: 'str'?
0.9.1	No	N/A	Install error for ruamel.yaml-0.14.12: NameError: name 'Str' is not defined. Did you mean: 'str'?
0.10.1	No	N/A	Install error for ruamel.yaml-0.14.12: NameError: name 'Str' is not defined. Did you mean: 'str'?
0.11.3	No	N/A	Install error for ruamel.yaml-0.14.12: NameError: name 'Str' is not defined. Did you mean: 'str'?
0.12.1	No	N/A	Install error for ruamel.yaml-0.14.12: NameError: name 'Str' is not defined. Did you mean: 'str'?
0.16.0	Yes	No	numpy error: ImportError: cannot import name 'float_' from 'numpy'. Did you mean: 'float16'?
0.20.0	No	N/A	pip error: ERROR: No matching distribution found for python-libsbml-experimental==5.18.3
0.24.0	No	N/A	pip error installing libsml: CMake Could NOT find SWIG (missing: SWIG_EXECUTABLE SWIG_DIR)
0.26.0	Yes	No	AttributeError: module 'numpy' has no attribute 'float'.
0.28.0	Yes	No	cobra.manipulation.initialize_growth_medium does not exist
0.29.1	Yes	No	cobra.manipulation.initialize_growth_medium does not exist

[ning-y]

There does not seem to be a simple solution by pinning COBRApy. I will first try to fix the errors with COBRApy version 0.3.0, because this is the closest installable COBRApy version to the version 0.2.0 referenced in some of the unit tests.

gim3e/gim3e/test/unit_tests.py

Line 56 in 37a24b2

if cobra_version == '0.2.0':

The warning and error messages reported in unit tests for COBRApy version 0.3.0 are:

cobra/solvers/__init__.py:59 UserWarning: No LP solvers found
cobra/core/__init__.py:20 UserWarning: ArrayBasedModel requires scipy
cobra/io/__init__.py:23 UserWarning: cobra.io.mat requires scipy
cobra/flux_analysis/essentiality.py:18 UserWarning: moma does not appear to be functional on your system
test_convert_objective_to_constraint (gim3e.test.unit_tests.TestGim3eCore.test_convert_objective_to_constraint) ... No working solvers detected.
ERROR
test_convert_to_irreversible (gim3e.test.unit_tests.TestGim3eCore.test_convert_to_irreversible) ... ERROR
test_evaluate_penalties (gim3e.test.unit_tests.TestGim3eCore.test_evaluate_penalties) ... No working solvers detected.
ERROR
test_irreversible_flux_variability_analysis (gim3e.test.unit_tests.TestGim3eCore.test_irreversible_flux_variability_analysis) ... No working solvers detected.
ERROR
test_optimize_turnover_metabolite (gim3e.test.unit_tests.TestGim3eCore.test_optimize_turnover_metabolite) ... No working solvers detected.
ERROR
test_solve_feasible (gim3e.test.unit_tests.TestGim3eCore.test_solve_feasible) ... No working solvers detected.
ERROR
test_solve_milp_feasible (gim3e.test.unit_tests.TestGim3eCore.test_solve_milp_feasible) ... No working solvers detected.
ERROR

======================================================================
ERROR: test_convert_objective_to_constraint (gim3e.test.unit_tests.TestGim3eCore.test_convert_objective_to_constraint)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ning/repos/gim3e/lib/python3.12/site-packages/gim3e/test/unit_tests.py", line 169, in test_convert_objective_to_constraint
    tolerance_integer = gim3e.integer_tolerances[the_solver]
                        ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: None

======================================================================
ERROR: test_convert_to_irreversible (gim3e.test.unit_tests.TestGim3eCore.test_convert_to_irreversible)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ning/repos/gim3e/lib/python3.12/site-packages/gim3e/test/unit_tests.py", line 126, in test_convert_to_irreversible
    gim3e.convert_to_irreversible_with_indicators(
  File "/home/ning/repos/gim3e/lib/python3.12/site-packages/gim3e/core/gim3e.py", line 606, in convert_to_irreversible_with_indicators
    reaction.reversibility = 0
    ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: property 'reversibility' of 'Reaction' object has no setter

[...truncated: the remaining unit tests fail with the KeyError]

"No working solvers detected" originates from a function within GIM3E itself,

gim3e/gim3e/core/gim3e.py

Lines 2598 to 2627 in 37a24b2

	def check_solver(solver = 'cplex'):
	""" A simple function to verify the solver to employ.
	"""
	from cobra import solvers
	available_solvers = solvers.solver_dict.keys()
	if solver not in available_solvers:
	if 'cplex' in available_solvers:
	print("Switched solver from " + solver + " to cplex.")
	return 'cplex'
	# Gurobi should work well but it is not
	# tested as thoroughly as cplex
	elif 'gurobi' in available_solvers:
	print("Switched solver from " + solver + " to gurobi.")
	return 'gurobi'
	# glpk works well for LPs but is usually not effective
	# for MILPs
	elif 'glpk' in available_solvers:
	print("Switched solver from " + solver + " to glpk.")
	return 'glpk'
	# Otherwise, none available.
	# Also note Java_glpk not tested/supported
	else:
	print("No working solvers detected.")
	return None
	else:
	return solver

Indeed, the cobra.solvers.solver_dict is empty:

Python 3.12.8 | packaged by conda-forge | (main, Dec  5 2024, 14:24:40) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cobra import solvers
cobra/solvers/__init__.py:59 UserWarning: No LP solvers found
cobra/core/__init__.py:20 UserWarning: ArrayBasedModel requires scipy
cobra/io/__init__.py:23 UserWarning: cobra.io.mat requires scipy
cobra/flux_analysis/essentiality.py:18 UserWarning: moma does not appear to be functional on your system
>>> solvers.solver_dict
{}

Following the instructions in COBRApy's version 0.3.0 INSTALL.md did not resolve this issue. Cloning COBRApy version 0.3.0 for a local pip install didn't work either. Installing COBRApy from PyPI with --no-binary :all: also did not work.

[ning-y]

I've temporarily given up on COBRApy version 0.3.0, because I could not find a way to get COBRApy to install with any solver at all. My next choice is COBRApy version 0.8.2, because this is the closest available COBRApy version on PyPI which introduces the use of swiglpk in COBRApy. This swiglpk dependency should be packaged with GLPK itself, so hopefully that will resolve the inability of COBRApy to find a solver.

The ruamel.yaml error is due to the deprecation of ast.Str in python>3.8. This is used in ruamel.yaml's setup.py (see L13-L14 and L68-L70):

Strangely, despite the aforementioned deprecation notice being for python>3.8, python version 3.8.20 does not fix the error; whereas python 3.7.0 (plus the installation of system package swig on Debian) successfully overcame that ast.Str error but encounters a known installation error for ruamel<0.15 due to incompatibility with python 3.7. Installation succeeded with python 3.6.0.

(It looks like during the installation, pip tried swiglpk-5.0.12 but encountered an ImportError: cannot import name 'NoReturn' during the install, and then resolved this automatically by trying and succeeding in install swiglpk-5.0.10 instead.)

However, despite indications that swiglpk installed successfully with GLPK (because the pip installation of swiglpk failed until libglpk-dev was installed on the system, and the installed swiglpk==5.0.10 could run the example in the swiglpk README successfully) COBRApy version 0.8.0 still fails to pick up GLPK as a solver. I also replicated this with COBRApy version 0.6.2, being the first stable release of COBRApy using swiglpk (but 0.6.2 is not available on PyPI).

Python 3.6.0 | packaged by conda-forge | (default, Feb  9 2017, 14:36:55) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cobra import solvers
/home/ning/miniforge3/envs/python3.6/lib/python3.6/site-packages/cobra/solvers/__init__.py:73: UserWarning: No LP solvers found
  warn("No LP solvers found")
>>> solvers.solver_dict
{}

Next, I might try installing COBRApy via conda-forge/bioconda, verifying that it has a non-empty solvers.solver_dict, and then checking the GIM3E test suite again.

[ning-y]

Now, I will try install COBRApy via conda-forge/bioconda. The hope is that the conda-forge/bioconda packaged COBRApy will come with solvers, because as above, trying to install COBRA via PyPI/pip fails to populate cobra.solvers.solver_dict even while using the --no-binary flag.

The first challenge seems to be finding a version of COBRApy which has cobra.solvers.solver_dict. GIM3E directly accesses this object.

COBRApy bioconda version/build	Can access cobra.solvers.solver_dict	Value of cobra.solvers.solver_dict
0.4.0b6/py35_0	Yes	{}
0.4.0/py35_1	Yes	{}
0.4.0/py36_1	Yes	{}
0.10.1	No	N/A
0.29.1	No	N/A

It is strange that 0.4.0 came without solvers, because there was an explicit bug report (opencobra/cobrapy#237) for this version that COBRApy on bioconda did not come packaged with solvers, and this bug was addressed by a bioconda pull request (bioconda/bioconda-recipes#996) which should have bumped the build for 0.4.0 to fix this. Even the specific build resulting from that pull request (cobra==0.4.0b6==py35_0) did not work. There is no version between 0.4.0 and 0.10.1 available on bioconda.

[ning-y]

My attempt to install COBRApy from bioconda did not fix the issue of COBRApy not being packaged with any solvers. However, reading the aforementioned bioconda pull request (bioconda/bioconda-recipes#996) led me to try what worked for those authors at that point in time, which was the installation of Cython.

On a Docker container running Python 3.6, pinning COBRApy==0.8.0 and installing Cython prior to GIM3E indeed fixes the lack of solvers in the COBRApy installation. However, the tests now fail with an ImportError for cobra.manipulation.initialize_growth_medium.

docker run -it --rm python:3.6.15-slim-bullseye bash
apt update
# ed is used to fix a "from __future__ import ..." statement,
# gcc is used in pip installation of swiglpk
# git is used to clone gim3e,
# libglpk-dev and swig are swiglpk dependencies,
apt install ed gcc git libglpk-dev swig -y
git clone https://github.com/brianjamesschmidt/gim3e/
cd gim3e
pip install modernize autopep8 cython
modernize -w ez_setup.py setup.py gim3e/
autopep8 -r --aggressive --in-place gim3e
sed -i -e "s/cobra>=0.2.0/cobra==0.8.0/" setup.py
printf '%s\n' 3m1 w q | ed -s gim3e/core/gim3e.py
pip install .
python3 -c "from cobra import solvers; print(solvers.solver_dict)"
# solvers.solver_dict is non-empty with a cglpk solver...
python3 gim3e/test_all.py
# ...but the tests fail with ImportError for cobra.manipulation.initialize_growth_medium

There doesn't seem to be a way to avoid this ImportError.

COBRApy version	Installed through	Has solvers	Runs test without ImportError ｜
0.6.2	pip/source	No	No
0.8.0	pip/PyPI	Yes	No

cobra.manupulation.initialize_growth_media was removed in opencobra/cobrapy@0f6cdf9 (roughly cobra>=0.6.0), alongside many other breaking changes. But swiglpk support (which solves the lack of solvers problem) was added roughly from cobra>=0.6.2.

[ning-y]

From my experience thus far, COBRApy is best pinned to 0.2.0, because even 0.3.0 seems to have breaking changes that make it incompatible with GIM3E. COBRApy 0.2.0 is not compatible with python3, but that can be fixed using modernize and possibly autopep8. It is tricky getting an installation of COBRApy that comes packaged with a solver. I had to infer the requirements to get from a COBRApy installation with no packaged solvers versus one packaged with GLPK, from the bioconda recipe for COBRApy 0.4.0 (bioconda/bioconda-recipes#996). What I had been missing was a glpk module from PyPI which provides Cython "headers" for GLPK. The whole installation process is described in this DOCKERFILE:

FROM python:3.6.15-slim-bullseye

# cmake and g++ are required to install python-libsbml
# ed is used to fix a "from __future__ import ..." statement,
# gcc is used in pip installation of swiglpk
# git is used to clone gim3e,
# libglpk-dev and swig are swiglpk dependencies,
RUN apt update && apt install cmake ed gcc git g++ libglpk-dev swig -y
# These convert GIM3E to python2/3-compatible syntax
RUN pip install autopep8 modernize
# These are build deps copied from the bioconda recipe for GIM3E v0.4.0
# See: https://github.com/bioconda/bioconda-recipes/blob/43e6dc59b5663e455b63a7963aa2014e9b5b1c65/recipes/cobra/meta.yaml
RUN pip install cython setuptools six
# As above, but for run deps
RUN pip install lxml numpy python-libsbml scipy
# During cobra install, cython complains that 'glpk.pxd' [is] not found
RUN pip install glpk

ARG COBRAPY_VERSION=0.2.0
RUN git clone https://github.com/opencobra/cobrapy && \
    git -C /cobrapy checkout $COBRAPY_VERSION && \
    modernize -w /cobrapy/ez_setup.py /cobrapy/setup.py /cobrapy/cobra/ && \
    autopep8 -r --aggressive --in-place /cobrapy/cobra/
RUN git clone https://github.com/brianjamesschmidt/gim3e && \
    modernize -w /gim3e/ez_setup.py /gim3e/setup.py /gim3e/gim3e/ && \
    autopep8 -r --aggressive --in-place /gim3e/gim3e/ && \
    sed -i -e "s/cobra>=0.2.0/cobra==$COBRAPY_VERSION/" /gim3e/setup.py && \
    printf '%s\n' 3m1 w q | ed -s /gim3e/gim3e/core/gim3e.py

RUN cd /cobrapy && python3 setup.py install
RUN pip install /gim3e
RUN echo "alias listsolvers=\"python3 -c 'from cobra import solvers; print(solvers.solver_dict)'\"" >> /root/.bashrc && \
    echo "alias testall=\"python3 /gim3e/gim3e/test_all.py\"" >> /root/.bashrc

Now GIM3E is installed with COBRApy version 0.2.0 with GLPK as solver. The tests have no ImportErrors. The next error to solve has to do with the loading of pickled models:

======================================================================
ERROR: test_solve_milp_feasible (gim3e.test.unit_tests.TestGim3eCore)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/gim3e/test/unit_tests.py", line 48,
 in setUp
    self.model = create_test_model(test_pickle)
  File "/usr/local/lib/python3.6/site-packages/gim3e/test/__init__.py", line 43, i
n create_test_model
    model = load(infile)
TypeError: unhashable type: 'Metabolite'

The relevant part is:

gim3e/gim3e/test/__init__.py

Lines 36 to 43 in 37a24b2

	try:
	from cPickle import load
	except:
	from pickle import load
	with open(test_pickle, "rb") as infile:
	model = load(infile)
	initialize_growth_medium(model, 'LB')