Cannot build on python 3.7.2, ‘PyThreadState’ has no member named ‘exc_type’ and similar errors

[estein-de]

Environment:

CPython 3.7.2
Cython 0.29.13 installed from PyPI

Attempting to install gil_load 0.3.5 from PyPI fails. See errors below.

$ pip3 install gil_load
Collecting gil_load
  Using cached https://files.pythonhosted.org/packages/2e/32/be5dbb75fe4ec3308578f5e3164248bd5509ebe59591fa72d9adf244ba9f/gil_load-0.3.5.tar.gz
Building wheels for collected packages: gil-load
  Building wheel for gil-load (setup.py) ... error
  Complete output from command /home/vagrant/venvs/3.7.2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-q2p2qbsg/gil-load/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-0s_x6skj --python-tag cp37:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/gil_load
  copying gil_load/__version__.py -> build/lib.linux-x86_64-3.7/gil_load
  copying gil_load/__init__.py -> build/lib.linux-x86_64-3.7/gil_load
  running build_ext
  building 'gil_load.gil_load' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/gil_load
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/vagrant/.pyenv/versions/3.7.2/include/python3.7m -c gil_load/gil_load.c -o build/temp.linux-x86_64-3.7/gil_load/gil_load.o
  gil_load/gil_load.c: In function ‘__Pyx__ExceptionSave’:
  gil_load/gil_load.c:6848:19: error: ‘PyThreadState’ has no member named ‘exc_type’
       *type = tstate->exc_type;
                     ^
  gil_load/gil_load.c:6849:20: error: ‘PyThreadState’ has no member named ‘exc_value’
       *value = tstate->exc_value;
                      ^
  gil_load/gil_load.c:6850:17: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       *tb = tstate->exc_traceback;
                   ^
  gil_load/gil_load.c: In function ‘__Pyx__ExceptionReset’:
  gil_load/gil_load.c:6857:22: error: ‘PyThreadState’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                        ^
  gil_load/gil_load.c:6858:23: error: ‘PyThreadState’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                         ^
  gil_load/gil_load.c:6859:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                      ^
  gil_load/gil_load.c:6860:11: error: ‘PyThreadState’ has no member named ‘exc_type’
       tstate->exc_type = type;
             ^
  gil_load/gil_load.c:6861:11: error: ‘PyThreadState’ has no member named ‘exc_value’
       tstate->exc_value = value;
             ^
  gil_load/gil_load.c:6862:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tstate->exc_traceback = tb;
             ^
  gil_load/gil_load.c: In function ‘__Pyx__GetException’:
  gil_load/gil_load.c:6907:22: error: ‘PyThreadState’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                        ^
  gil_load/gil_load.c:6908:23: error: ‘PyThreadState’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                         ^
  gil_load/gil_load.c:6909:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                      ^
  gil_load/gil_load.c:6910:11: error: ‘PyThreadState’ has no member named ‘exc_type’
       tstate->exc_type = local_type;
             ^
  gil_load/gil_load.c:6911:11: error: ‘PyThreadState’ has no member named ‘exc_value’
       tstate->exc_value = local_value;
             ^
  gil_load/gil_load.c:6912:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tstate->exc_traceback = local_tb;
             ^
  gil_load/gil_load.c: In function ‘__Pyx__ExceptionSwap’:
  gil_load/gil_load.c:8083:22: error: ‘PyThreadState’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                        ^
  gil_load/gil_load.c:8084:23: error: ‘PyThreadState’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                         ^
  gil_load/gil_load.c:8085:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                      ^
  gil_load/gil_load.c:8086:11: error: ‘PyThreadState’ has no member named ‘exc_type’
       tstate->exc_type = *type;
             ^
  gil_load/gil_load.c:8087:11: error: ‘PyThreadState’ has no member named ‘exc_value’
       tstate->exc_value = *value;
             ^
  gil_load/gil_load.c:8088:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tstate->exc_traceback = *tb;
             ^
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for gil-load
  Running setup.py clean for gil-load
Failed to build gil-load
Installing collected packages: gil-load
  Running setup.py install for gil-load ... error
    Complete output from command /home/vagrant/venvs/3.7.2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-q2p2qbsg/gil-load/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-cs0rw4he/install-record.txt --single-version-externally-managed --compile --install-headers /home/vagrant/venvs/3.7.2/include/site/python3.7/gil-load:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/gil_load
    copying gil_load/__version__.py -> build/lib.linux-x86_64-3.7/gil_load
    copying gil_load/__init__.py -> build/lib.linux-x86_64-3.7/gil_load
    running build_ext
    building 'gil_load.gil_load' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/gil_load
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/vagrant/.pyenv/versions/3.7.2/include/python3.7m -c gil_load/gil_load.c -o build/temp.linux-x86_64-3.7/gil_load/gil_load.o
    gil_load/gil_load.c: In function ‘__Pyx__ExceptionSave’:
    gil_load/gil_load.c:6848:19: error: ‘PyThreadState’ has no member named ‘exc_type’
         *type = tstate->exc_type;
                       ^
    gil_load/gil_load.c:6849:20: error: ‘PyThreadState’ has no member named ‘exc_value’
         *value = tstate->exc_value;
                        ^
    gil_load/gil_load.c:6850:17: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         *tb = tstate->exc_traceback;
                     ^
    gil_load/gil_load.c: In function ‘__Pyx__ExceptionReset’:
    gil_load/gil_load.c:6857:22: error: ‘PyThreadState’ has no member named ‘exc_type’
         tmp_type = tstate->exc_type;
                          ^
    gil_load/gil_load.c:6858:23: error: ‘PyThreadState’ has no member named ‘exc_value’
         tmp_value = tstate->exc_value;
                           ^
    gil_load/gil_load.c:6859:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         tmp_tb = tstate->exc_traceback;
                        ^
    gil_load/gil_load.c:6860:11: error: ‘PyThreadState’ has no member named ‘exc_type’
         tstate->exc_type = type;
               ^
    gil_load/gil_load.c:6861:11: error: ‘PyThreadState’ has no member named ‘exc_value’
         tstate->exc_value = value;
               ^
    gil_load/gil_load.c:6862:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         tstate->exc_traceback = tb;
               ^
    gil_load/gil_load.c: In function ‘__Pyx__GetException’:
    gil_load/gil_load.c:6907:22: error: ‘PyThreadState’ has no member named ‘exc_type’
         tmp_type = tstate->exc_type;
                          ^
    gil_load/gil_load.c:6908:23: error: ‘PyThreadState’ has no member named ‘exc_value’
         tmp_value = tstate->exc_value;
                           ^
    gil_load/gil_load.c:6909:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         tmp_tb = tstate->exc_traceback;
                        ^
    gil_load/gil_load.c:6910:11: error: ‘PyThreadState’ has no member named ‘exc_type’
         tstate->exc_type = local_type;
               ^
    gil_load/gil_load.c:6911:11: error: ‘PyThreadState’ has no member named ‘exc_value’
         tstate->exc_value = local_value;
               ^
    gil_load/gil_load.c:6912:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         tstate->exc_traceback = local_tb;
               ^
    gil_load/gil_load.c: In function ‘__Pyx__ExceptionSwap’:
    gil_load/gil_load.c:8083:22: error: ‘PyThreadState’ has no member named ‘exc_type’
         tmp_type = tstate->exc_type;
                          ^
    gil_load/gil_load.c:8084:23: error: ‘PyThreadState’ has no member named ‘exc_value’
         tmp_value = tstate->exc_value;
                           ^
    gil_load/gil_load.c:8085:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         tmp_tb = tstate->exc_traceback;
                        ^
    gil_load/gil_load.c:8086:11: error: ‘PyThreadState’ has no member named ‘exc_type’
         tstate->exc_type = *type;
               ^
    gil_load/gil_load.c:8087:11: error: ‘PyThreadState’ has no member named ‘exc_value’
         tstate->exc_value = *value;
               ^
    gil_load/gil_load.c:8088:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         tstate->exc_traceback = *tb;
               ^
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/home/vagrant/venvs/3.7.2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-q2p2qbsg/gil-load/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-cs0rw4he/install-record.txt --single-version-externally-managed --compile --install-headers /home/vagrant/venvs/3.7.2/include/site/python3.7/gil-load" failed with error code 1 in /tmp/pip-install-q2p2qbsg/gil-load/
You are using pip version 19.0.1, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$

I know that gil_load is not been tested on Python 3.7, so this isn't a regression bug - just missing support.

It looks like other cython based modules are having similar issues:

• Add support for Python 3.7 kundajelab/genomelake#21
• Python 3.7 Build failure: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’? mcfletch/pyopengl#11
• etc..

[estein-de]

Alternatively, when I install gil_load directly from the source code like so (I had to figure out that 0.3.5 was commit 36de61d based on browsing the git history as the repo has no tags), I can compile and install.

$ python setup.py install --use-cython
Compiling gil_load/gil_load.pyx because it changed.
[1/1] Cythonizing gil_load/gil_load.pyx
/home/vagrant/venvs/3.7.2/lib/python3.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /vagrant/apps/gil_load/gil_load/gil_load.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
running install
running bdist_egg
running egg_info
creating gil_load.egg-info
writing gil_load.egg-info/PKG-INFO
writing dependency_links to gil_load.egg-info/dependency_links.txt
writing top-level names to gil_load.egg-info/top_level.txt
writing manifest file 'gil_load.egg-info/SOURCES.txt'
reading manifest file 'gil_load.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'gil_load.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/gil_load
copying gil_load/__init__.py -> build/lib.linux-x86_64-3.7/gil_load
copying gil_load/__version__.py -> build/lib.linux-x86_64-3.7/gil_load
running build_ext
building 'gil_load.gil_load' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/gil_load
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/vagrant/.pyenv/versions/3.7.2/include/python3.7m -c gil_load/gil_load.c -o build/temp.linux-x86_64-3.7/gil_load/gil_load.o
gil_load/gil_load.c: In function ‘__pyx_pw_8gil_load_8gil_load_6_run’:
gil_load/gil_load.c:3691:17: warning: ‘__pyx_v_f’ may be used uninitialized in this function [-Wmaybe-uninitialized]
                 (void)(fprintf(__pyx_v_f, ((char const *)"%s  GIL load: %.2f (%.2f, %.2f, %.2f)\n"), __pyx_v_timestamp, __pyx_v_8gil_load_8gil_load_gil_load, __pyx_v_8gil_load_8gil_load_gil_load_1m, __pyx_v_8gil_load_8gil_load_gil_load_5m, __pyx_v_8gil_load_8gil_load_gil_load_15m));
                 ^
gil_load/gil_load.c:3279:9: note: ‘__pyx_v_f’ was declared here
   FILE *__pyx_v_f;
         ^
gcc -pthread -shared -L/home/vagrant/.pyenv/versions/3.7.2/lib -Wl,-rpath=/home/vagrant/.pyenv/versions/3.7.2/lib -L/home/vagrant/.pyenv/versions/3.7.2/lib -Wl,-rpath=/home/vagrant/.pyenv/versions/3.7.2/lib build/temp.linux-x86_64-3.7/gil_load/gil_load.o -L/home/vagrant/.pyenv/versions/3.7.2/lib -lpython3.7m -o build/lib.linux-x86_64-3.7/gil_load/gil_load.cpython-37m-x86_64-linux-gnu.so -lrt
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/gil_load
copying build/lib.linux-x86_64-3.7/gil_load/__init__.py -> build/bdist.linux-x86_64/egg/gil_load
copying build/lib.linux-x86_64-3.7/gil_load/__version__.py -> build/bdist.linux-x86_64/egg/gil_load
copying build/lib.linux-x86_64-3.7/gil_load/gil_load.cpython-37m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/gil_load
byte-compiling build/bdist.linux-x86_64/egg/gil_load/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/gil_load/__version__.py to __version__.cpython-37.pyc
creating stub loader for gil_load/gil_load.cpython-37m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/gil_load/gil_load.py to gil_load.cpython-37.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying gil_load.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying gil_load.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying gil_load.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying gil_load.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
gil_load.__pycache__.gil_load.cpython-37: module references __file__
creating dist
creating 'dist/gil_load-0.3.5-py3.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing gil_load-0.3.5-py3.7-linux-x86_64.egg
creating /home/vagrant/venvs/3.7.2/lib/python3.7/site-packages/gil_load-0.3.5-py3.7-linux-x86_64.egg
Extracting gil_load-0.3.5-py3.7-linux-x86_64.egg to /home/vagrant/venvs/3.7.2/lib/python3.7/site-packages
Adding gil-load 0.3.5 to easy-install.pth file

Installed /home/vagrant/venvs/3.7.2/lib/python3.7/site-packages/gil_load-0.3.5-py3.7-linux-x86_64.egg
Processing dependencies for gil-load==0.3.5
Finished processing dependencies for gil-load==0.3.5

There are a few things from the output that I wonder might be causing problems of some kind, but I don't know:

Cython directive 'language_level' not set, using 2 for now (Py2).

gil_load/gil_load.c: In function ‘__pyx_pw_8gil_load_8gil_load_6_run’:
gil_load/gil_load.c:3691:17: warning: ‘__pyx_v_f’ may be used uninitialized in this function [-Wmaybe-uninitialized]
                 (void)(fprintf(__pyx_v_f, ((char const *)"%s  GIL load: %.2f (%.2f, %.2f, %.2f)\n"), __pyx_v_timestamp, __pyx_v_8gil_load_8gil_load_gil_load, __pyx_v_8gil_load_8gil_load_gil_load_1m, __pyx_v_8gil_load_8gil_load_gil_load_5m, __pyx_v_8gil_load_8gil_load_gil_load_15m));
                 ^
gil_load/gil_load.c:3279:9: note: ‘__pyx_v_f’ was declared here
   FILE *__pyx_v_f;
         ^

Is this an easy fix for gil_load?

[chrisjbillington]

Thanks for the report! I'm looking into it, and I think the problem is simply that I distribute C files instead of cython files. If I allow cython to generate the C code at build time instead of distributing pre-generated C, then things ought to be less fragile to version changes. Things are working for me locally with Python 3.7, and I'm just fixing up some issues with this package (it was actually in the middle of a pretty big change and so git-master isn't functional right now) and will release a new version soon!

[chrisjbillington]

I'm not sure about the warning you're seeing "note: ‘__pyx_v_f’ was declared here", I don't see that one - but then again I'm working on git-master. The Cython language level warning I do see - I guess I'll declare language level = 3 and confirm everything still works with the new version.

[estein-de]

That makes sense. Can you git tag the already-released versions on the repo, as well? Right now anyone trying to collaborate/test will have a bit of a harder time.

[chrisjbillington]

Ah, sure thing, have pushed some tags!

Version 0.4 should be out today or tomorrow once I finish updating the README.

[estein-de]

I am standing by to test! :)

[estein-de]

@chrisjbillington let me know if there is anything I can do to help you out!

[estein-de]

Is there any way you could create a 0.3.6 release for just the build issues, without mixing it in with the larger effort to track different threads separately in 0.4.x?

Are the changes simple to just fix the build? Can you explain what is involved?

[chrisjbillington]

Sorry for the delay, there was not much work getting 0.4 released, I just didn't get to it last week. But, I have now released it. It works a little bit differently (requires you to run your script with python -m gil_load prefixed on the command line), so you should check the readme for instructions.

Yes, I could make a 0.3.x release that fixed the build issue, but I am not inclined to do this unless there is a good reason, since 0.4 supersedes 0.3. Let me know if this is something important to you, otherwise hopefully 0.4 works for you!

[estein-de]

Ah. This does not help me particularly much, as I'm running mod_wsgi. It does not look like this file is intended to be used in such an environment. Any tips on how to make this work in a nice way?

The gil_load/__main__.py file contains (as of 0.4):

import gil_load
import os
import sys

preload_path = gil_load.preload_path

LD_PRELOAD = os.getenv('LD_PRELOAD')
if LD_PRELOAD is not None:
    preload_path += ':' + LD_PRELOAD

os.environ['LD_PRELOAD'] = preload_path

os.execv(sys.executable, [sys.executable] + sys.argv[1:])

[chrisjbillington]

You could manually set LD_PRELOAD in your environment prior to starting the web server. Or, is there a way to set environment variables for the Python process(es) spawned by mod_wsgi? In either case, setting LD_PRELOAD to gil_load.preload_path (which is just the path to preload.so within the gil_load install directory) will do the trick. That's all gil_load cares about, and __main__.py is just a wrapper to set the environment variable as you can see. So long as LD_PRELOAD is set to contain preload.so, you can run your code as normal in whatever environment it is in, and gil_load should work.

It would be preferable to set it only for the Python process, since the patched libc functions will slow down your web server slightly and I think running LD_PRELOAD in a web server is probably not a good idea security-wise since it increases the attack surface to include bugs in my code (though of course that is true to a lesser extent regardless of how you use my code!).

The only thing is that the environment variable must be set prior to Python process startup.

If this is still a pain in the neck, I can modify the code so that if LD_PRELOAD has not been set it can still monitor overall GIL load, but not the contention or individual thread statistics.

[chrisjbillington]

According to the mod_wsgi docs: "The SetEnv directive has no effect on ‘os.environ’ and there is no way through Apache configuration directives to affect what is in the process environment."

It sounds like the Python interpreter for mod_wsgi runs in the same process as Apache. So I think this leaves the only option as setting LD_PRELOAD for the Apache process prior to running it.

Tricky. I'll have a bit of a think as to whether there is anything I can do to gil_load to make it work in contexts like this! I suspect there isn't, other than dropping the per-thread statistics.

[estein-de]

mod_wsgi does not always run in the same process as apache. I'm using the WSGIDaemonProcess option for example.

[estein-de]

Any further thoughts on this @chrisjbillington ?

[chrisjbillington]

A few thoughts:

Environment variables are inherited, so if you set LD_PRELOAD in a way that affected Apache as a whole, it would affect the Python interpreter too whether it's a child process or otherwise. The drawback is that it would slow the web server down slightly by overriding some C stdlib functions, and possibly this might confuse some debuggers or other tools. I wouldn't expect the slowdown to be major though.

I don't know how mod_wsgi works, but can you control the Python process shortly after startup, at all? If so, you could set LD_PRELOAD (this would have no effect on the current process), and then fork, and LD_PRELOAD would take effect in the child process resulting from the fork. Though forking might not be compatible with assumptions made by mod_wsgi, I'm not sure.

Other tools can track the GIL, such as py-spy: https://github.com/benfred/py-spy. Does that suit your needs? You can attach it to a running process by PID.