Change in rocker/binder:latest stops pystan installing - sometimes

[perllaghu]

Container image name

rocker/binder:latest

Container image digest

rocker/binder@sha256:c5713a3cb8f6cc3619e99c9ea910c11e3d4aa1e97bf8b75133098ca23ab77226

What operating system are you seeing the problem on?

Linux

System information

• Two machines:
• physical hardware: Ubuntu (Linux Mint 22.2). This is my workstation
• virtual machine: Rocky Linux 9.7. This is essentially the same as the runner that does the builds for my deployment processes.
• Both running Docker 29.2.0

Bug description

Issue was exposed as part of a complex Docker build process - however I have investigated further.

Last built around August 2025, and all my images built fine. Doing a rebuild February 2026, and the pystan install fails

On the Ubuntu system, the issue does not show: all my images build fine.
On the Rocky system, the build fails with the following error:

Building wheels for collected packages: pysimdjson
  Building wheel for pysimdjson (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for pysimdjson (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [92 lines of output]
      /tmp/pip-build-env-mx61th62/overlay/lib/python3.13/site-packages/setuptools/dist.py:765: SetuptoolsDeprecationWarning: License classifiers are deprecated.
      !!
      
              ********************************************************************************
              Please consider removing the following classifiers in favor of a SPDX license expression:
      
              License :: OSI Approved :: MIT License
      
              See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
              ********************************************************************************
      
      !!
        self._finalize_license_expression()
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-313/simdjson
      copying simdjson/__init__.py -> build/lib.linux-x86_64-cpython-313/simdjson
      copying simdjson/__init__.pyi -> build/lib.linux-x86_64-cpython-313/simdjson
      copying simdjson/py.typed -> build/lib.linux-x86_64-cpython-313/simdjson
      running build_ext
      building 'csimdjson' extension
      creating build/temp.linux-x86_64-cpython-313/simdjson
      g++ -pthread -B /opt/conda/share/python_compiler_compat -fno-strict-overflow -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /opt/conda/include -fPIC -O2 -isystem /opt/conda/include -fPIC -DSIMDJSON_IMPLEMENTATION_FALLBACK=1 -I/opt/conda/include/python3.13 -c simdjson/csimdjson.cpp -o build/temp.linux-x86_64-cpython-313/simdjson/csimdjson.o
      simdjson/csimdjson.cpp:1288:72: warning: ‘Py_UNICODE’ is deprecated [-Wdeprecated-declarations]
       1288 | static CYTHON_INLINE size_t __Pyx_Py_UNICODE_strlen(const Py_UNICODE *u)
            |                                                                        ^
      In file included from /opt/conda/include/python3.13/unicodeobject.h:1014,
                       from /opt/conda/include/python3.13/Python.h:80,
                       from simdjson/csimdjson.cpp:44:
      /opt/conda/include/python3.13/cpython/unicodeobject.h:10:37: note: declared here
         10 | Py_DEPRECATED(3.13) typedef wchar_t Py_UNICODE;
            |                                     ^~~~~~~~~~
      simdjson/csimdjson.cpp: In function ‘size_t __Pyx_Py_UNICODE_strlen(const Py_UNICODE*)’:
      simdjson/csimdjson.cpp:1290:23: warning: ‘Py_UNICODE’ is deprecated [-Wdeprecated-declarations]
       1290 |     const Py_UNICODE *u_end = u;
            |                       ^~~~~
      /opt/conda/include/python3.13/cpython/unicodeobject.h:10:37: note: declared here
         10 | Py_DEPRECATED(3.13) typedef wchar_t Py_UNICODE;
            |                                     ^~~~~~~~~~
      simdjson/csimdjson.cpp: In function ‘int __Pyx_PyInt_As_int(PyObject*)’:
      simdjson/csimdjson.cpp:36507:42: error: too few arguments to function ‘int _PyLong_AsByteArray(PyLongObject*, unsigned char*, size_t, int, int, int)’
      36507 |                 ret = _PyLong_AsByteArray((PyLongObject *)v,
            |                       ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~
      36508 |                                            bytes, sizeof(val),
            |                                            ~~~~~~~~~~~~~~~~~~~
      36509 |                                            is_little, !is_unsigned);
            |                                            ~~~~~~~~~~~~~~~~~~~~~~~~
      In file included from /opt/conda/include/python3.13/longobject.h:107,
                       from /opt/conda/include/python3.13/Python.h:82:
      /opt/conda/include/python3.13/cpython/longobject.h:111:17: note: declared here
        111 | PyAPI_FUNC(int) _PyLong_AsByteArray(PyLongObject* v,
            |                 ^~~~~~~~~~~~~~~~~~~
      simdjson/csimdjson.cpp: In function ‘size_t __Pyx_PyInt_As_size_t(PyObject*)’:
      simdjson/csimdjson.cpp:36780:42: error: too few arguments to function ‘int _PyLong_AsByteArray(PyLongObject*, unsigned char*, size_t, int, int, int)’
      36780 |                 ret = _PyLong_AsByteArray((PyLongObject *)v,
            |                       ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~
      36781 |                                            bytes, sizeof(val),
            |                                            ~~~~~~~~~~~~~~~~~~~
      36782 |                                            is_little, !is_unsigned);
            |                                            ~~~~~~~~~~~~~~~~~~~~~~~~
      /opt/conda/include/python3.13/cpython/longobject.h:111:17: note: declared here
        111 | PyAPI_FUNC(int) _PyLong_AsByteArray(PyLongObject* v,
            |                 ^~~~~~~~~~~~~~~~~~~
      simdjson/csimdjson.cpp: In function ‘long int __Pyx_PyInt_As_long(PyObject*)’:
      simdjson/csimdjson.cpp:37239:42: error: too few arguments to function ‘int _PyLong_AsByteArray(PyLongObject*, unsigned char*, size_t, int, int, int)’
      37239 |                 ret = _PyLong_AsByteArray((PyLongObject *)v,
            |                       ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~
      37240 |                                            bytes, sizeof(val),
            |                                            ~~~~~~~~~~~~~~~~~~~
      37241 |                                            is_little, !is_unsigned);
            |                                            ~~~~~~~~~~~~~~~~~~~~~~~~
      /opt/conda/include/python3.13/cpython/longobject.h:111:17: note: declared here
        111 | PyAPI_FUNC(int) _PyLong_AsByteArray(PyLongObject* v,
            |                 ^~~~~~~~~~~~~~~~~~~
      simdjson/csimdjson.cpp: In function ‘char __Pyx_PyInt_As_char(PyObject*)’:
      simdjson/csimdjson.cpp:37636:42: error: too few arguments to function ‘int _PyLong_AsByteArray(PyLongObject*, unsigned char*, size_t, int, int, int)’
      37636 |                 ret = _PyLong_AsByteArray((PyLongObject *)v,
            |                       ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~
      37637 |                                            bytes, sizeof(val),
            |                                            ~~~~~~~~~~~~~~~~~~~
      37638 |                                            is_little, !is_unsigned);
            |                                            ~~~~~~~~~~~~~~~~~~~~~~~~
      /opt/conda/include/python3.13/cpython/longobject.h:111:17: note: declared here
        111 | PyAPI_FUNC(int) _PyLong_AsByteArray(PyLongObject* v,
            |                 ^~~~~~~~~~~~~~~~~~~
      simdjson/csimdjson.cpp: In function ‘PyObject* __Pyx_PyGen_Send(PyGenObject*, PyObject*)’:
      simdjson/csimdjson.cpp:38061:13: error: ‘_PyGen_SetStopIterationValue’ was not declared in this scope; did you mean ‘__Pyx_PyGen__FetchStopIterationValue’?
      38061 |             _PyGen_SetStopIterationValue(result);
            |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
            |             __Pyx_PyGen__FetchStopIterationValue
      error: command '/usr/bin/g++' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pysimdjson
Failed to build pysimdjson
error: failed-wheel-build-for-install

× Failed to build installable wheels for some pyproject.toml based projects
╰─> pysimdjson

Testing process

• If I run the pip install pystan within a virtual environment on the rocker machine, it installs fine
• If I create a bare rocker container with docker run -it --rm rocker/binder:latest bash and do the pip install pystan it fails
• If I create a bare rocker container with docker run -it --rm rocker/binder:4.4.2 bash and do the pip install pystan it installs fine
• If I create a bare rocker container with docker run -it --rm rocker/binder:no-xdg-set bash and do the pip install pystan it also installs fine.

... ergo, some change in the image between July 2025 & January 2026 is creating a flaw.

How to reproduce this bug?

`docker run -it --rm rocker/binder:latest bash` then `pip install pystan` at the command prompt

[eitsupi]

I think the base image has changed completely in that time, maybe you need to use mamba?

[cboettig]

@eitsupi is right, rocker/binder is now based on the jupyterhub team's base images, which are conda based, though conda also supports pip installs.

I think the reason for the failure is that the new base image is on python 3.13, and pystan is currently only supporting up through python 3.12.

For what it's worth, rocker/ml images also combine python support with R/RStudio, but are pip based and run on python 3.12, pystan is installing fine there for me.

[perllaghu]

This may be a higher-level thing that just rocker/binder - but can you add functionally specific tags to images?

My problem stemmed from building from :latest [where there were no other options], then latest changed under me.

[eitsupi]

What do you mean of "functionally specific" ?

There are no others, as we no longer build rocker/binder images based on the previous base image.

You can pin it if you really want it to stay in place.
https://rocker-project.org/use/reproducibility.html

[eitsupi]

Related to rocker-org/rocker-versioned2#903

[eitsupi]

IIUC, the pysimdjson binaries for Python 3.13 are available on conda-forge, so I don't see any reason why you can't just install it with conda/mamba.
https://anaconda.org/channels/conda-forge/packages/pysimdjson/files

[perllaghu]

IIUC, the pysimdjson binaries for Python 3.13 are available on conda-forge, so I don't see any reason why you can't just install it with conda/mamba

That is my next piece of work, yes.

.... however persistent tags are generally a useful thing for those of us building upon base images :)

[cboettig]

.... however persistent tags are generally a useful thing for those of us building upon base images :)

I believe all images in rocker are built to GitHub Container Registry as well as docker hub. GHCR has dated hash tags of all versions, e.g. for rocker/binder https://github.com/rocker-org/rocker-versioned2/pkgs/container/binder/versions

I find this is generally more robust than tagging by specific versions of say, R or python when it comes to these more complexly orchestrated packages like binder. We could certainly document the existence if these a bit better.

[perllaghu]

What do you mean of "functionally specific" ?
With Jupyter, they seem to have multiple tags, some of which will update to point to the latest iteration of that "function": python-3.13 will point to the lastest build with 3.13... and will creep up with each semvar patch, whilst python 3.13.7 will not be updated when they switch to python 3.13.8.

The one that's most useful for me is the dated tag - eg: 2025-12-08.
This way I know when the last working version was built from.... and I may expect complications if I build from there.

Jupyter certainly use more tags that rocker will want... For rocker/binder, it could be useful to track a date, R versions, jupyterlab versions (I've been bitten when they muck around with how extensions work), mamba/conda/python versions?

[cboettig]

@perllaghu thanks, dates are included on GHCR though.

I hear you on tags of versions, but it really is not trivial. for some users it's enough to know python version, or python version and jupyter version. To this we could add the R version, and RStudio version. Of course knowing that it's a ubuntu base and the ubuntu version is also critical. Then, there's other key software, likewise not locked to distro version, like the GDAL version. Then there's whether the image is conda or not. In the ML space we also have CUDA versions. What's really important to one user is often not to another user. When you start doing more than 3 version tags in a docker image tag the complexity gets out of hand quickly.

For more narrowly scoped images a specific version in tags is fine, but I have not found that it scales easily here. Like you say, it's much easier to go back to the date of the specific image. GHCR.io is built with this in mind, the images have immutable hashes as tags, and timestamps in the metadata, much like git itself. Moreover, this pattern is universal to GHCR, and a natural extension of the open containers registry specification rather than a tagging convention we just invent here.

Using hash-tagged versions in production deployments is a widely used practice, e.g. you can see this in 2i2c's deployment of how they manage jupyterhubs.