Need some way to lock the version of boost to what we build against

[greglandrum]

Boost version numbers end up in the library names, so using DLLs from newer boost versions does not work.

This is connected to this rdkit issue.
rdkit/rdkit#1957

[greglandrum]

@rvianello : the immediately obvious way to do this would be to just directly match the version numbers in the host and run sections.
Can you think of a better solution?

[rvianello]

Since a conda_build_config.yamlfile is already present for the rdkit recipe, maybe we could try using the pinning mechanism that is documented to work for build variants (it shouldn't necessarily imply building multiple rdkit packages).

The section with title "Pinning at the variant level" at https://conda.io/docs/user-guide/tasks/build-packages/variants.html provides an example that specifically addresses the boost case. The following section ("Pinning at the recipe level") illustrates how to obtain a similar result in the meta.yaml file alone.

[markussitzmann]

Does this already work? I merged all changes @greglandrum and @rvianello did in their forks and build in a fresh container, it build RDKit 2018.03.3, but at the end it is missing boost-1.65.1 and the container somehow has 1.67 installed:

`The following NEW packages will be INSTALLED:

blas:            1.0-mkl                         
bzip2:           1.0.6-h14c3975_5                
ca-certificates: 2018.03.07-0                    
cairo:           1.14.12-h8948797_3              
certifi:         2018.4.16-py36_0                
fontconfig:      2.13.0-h9420a91_0               
freetype:        2.9.1-h8a8886c_0                
glib:            2.56.1-h000015b_0               
icu:             58.2-h9c2bf20_1                 
intel-openmp:    2018.0.3-0                      
jpeg:            9b-h024ee3a_2                   
libboost:        1.67.0-h46d08c1_4               
libedit:         3.1.20170329-h6b74fdf_2         
libffi:          3.2.1-hd88cf55_4                
libgcc-ng:       7.2.0-hdf63c60_3                
libgfortran-ng:  7.2.0-hdf63c60_3                
libpng:          1.6.34-hb9fc6fc_0               
libstdcxx-ng:    7.2.0-hdf63c60_3                
libtiff:         4.0.9-he85c1e1_1                
libuuid:         1.0.3-h1bed415_2                
libxcb:          1.13-h1bed415_1                 
libxml2:         2.9.8-h26e45fe_1                
mkl:             2018.0.3-1                      
mkl_fft:         1.0.4-py36h4414c95_1            
mkl_random:      1.0.1-py36h4414c95_1            
ncurses:         6.1-hf484d3e_0                  
numpy:           1.15.0-py36h1b885b7_0           
numpy-base:      1.15.0-py36h3dfced4_0           
olefile:         0.45.1-py36_0                   
openssl:         1.0.2o-h14c3975_1               
pandas:          0.23.3-py36h04863e7_0           
pcre:            8.42-h439df22_0                 
pillow:          5.2.0-py36heded4f4_0            
pip:             10.0.1-py36_0                   
pixman:          0.34.0-hceecf20_3               
py-boost:        1.67.0-py36h04863e7_4           
python:          3.6.6-hc3d631a_0                
python-dateutil: 2.7.3-py36_0                    
pytz:            2018.5-py36_0                   
rdkit:           2018.03.3.0-py36h6bb024c_1 local
readline:        7.0-ha6073c6_4                  
setuptools:      39.2.0-py36_0                   
six:             1.11.0-py36_1                   
sqlite:          3.24.0-h84994c4_0               
tk:              8.6.7-hc745277_3                
wheel:           0.31.1-py36_0                   
xz:              5.2.4-h14c3975_4                
zlib:            1.2.11-ha838bed_2               

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
import: 'rdkit'
Traceback (most recent call last):
File "/opt/conda/conda-bld/rdkit_1533517910182/test_tmp/run_test.py", line 2, in
import rdkit
File "/opt/conda/conda-bld/rdkit_1533517910182/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.6/site-packages/rdkit/init.py", line 2, in
from .rdBase import rdkitVersion as version
ImportError: libboost_python3.so.1.65.1: cannot open shared object file: No such file or directory
Tests failed for rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2 - moving package to /opt/conda/conda-bld/broken
WARNING:conda_build.build:Tests failed for rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2 - moving package to /opt/conda/conda-bld/broken
removing: rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2
TESTS FAILED: rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2
ERROR: Service 'rdkit-conda-compile' failed to build: The command '/bin/sh -c CONDA_PY=36 conda install jinja2 conda-build --yes && CONDA_PY=36 conda build boost --quiet --no-anaconda-upload && CONDA_PY=36 conda build nox --quiet --no-anaconda-upload && CONDA_PY=36 conda build cairo_nox --quiet --no-anaconda-upload && CONDA_PY=36 conda build cairocffi --quiet --no-anaconda-upload && CONDA_PY=36 conda build eigen --quiet --no-anaconda-upload && CONDA_PY=36 CONDA_NPY=112 conda build rdkit --quiet --no-anaconda-upload' returned a non-zero code: 1`

[greglandrum]

@markussitzmann : when I do a fresh build using the development branch of this recipe everything works fine. Here's what gets installed for me on a centos6 container:

(py36_rdkit) [root@414215a97f78 conda-rdkit]# conda install --use-local rdkit
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/py36_rdkit

  added / updated specs: 
    - rdkit


The following NEW packages will be INSTALLED:

    blas:            1.0-mkl                         
    bzip2:           1.0.6-h14c3975_5                
    ca-certificates: 2018.03.07-0                    
    cairo:           1.14.12-h8948797_3              
    certifi:         2018.4.16-py36_0                
    fontconfig:      2.13.0-h9420a91_0               
    freetype:        2.9.1-h8a8886c_0                
    glib:            2.56.1-h000015b_0               
    icu:             58.2-h9c2bf20_1                 
    intel-openmp:    2018.0.3-0                      
    jpeg:            9b-h024ee3a_2                   
    libboost:        1.65.1-habcd387_4               
    libedit:         3.1.20170329-h6b74fdf_2         
    libffi:          3.2.1-hd88cf55_4                
    libgcc-ng:       7.2.0-hdf63c60_3                
    libgfortran-ng:  7.2.0-hdf63c60_3                
    libpng:          1.6.34-hb9fc6fc_0               
    libstdcxx-ng:    7.2.0-hdf63c60_3                
    libtiff:         4.0.9-he85c1e1_1                
    libuuid:         1.0.3-h1bed415_2                
    libxcb:          1.13-h1bed415_1                 
    libxml2:         2.9.8-h26e45fe_1                
    mkl:             2018.0.3-1                      
    mkl_fft:         1.0.4-py36h4414c95_1            
    mkl_random:      1.0.1-py36h4414c95_1            
    ncurses:         6.1-hf484d3e_0                  
    numpy:           1.15.0-py36h1b885b7_0           
    numpy-base:      1.15.0-py36h3dfced4_0           
    olefile:         0.45.1-py36_0                   
    openssl:         1.0.2o-h14c3975_1               
    pandas:          0.23.3-py36h04863e7_0           
    pcre:            8.42-h439df22_0                 
    pillow:          5.2.0-py36heded4f4_0            
    pip:             10.0.1-py36_0                   
    pixman:          0.34.0-hceecf20_3               
    py-boost:        1.65.1-py36hf484d3e_4           
    python:          3.6.6-hc3d631a_0                
    python-dateutil: 2.7.3-py36_0                    
    pytz:            2018.5-py36_0                   
    rdkit:           2018.03.3.0-py36h71b666b_1 local
    readline:        7.0-ha6073c6_4                  
    setuptools:      39.2.0-py36_0                   
    six:             1.11.0-py36_1                   (py36_rdkit) [root@414215a97f78 conda-rdkit]# conda install --use-local rdkit
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/py36_rdkit

  added / updated specs: 
    - rdkit


The following NEW packages will be INSTALLED:

    blas:            1.0-mkl                         
    bzip2:           1.0.6-h14c3975_5                
    ca-certificates: 2018.03.07-0                    
    cairo:           1.14.12-h8948797_3              
    certifi:         2018.4.16-py36_0                
    fontconfig:      2.13.0-h9420a91_0               
    freetype:        2.9.1-h8a8886c_0                
    glib:            2.56.1-h000015b_0               
    icu:             58.2-h9c2bf20_1                 
    intel-openmp:    2018.0.3-0                      
    jpeg:            9b-h024ee3a_2                   
    libboost:        1.65.1-habcd387_4               
    libedit:         3.1.20170329-h6b74fdf_2         
    libffi:          3.2.1-hd88cf55_4                
    libgcc-ng:       7.2.0-hdf63c60_3                
    libgfortran-ng:  7.2.0-hdf63c60_3                
    libpng:          1.6.34-hb9fc6fc_0               
    libstdcxx-ng:    7.2.0-hdf63c60_3                
    libtiff:         4.0.9-he85c1e1_1                
    libuuid:         1.0.3-h1bed415_2                
    libxcb:          1.13-h1bed415_1                 
    libxml2:         2.9.8-h26e45fe_1                
    mkl:             2018.0.3-1                      
    mkl_fft:         1.0.4-py36h4414c95_1            
    mkl_random:      1.0.1-py36h4414c95_1            
    ncurses:         6.1-hf484d3e_0                  
    numpy:           1.15.0-py36h1b885b7_0           
    numpy-base:      1.15.0-py36h3dfced4_0           
    olefile:         0.45.1-py36_0                   
    openssl:         1.0.2o-h14c3975_1               
    pandas:          0.23.3-py36h04863e7_0           
    pcre:            8.42-h439df22_0                 
    pillow:          5.2.0-py36heded4f4_0            
    pip:             10.0.1-py36_0                   
    pixman:          0.34.0-hceecf20_3               
    py-boost:        1.65.1-py36hf484d3e_4           
    python:          3.6.6-hc3d631a_0                
    python-dateutil: 2.7.3-py36_0                    
    pytz:            2018.5-py36_0                   
    rdkit:           2018.03.3.0-py36h71b666b_1 local
    readline:        7.0-ha6073c6_4                  
    setuptools:      39.2.0-py36_0                   
    six:             1.11.0-py36_1                   
    sqlite:          3.24.0-h84994c4_0               
    tk:              8.6.7-hc745277_3                
    wheel:           0.31.1-py36_0                   
    xz:              5.2.4-h14c3975_4                
    zlib:            1.2.11-ha838bed_2               

Proceed ([y]/n)? 

    sqlite:          3.24.0-h84994c4_0               
    tk:              8.6.7-hc745277_3                
    wheel:           0.31.1-py36_0                   
    xz:              5.2.4-h14c3975_4                
    zlib:            1.2.11-ha838bed_2               

Proceed ([y]/n)? 

[markussitzmann]

@greglandrum Thanks for the testing and reply - although I hate it :-) ... I definitely have put boost-1.65.1 everywhere and diff'ed with yours and Riccardo's branches/forks. But somehow it ends up with boost-1.67.

I can retry only tonight. On which distro did you build? Centos? I build on debian:stretch

[greglandrum]

@markussitzmann : It also didn't work for me when I tried to force boost 1.65.1. The current branch is set to use 1.65, which grabs 1.65.1 by default.

I'm using centos, but that shouldn't make a difference.

Can you point me to the code you're using to do the build?

[markussitzmann]

While looking for the code I might have found the error I made but as I said I can only try tonight again (did the fixes in my repository but checked out the wrong branch during the Docker build - and my attempts were revolving around this 1.65 vs 1.65.1 thingy). Thanks again - I only will come back if I still struggle ;-).

[markussitzmann]

@greglandrum: hmm, when I run the development branch of this recipe here, the build works now, however now I receive boost 1.63 as built boost package and then starting rdkit fails because it is missing 1.65.1 ... a bit mysterious.

[markussitzmann]

I run this Dockerfile:

https://github.com/chembience/chembience/blob/master/context/base/rdkit-conda-compile/Dockerfile

after replacing line 44-46 with the standard conda-rdkit repository and branch development:

RUN git clone https://github.com/rdkit/conda-rdkit.git
WORKDIR conda-rdkit
RUN git checkout development

[greglandrum]

Unless you need it for something else, you can probably skip the boost build now (line 51 in your Dockerfile); that will use the boost build recipe from the RDKit channel, which will use 1.63. You almost certainly don't need to build eigen either.

I don't see anything else obviously wrong in the dockerfile.

When you get failures: do they happen during the build or later when you try and use the contents of the image?