Migration to conda-build 3 and new compilers from Anaconda

[msarahan]

Mega-thread for discussion and issue tracking with migration.

Currently known things to address:

• how does toolchain package translate to compiler activation flags with new compilers?
• Pinning is managed with conda_build_config.yaml files in conda-build 3. How to migrate conda-forge's existing pinning scheme to that?
• Compatibility of Anaconda compiler's flags with conda-forge's existing defaults: are the default flags in the anaconda packages good enough that conda-forge need not carry its own customization? Note: customization risks incompatibility.
• conda-build-all and job computation. CB3 has native matrix support, and outputs jobs as rendered MetaData objects. In Anaconda's CI system, we dump the conda_build_config.yaml files from those rendered objects, and use those dumped conda_build_config.yaml files to build the original templated recipe. We probably need a way to output temporary folders for the CI's to process as jobs. CB3's matrix support does not use env vars (but could).
• Decide on usage of run_exports. run_exports essentially is the feature where a package declares what its runtime needs are if anyone uses that package at build time. This feature can make pinning much simpler, but perhaps makes recipes harder to read, because runtime dependencies become indirectly specified. Anaconda's recipes use run_exports very liberally.

Added 10/18/2017:

• Implement windows activation script wrappers that are compatible with Appveyor's VS situation

Lots more I'm probably missing here, so please comment and/or modify this issue.

[ocefpaf]

1 and 3 are kind of related and I would love to hear @isuruf's thoughts on those.

2 is, IMO, a no brainier. We kill the current scheme, serve a canonical conda_build_config.yaml file somewhere (maybe the conda build setup will be useful for this) and, in some rare cases, use a local file to override the default one.

It kind of solves the bot issue we have at the moment but we would need to trigger rebuilds when conda_build_config.yaml is updated. Or just let things "break" and re-build as needed (not so different from what happens right now with the dead bot).

A long term solution would be a service that rebuilds stuff following the dependency tree. One of my ideas for the Google Summer of Code 2018 BTW.

I don't have an option on 4 besides outsourcing as much as we can from conda-build-all to conda-build.

[isuruf]

how does toolchain package translate to compiler activation flags with new compilers?

I think we can remove toolchain. Do the new compilers have flags like -I${PREFIX}/include and -Wl,-rpath,${PREFIX}/lib ?

[msarahan]

I don't think those are defaults. I have needed to add them to some recipes. @mingwandroid, can you tell us why they are not defaults? Are they unsafe in some cases?

[mingwandroid]

I would rather not add -I${PREFIX}/include or -L${PREFIX}/lib because that un-generalizes the compilers.

[mingwandroid]

Same for -Wl,-rpath,... these things should be added when needed.

[isuruf]

I would rather not add -I${PREFIX}/include or -L${PREFIX}/lib because that un-generalizes the compilers.

I always think of $PREFIX as similar to /usr where the system compiler gcc checks for headers and libraries. In that case it makes sense for the gcc compiler in a conda-build environment to check in $PREFIX. This doesn't have to be in a normal conda environment, but can be done in a conda-build environment, just like we disable setuptools downloading in a conda-build environment. (I don't know anything about the new compilers, so I don't have any idea how this can be done)

[mingwandroid]

Adding a check for $CONDA_BUILD would be 'ok' in my opinion, still I do not like passing flags to builds that do not need them and serve no purpose.

[msarahan]

I would like to add a check for $CONDA_BUILD and add these flags when true. When builds fail due to missing this, it is an annoying hurdle for people to learn about. I will try to add this into conda-build today.

[jakirkham]

By adding into conda-build, do you mean setting environment variables for header and library search paths? If so, strong +1 from me. 😄

[msarahan]

In the activate.sh scripts provided by the compiler wrapper packages, gcc_linux-64 & clang_osx-64

if [ "$CONDA_BUILD" == "1" ]; then
    CFLAGS="$CFLAGS -I$PREFIX/include"
    LDFLAGS="$LDFLAGS -L$PREFIX/lib -Wl,-rpath,$PREFIX/lib"
fi

CXXFLAGS would be set equivalently in activate scripts for the CXX packages.

(open to suggestions on reordering arguments)

[msarahan]

So, I guess to clarify, this isn't going to be set by conda-build, per se. Conda-build will set CONDA_BUILD, and then the compiler activate scripts will behave a little differently to add these flags.

[mingwandroid]

A good example of where -I${PREFIX}/include and -L${PREFIX}/lib does cause real trouble is when you have a dependency on a package that depends on libiconv on linux. This package should not exist on linux since glibc implements iconv and libiconv was extracted from it.

Adding those flags causes an undeclared, unnecessary and incorrect dependency to creep in to your build. If the package that did depend on libiconv is later corrected so that it doesn't depend on libiconv, you end up with a broken package that links to iconv symbols it never should've linked to (and worse, has a DT_NEEDED entry of libiconv.so).

So I'm still reluctantly saying 'ok', but it's really wrong IMHO.

[msarahan]

Yeah, too bad there's no completely clear win here. I think the benefits of adding it outweigh the costs, but there also needs to be a way to disable it.