Automated docker build

[holiman]

There should be an automated docker build at hub.docker.com/r/ethereum .

• Define a docker file
• Set up an automated build

Preferrably using a multistage build to get the size down as much as possible.
Examples:

• Multistage cpp-ethereum (though, seems to be failing): https://hub.docker.com/r/ethereum/eth/~/dockerfile/
• Old cpp-ethereum: https://hub.docker.com/r/ethereum/client-go/~/dockerfile/

[holiman]

When trying to build with a multistage docker file:

RUN apk add --no-cache \
        git \
        cmake \
        g++ \
        make \
        linux-headers \
        leveldb-dev --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/
#RUN git clone --recursive https://github.com/ethereum/cpp-ethereum --branch develop --single-branch --depth 1
RUN git clone --recursive https://github.com/ethereum/testeth --branch develop --single-branch --depth 1
RUN mkdir /build
#todo -DSTATIC_LINKING=1  ??
RUN cd /build && cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo  -DEVM_OPTIMIZE=0
RUN cd /build && make -j8 && make install
# Install stage 
FROM alpine:latest 
RUN apk add --no-cache \
        libstdc++ \
        leveldb --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/
COPY --from=builder /usr/bin/testeth /usr/bin/testeth
ENTRYPOINT ["/usr/bin/testeth"]

I ran into the following error:

common.mkdir bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux
gcc.compile.c++ bin.v2/libs/filesystem/build/gcc-6.3.0/release/link-static/threading-multi/portability.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux/pin_thread.o
In file included from /usr/include/pthread.h:30:0,
                 from libs/fiber/src/numa/linux/pin_thread.cpp:10:
libs/fiber/src/numa/linux/pin_thread.cpp: In function 'void boost::fibers::numa::pin_thread(uint32_t)':
libs/fiber/src/numa/linux/pin_thread.cpp:27:5: error: 'memset' was not declared in this scope
     CPU_ZERO( & set);
     ^

    "/usr/bin/c++" "-std=c++11"   -O3 -finline-functions -Wno-inline -Wall -pthread -m64  -DBOOST_ALL_NO_LIB=1 -DBOOST_DISABLE_ASSERTS -DBOOST_FIBERS_SOURCE -DBOOST_FILESYSTEM_STATIC_LINK=1 -DBOOST_SYSTEM_STATIC_LINK=1 -DNDEBUG  -I"." -c -o "bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux/pin_thread.o" "libs/fiber/src/numa/linux/pin_thread.cpp"

...failed gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux/pin_thread.o...
common.mkdir bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo
gcc.compile.c++ bin.v2/libs/filesystem/build/gcc-6.3.0/release/link-static/threading-multi/utf8_codecvt_facet.o
gcc.compile.c++ bin.v2/libs/filesystem/build/gcc-6.3.0/release/link-static/threading-multi/operations.o
gcc.archive bin.v2/libs/filesystem/build/gcc-6.3.0/release/link-static/threading-multi/libboost_filesystem-mt.a
common.copy stage/lib/libboost_filesystem-mt.a
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/algorithm.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/round_robin.o
common.mkdir bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/numa
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/shared_work.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/work_stealing.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/barrier.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/algo/numa/work_stealing.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/condition_variable.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/future.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/fiber.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/context.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/mutex.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/properties.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/recursive_mutex.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/recursive_timed_mutex.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/timed_mutex.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/scheduler.o
gcc.compile.c++ bin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi/numa/linux/topology.o
...skipped <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>libboost_fiber-mt.a(clean) for lack of <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>numa/linux/pin_thread.o...
...skipped <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>libboost_fiber-mt.a for lack of <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>numa/linux/pin_thread.o...
...skipped <pstage/lib>libboost_fiber-mt.a for lack of <pbin.v2/libs/fiber/build/gcc-6.3.0/release/link-static/threading-multi>libboost_fiber-mt.a...
...failed updating 2 targets...
...skipped 6 targets...
...updated 124 targets...
make[2]: *** [CMakeFiles/Boost-fiber.dir/build.make:114: Boost-fiber-prefix/src/Boost-fiber-stamp/Boost-fiber-build] Error 1
make[1]: *** [CMakeFiles/Makefile2:68: CMakeFiles/Boost-fiber.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

[hunter ** FATAL ERROR **] Build step failed (dir: /root/.hunter/_Base/a412c45/e35271b/2450d4d/Build/Boost/__fiber
[hunter ** FATAL ERROR **] [Directory:/root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/projects/Boost/fiber]

------------------------------ WIKI -------------------------------
    https://github.com/ruslo/hunter/wiki/error.external.build.failed
-------------------------------------------------------------------

CMake Error at /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/modules/hunter_wiki.cmake:12 (message):
Call Stack (most recent call first):
  /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/modules/hunter_fatal_error.cmake:20 (hunter_wiki)
  /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/modules/hunter_download.cmake:648 (hunter_fatal_error)
  /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/projects/Boost/fiber/hunter.cmake:16 (hunter_download)
  /root/.hunter/_Base/Download/Hunter/0.19.232/a412c45/Unpacked/cmake/modules/hunter_add_package.cmake:53 (include)
  CMakeLists.txt:106 (hunter_add_package)


-- Configuring incomplete, errors occurred!
See also "/build/CMakeFiles/CMakeOutput.log".
See also "/build/CMakeFiles/CMakeError.log".
The command '/bin/sh -c cd /build && cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo  -DEVM_OPTIMIZE=0' returned a non-zero code: 1

[holiman]

This works:

cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo -DTOOLS=OFF -DTESTS=OFF

This does not :

cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo  -DEVM_OPTIMIZE=0

I'm guessing either TOOLS or TESTS cause the failure above. cc @chfast

[holiman]

Seems to fail when I leave out -DTOOLS=OFF

[holiman]

When I use TESTS (but TOOLS disabled), the cmake works, but make fails with the following error :

[ 97%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libtesteth/blockchainTest.cpp.o
[ 98%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libtesteth/testHelperTest.cpp.o
[ 98%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3core/memorydb.cpp.o
[ 99%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3core/overlaydb.cpp.o
[ 99%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3jsonrpc/AccountHolder.cpp.o
[ 99%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3jsonrpc/Client.cpp.o
make[2]: *** No rule to make target 'deps/lib/libbinaryen.a', needed by 'test/testeth'.  Stop.
make[2]: *** Waiting for unfinished jobs....
[100%] Building CXX object test/CMakeFiles/testeth.dir/unittests/libweb3jsonrpc/jsonrpc.cpp.o
make[1]: *** [CMakeFiles/Makefile2:963: test/CMakeFiles/testeth.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
The command '/bin/sh -c cd /build && make -j2 && make install' returned a non-zero code: 2

(note^ I turned down paralellism to 2, in case it was a race issue)

[holiman]

(tried with 1 aswell, same problem)

[holiman]

It builds it correctly, but seems to install it at another place than we search for it

[100%] Built target s2wasm
[ 10%] Performing install step for 'binaryen'
[  7%] Built target wasm
[ 11%] Built target ir
[ 14%] Built target asmjs
[ 16%] Built target cfg
[ 59%] Built target passes
[ 67%] Built target support
[ 70%] Built target wasm-reduce
[ 74%] Built target emscripten-optimizer
[ 77%] Built target wasm-opt
[ 80%] Built target wasm-shell
[ 82%] Built target wasm-as
[ 84%] Built target binaryen
[ 86%] Built target asm2wasm
[ 88%] Built target wasm-merge
[ 90%] Built target wasm-metadce
[ 92%] Built target wasm-dis
[ 94%] Built target wasm-ctor-eval
[ 96%] Built target wasm2asm
[100%] Built target s2wasm
Install the project...
-- Install configuration: "Release"
-- Installing: /build/deps/lib64/libbinaryen.a
-- Installing: /build/deps/include/binaryen-c.h
-- Installing: /build/deps/share/binaryen/wasm.js
-- Installing: /build/deps/share/binaryen/binaryen.js
-- Installing: /build/deps/bin/wasm-shell
-- Installing: /build/deps/bin/wasm-opt
-- Installing: /build/deps/bin/wasm-merge
-- Installing: /build/deps/bin/wasm-metadce
-- Installing: /build/deps/bin/asm2wasm
-- Installing: /build/deps/bin/wasm2asm
-- Installing: /build/deps/bin/s2wasm
-- Installing: /build/deps/bin/wasm-as
-- Installing: /build/deps/bin/wasm-dis
-- Installing: /build/deps/bin/wasm-ctor-eval
-- Installing: /build/deps/bin/wasm-reduce
[ 10%] Completed 'binaryen'
[ 10%] Built target binaryen

Notably: Installing: /build/deps/lib64/libbinaryen.a, err msg: No rule to make target 'deps/lib/libbinaryen.a'. lib64 vs lib.

[pirapira]

try git submodule init and git submodule update.

[holiman]

Ok, finally got it working. Here's my docker image

# Build stage 
FROM alpine:latest as builder
RUN apk add --no-cache \
        git \
        cmake \
        g++ \
        make \
        linux-headers \
        leveldb-dev --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/
RUN git clone --recursive https://github.com/ethereum/testeth --branch develop --single-branch --depth 1
RUN mkdir /build
# See https://github.com/ethereum/cpp-ethereum/issues/4834
RUN cd /build && cmake /testeth -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo -DTOOLS=OFF
RUN cmake --version
RUN cd /build && make -j1 && make install; exit 0
RUN cd /build && cp -r deps/lib64/libbinaryen.a deps/lib/; cp -r deps/lib64/libmpir.a deps/lib/
RUN cd /build && make -j1 && make install

# Install stage 
FROM alpine:latest
RUN apk add --no-cache \
        libstdc++ \
        leveldb --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/
COPY --from=builder /build/test/testeth /usr/bin/testeth

#ENTRYPOINT ["/usr/bin/testeth"]

I disabled the entrypoint, but you should probably enable it if you use this as a 'regular' docker image. And you can probably set -j8 back aswell, I lowered it to ensure that the build failure wasn't due to paralellism.

[pirapira]

Thank you. I'll try the old cpp image to see what should happen.

(I've never used docker for testeth because my machine somewhere can compile testeth reasonably fast.)