Miner build fails because builder cannot remove non-empty .gradle directory

[t-nelis]

Background info:

$ git rev-parse HEAD
86eb731ee76446328769fbc018d3553d4788a154
$ date --utc
Sat Apr 28 13:33:33 UTC 2018
$ sudo docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: <unknown>
 Go version:      go1.8.4
 Git commit:      584d391/1.13.1
 Built:           Thu Nov 23 21:40:58 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: <unknown>
 Go version:      go1.8.4
 Git commit:      584d391/1.13.1
 Built:           Thu Nov 23 21:40:58 2017
 OS/Arch:         linux/amd64
 Experimental:    false

docker-compose build --pull miner fails with:

...
------------------------------------------------------------
Root project
------------------------------------------------------------
...
BUILD SUCCESSFUL in 1m 48s
8 actionable tasks: 7 executed, 1 up-to-date
rm: can't remove './.gradle': Directory not empty
ERROR: Service 'miner' failed to build: The command '/bin/sh -c apk add --update git                                     &&   git clone --depth 1 -b research/casper https://github.com/ethereum/ethereumj &&   git clone --depth 1 -b research/casper https://github.com/ether-camp/ethereum-harmony &&   (cd ./ethereumj && ./gradlew clean install -x test && cd ..)         &&   (cd ./ethereum-harmony && ./gradlew clean dependencyManagement distTar -x test -PuseMavenLocal && cd ..) &&   tar -xf ./ethereum-harmony/build/distributions/harmony.ether.camp.tar -C ./ &&   rm -rf ethereumj ethereum-harmony &&   rm -rf ./.gradle ./.m2 &&   apk del git' returned a non-zero code: 1

The offending files:

$ sudo docker container run --rm --entrypoint=find $(sudo docker container commit 1a2b8091e7f2) .gradle
.gradle
.gradle/daemon
.gradle/daemon/4.6
.gradle/daemon/4.6/registry.bin.lock
.gradle/daemon/4.6/registry.bin

It doesn't look like the failing rm process is the one run in the Dockerfile; the same command (with -r) succeeds, as expected:

$ sudo docker container run --rm --interactive --tty --entrypoint=bash 697a4659cb57
bash-4.4# rm -rf .gradle
bash-4.4# echo $?
0

Can't find other relevant invocations in this repository however.

$ grep --recursive --fixed-string .gradle
./validator/Dockerfile:  rm -rf ./.gradle ./.m2 && \
./miner/Dockerfile:  rm -rf ./.gradle ./.m2 && \
./node/Dockerfile:  rm -rf ./.gradle ./.m2 && \

On a hunch I decided to naively remove the invocation in the Dockerfile in order to confirm.

$ git diff
diff --git a/casper/miner/Dockerfile b/casper/miner/Dockerfile
index 1722e32..949718b 100644
--- a/casper/miner/Dockerfile
+++ b/casper/miner/Dockerfile
@@ -16,7 +16,6 @@ RUN \
   (cd ./ethereum-harmony && ./gradlew clean dependencyManagement distTar -x test -PuseMavenLocal && cd ..) && \
   tar -xf ./ethereum-harmony/build/distributions/harmony.ether.camp.tar -C ./ && \
   rm -rf ethereumj ethereum-harmony && \
-  rm -rf ./.gradle ./.m2 && \
   apk del git
 
 # Inject config and startup script

The build succeeded.

Successfully built b7e546667e99

I'm a tad perplexed by this. Can anybody reproduce? (If not feel free to close this -- this doesn't look right.)

Thanks.

[mkalinin]

I can reproduce it with just docker-compose build. Will dig in that problem later a bit