Skip to content

rebuild - #6324

Closed
jburel wants to merge 4 commits into
ome:developfrom
jburel:double-build
Closed

rebuild#6324
jburel wants to merge 4 commits into
ome:developfrom
jburel:double-build

Conversation

@jburel

@jburel jburel commented Jun 9, 2022

Copy link
Copy Markdown
Member

What this PR does

This PR highlights the failure introduced by the ZARR reader.

Running ./build.py will work in the .m2 folder is empty.
If it is not it will fail with

::::::::::::::::::::::::::::::::::::::::::::::
		::              FAILED DOWNLOADS            ::
		:: ^ see resolution messages for details  ^ ::
		::::::::::::::::::::::::::::::::::::::::::::::
		:: io.netty#netty-transport-native-epoll;4.1.46.Final!netty-transport-native-epoll.jar

This PR introduced a rebuild step to highlight the issue
A failure is expected.

cc @sbesson

@jburel

jburel commented Jun 14, 2022

Copy link
Copy Markdown
Member Author

I did another workflow:

  • remove .m2
  • change version in omero.properties of versions.OMEZarrReader to 0.2.1-SNAPSHOT
  • build openmicroscopy. This will fail with ZarrReader 0.2.1-SNAPSHOT not found (expected)
  • build ZarrReader
  • rebuild openmicroscopy
  • different error in that case
		::::::::::::::::::::::::::::::::::::::::::::::
		::              FAILED DOWNLOADS            ::
		:: ^ see resolution messages for details  ^ ::
		::::::::::::::::::::::::::::::::::::::::::::::
		:: junit#junit;4.12!junit.jar
		:: org.hamcrest#hamcrest-core;1.3!hamcrest-core.j

@sbesson

sbesson commented Jun 20, 2022

Copy link
Copy Markdown
Member

Tested various scenarios using the following Dockerfile

FROM ubuntu:20.04

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
    openjdk-8-jdk \
    git \ 
    maven \
    libmcpp-dev \
    python3-pip \
    wget
RUN wget -q https://github.com/ome/zeroc-ice-ubuntu2004/releases/download/0.2.0/ice-3.6.5-0.2.0-ubuntu2004-amd64.tar.gz && \
  tar xf ice-3.6.5-0.2.0-ubuntu2004-amd64.tar.gz && \
  mv ice-3.6.5-0.2.0 ice-3.6.5 && \
  mv ice-3.6.5 /opt && \
  rm ice-3.6.5-0.2.0-ubuntu2004-amd64.tar.gz
ENV PATH=/opt/ice-3.6.5/bin:$PATH

RUN git clone https://github.com/ome/openmicroscopy /src/openmicroscopy --depth 1
RUN git clone https://github.com/ome/bioformats /src/bioformats --depth 1
RUN git clone https://github.com/ome/ZarrReader /src/ZarrReader

# Step 1: build Bio-Formats
WORKDIR /src/bioformats
RUN mvn install -DskipTests

# Step 2: build ZarrReader
WORKDIR /src/ZarrReader
#RUN git fetch origin pull/38/merge && git checkout FETCH_HEAD
#RUN mvn install -DskipTests

# Step 3: build OMERO
WORKDIR /src/openmicroscopy
RUN python3 build.py
RUN python3 build.py
  • with step 3 i.e. building OMERO only (step 3), the build fails during the copy-server phase of the second execution
copy-server:
:: problems summary ::
:::: WARNINGS
		[NOT FOUND  ] io.netty#netty-transport-native-epoll;4.1.46.Final!netty-transport-native-epoll.jar (1ms)
	==== user-maven: tried
	  /root/.m2/repository/io/netty/netty-transport-native-epoll/4.1.46.Final/netty-transport-native-epoll-4.1.46.Final.jar
		::::::::::::::::::::::::::::::::::::::::::::::
		::              FAILED DOWNLOADS            ::
		:: ^ see resolution messages for details  ^ ::
		::::::::::::::::::::::::::::::::::::::::::::::
		:: io.netty#netty-transport-native-epoll;4.1.46.Final!netty-transport-native-epoll.jar
		::::::::::::::::::::::::::::::::::::::::::::::
  • with step 2+3 i.e. building the ZarrReader (current HEAD) + OMERO, the build fails during the retrieve phase of the first build execution
retrieve:
:: Apache Ivy 2.4.0 - 20141213170938 :: http://ant.apache.org/ivy/ ::
:: loading settings :: file = /src/openmicroscopy/etc/ivysettings.xml
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.ivy.util.url.IvyAuthenticator (file:/src/openmicroscopy/lib/repository/ivy-2.4.0.jar) to field java.net.Authenticator.theAuthenticator
WARNING: Please consider reporting this to the maintainers of org.apache.ivy.util.url.IvyAuthenticator
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
:: problems summary ::
:::: WARNINGS
		[NOT FOUND  ] joda-time#joda-time;2.2!joda-time.jar (0ms)
	==== user-maven: tried
	  /root/.m2/repository/joda-time/joda-time/2.2/joda-time-2.2.jar
		[NOT FOUND  ] com.google.guava#guava;29.0-jre!guava.jar(bundle) (0ms)
	==== user-maven: tried
	  /root/.m2/repository/com/google/guava/guava/29.0-jre/guava-29.0-jre.jar
		[NOT FOUND  ] org.objenesis#objenesis;2.5.1!objenesis.jar (0ms)
	==== user-maven: tried
	  /root/.m2/repository/org/objenesis/objenesis/2.5.1/objenesis-2.5.1.jar
		[NOT FOUND  ] org.checkerframework#checker-qual;2.11.1!checker-qual.jar (0ms)
	==== user-maven: tried
	  /root/.m2/repository/org/checkerframework/checker-qual/2.11.1/checker-qual-2.11.1.jar
		[NOT FOUND  ] com.google.errorprone#error_prone_annotations;2.3.4!error_prone_annotations.jar (0ms)
	==== user-maven: tried
	  /root/.m2/repository/com/google/errorprone/error_prone_annotations/2.3.4/error_prone_annotations-2.3.4.jar
		::::::::::::::::::::::::::::::::::::::::::::::
		::              FAILED DOWNLOADS            ::
		:: ^ see resolution messages for details  ^ ::
		::::::::::::::::::::::::::::::::::::::::::::::
		:: org.objenesis#objenesis;2.5.1!objenesis.jar
		:: joda-time#joda-time;2.2!joda-time.jar
		:: com.google.guava#guava;29.0-jre!guava.jar(bundle)
		:: org.checkerframework#checker-qual;2.11.1!checker-qual.jar
		:: com.google.errorprone#error_prone_annotations;2.3.4!error_prone_annotations.jar
		::::::::::::::::::::::::::::::::::::::::::::::  
  • with steps 2+3 i.e. building ZarrReader (with the deps PR) + OMERO, the build fails during the retrieve phase of the first execution
retrieve:
:: Apache Ivy 2.4.0 - 20141213170938 :: http://ant.apache.org/ivy/ ::
:: loading settings :: file = /src/openmicroscopy/etc/ivysettings.xml
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.ivy.util.url.IvyAuthenticator (file:/src/openmicroscopy/lib/repository/ivy-2.4.0.jar) to field java.net.Authenticator.theAuthenticator
WARNING: Please consider reporting this to the maintainers of org.apache.ivy.util.url.IvyAuthenticator
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
:: problems summary ::
:::: WARNINGS
		[NOT FOUND  ] joda-time#joda-time;2.2!joda-time.jar (0ms)
	==== user-maven: tried
	  /root/.m2/repository/joda-time/joda-time/2.2/joda-time-2.2.jar
		[NOT FOUND  ] com.google.guava#guava;29.0-jre!guava.jar(bundle) (0ms)
	==== user-maven: tried
	  /root/.m2/repository/com/google/guava/guava/29.0-jre/guava-29.0-jre.jar
		[NOT FOUND  ] ch.qos.logback#logback-core;1.2.0!logback-core.jar (0ms)
	==== user-maven: tried
	  /root/.m2/repository/ch/qos/logback/logback-core/1.2.0/logback-core-1.2.0.jar
		[NOT FOUND  ] org.objenesis#objenesis;2.5.1!objenesis.jar (0ms)
	==== user-maven: tried
	  /root/.m2/repository/org/objenesis/objenesis/2.5.1/objenesis-2.5.1.jar
		[NOT FOUND  ] org.checkerframework#checker-qual;2.11.1!checker-qual.jar (0ms)
	==== user-maven: tried
	  /root/.m2/repository/org/checkerframework/checker-qual/2.11.1/checker-qual-2.11.1.jar
		[NOT FOUND  ] com.google.errorprone#error_prone_annotations;2.3.4!error_prone_annotations.jar (0ms)
	==== user-maven: tried
	  /root/.m2/repository/com/google/errorprone/error_prone_annotations/2.3.4/error_prone_annotations-2.3.4.jar
		::::::::::::::::::::::::::::::::::::::::::::::
		::              FAILED DOWNLOADS            ::
		:: ^ see resolution messages for details  ^ ::
		::::::::::::::::::::::::::::::::::::::::::::::
		:: org.objenesis#objenesis;2.5.1!objenesis.jar
		:: joda-time#joda-time;2.2!joda-time.jar
		:: com.google.guava#guava;29.0-jre!guava.jar(bundle)
		:: org.checkerframework#checker-qual;2.11.1!checker-qual.jar
		:: com.google.errorprone#error_prone_annotations;2.3.4!error_prone_annotations.jar
		:: ch.qos.logback#logback-core;1.2.0!logback-core.jar
		::::::::::::::::::::::::::::::::::::::::::::::


:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
  • with steps 1+2+3 building Bio-Formats (current HEAD) + ZarrReader (with the deps PR) +OMERO, the build fails during the retrieve phase of the first execution
retrieve:
:: Apache Ivy 2.4.0 - 20141213170938 :: http://ant.apache.org/ivy/ ::
:: loading settings :: file = /src/openmicroscopy/etc/ivysettings.xml
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.ivy.util.url.IvyAuthenticator (file:/src/openmicroscopy/lib/repository/ivy-2.4.0.jar) to field java.net.Authenticator.theAuthenticator
WARNING: Please consider reporting this to the maintainers of org.apache.ivy.util.url.IvyAuthenticator
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
:: problems summary ::
:::: WARNINGS
		[NOT FOUND  ] ch.qos.logback#logback-core;1.2.0!logback-core.jar (0ms)
	==== user-maven: tried
	  /root/.m2/repository/ch/qos/logback/logback-core/1.2.0/logback-core-1.2.0.jar
		::::::::::::::::::::::::::::::::::::::::::::::
		::              FAILED DOWNLOADS            ::
		:: ^ see resolution messages for details  ^ ::
		::::::::::::::::::::::::::::::::::::::::::::::
		:: ch.qos.logback#logback-core;1.2.0!logback-core.jar
		::::::::::::::::::::::::::::::::::::::::::::::


:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

@sbesson

sbesson commented Jun 20, 2022

Copy link
Copy Markdown
Member

At least for the last few builds, the behavior is pretty consistent: The Maven build system downloads the POMs of all transitive dependencies (e.g. ch.qos.logback:logback-core:1.2.0 which is been pulled as dependency of https://github.com/ome/ome-common-java/blob/ef75101fdc1ce12b40425517f7cb3c437f8c3a25/pom.xml#L152-L156 but the dependency resolution excludes them so the JAR is never downloaded under ~/.m2.

This seems to be incompatible with assumptions of the Ivy configuration which expects both the JAR & the POM to be available and fails the resolution otherwise. Next I will try and enable verbose logging to see if the debugging statements give a hint on how to fix these resolution issues and/or adjust the upstream dependencies.

@dgault

dgault commented Jun 21, 2022

Copy link
Copy Markdown
Member

For the 1+2+3 scenario bumping logback-classic in ome-common should be enough to resolve.

Also for the other scennario JM mentioned:

remove .m2
change version in omero.properties of versions.OMEZarrReader to 0.2.1-SNAPSHOT
build openmicroscopy. This will fail with ZarrReader 0.2.1-SNAPSHOT not found (expected)
build ZarrReader
rebuild openmicroscopy
different error in that case

Resulting in

::::::::::::::::::::::::::::::::::::::::::::::
		::              FAILED DOWNLOADS            ::
		:: ^ see resolution messages for details  ^ ::
		::::::::::::::::::::::::::::::::::::::::::::::
		:: junit#junit;4.12!junit.jar
		:: org.hamcrest#hamcrest-core;1.3!hamcrest-core.j

For this scenario, bumping testng to 6.10 in ZarrReader (and I would also bump Bio-Formats) resolves the issue.

@dgault

dgault commented Jun 21, 2022

Copy link
Copy Markdown
Member

Along with the logback bump in common adding guava version 29.0-jre and joda-time version 2.2 as dependencies directly in ZarrReader seems to be enough to get sceanrio 2 + 3 building. Its not ideal but that may be a quick workaround.

@sbesson

sbesson commented Jun 21, 2022

Copy link
Copy Markdown
Member

@dgault one idea: would changing the order of the dependencies in the ZarrReader pom e.g. to pu formats-api at the top and/or adding ome-common first as in the Bio-Formats pom help in unifying the dependency order?
I agree a solution that would prevent the release of all Java components would be preferabl.

@dgault

dgault commented Jun 21, 2022

Copy link
Copy Markdown
Member

Does it mean a new BF release? this will imply releasing all the jar again

Simply adding to ZarrReader seemed to be enough to get things to build but bumping them all to keep things aligned would make sense if we were taking that approach

@jburel jburel mentioned this pull request Jun 22, 2022
@jburel

jburel commented Jun 22, 2022

Copy link
Copy Markdown
Member Author

I have replaced this PR by #6326
It combines the various options

@sbesson

sbesson commented Jun 22, 2022

Copy link
Copy Markdown
Member

As we wanted to make a decision re releasing OMERO.server 5.6.5 with/without the ZarrReader by tomorrow, what is the current consensus?
Would it make sense to try and updating the POM in ZarrReader as an incremental step and see if it unblocks the release and then schedule the unification of the dependencies in the follow-up Bio-Formats release?

@jburel jburel closed this Jun 27, 2022
@jburel
jburel deleted the double-build branch March 10, 2023 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants