Skip to content

Dependency updates - #38

Merged
sbesson merged 8 commits into
ome:mainfrom
dgault:dependency-updates
Jun 27, 2022
Merged

Dependency updates#38
sbesson merged 8 commits into
ome:mainfrom
dgault:dependency-updates

Conversation

@dgault

@dgault dgault commented Jun 9, 2022

Copy link
Copy Markdown
Member

Following on from conversation with @sbesson and @jburel, trying to remove some of the dependencies. Goal of this PR will be to update the below areas:

1 - formats API bumped to latest 6.10.0, this should be straight forward
2 - commons-io removed, only area impacted will be the list of used files. PR aims to keep the reproduce the same list
3 - initially remove netty-nio-client dependency, this looks safe and straight forward

As a follow up I will continue trying to strip out and remove the remaining amazon dependencies which essentially means removing the s3 functionality for the time being.

@sbesson
sbesson requested review from jburel and sbesson June 13, 2022 15:41
@jburel

jburel commented Jun 13, 2022

Copy link
Copy Markdown
Member

Building this PR in ome/openmicroscopy#6324 (I used my branch since this branch is not built on top of main and the version is old).
It now fails with new error

		::::::::::::::::::::::::::::::::::::::::::::::
		::              FAILED DOWNLOADS            ::
		:: ^ see resolution messages for details  ^ ::
		::::::::::::::::::::::::::::::::::::::::::::::
		:: org.objenesis#objenesis;2.5.1!objenesis.jar
		:: joda-time#joda-time;2.2!joda-time.jar
		:: com.google.guava#guava;29.0-jre!guava.jar(bundle)
		:: org.checkerframework#checker-qual;2.11.1!checker-qual.jar
		:: com.google.errorprone#error_prone_annotations;2.3.4!error_prone_annotations.jar
		:: ch.qos.logback#logback-core;1.2.0!logback-core.jar
		::::::::::::::::::::::::::::::::::::::::::::::

@sbesson

sbesson commented Jun 14, 2022

Copy link
Copy Markdown
Member

Have you established that removing/commenting https://github.com/ome/openmicroscopy/blob/f73008bc34d37f6034a21095c55eb2a8b6075e44/ivy.xml#L21 suffices to fix the build issues? i.e. is this component the only source of issues?

@jburel

jburel commented Jun 14, 2022

Copy link
Copy Markdown
Member

I have rebuilt using only ome/openmicroscopy#6323 and excluding the ZarrReader and it builds fine

@dgault

dgault commented Jun 14, 2022

Copy link
Copy Markdown
Member Author

Heres the dependency tree with this PR, other than logback the rest of these are related to jzarr:

[INFO] ome:OMEZarrReader:jar:0.1.6-SNAPSHOT
[INFO] +- com.bc.zarr:jzarr:jar:0.3.4:compile
[INFO] |  +- edu.ucar:cdm-core:jar:5.3.3:compile
[INFO] |  |  +- edu.ucar:udunits:jar:5.3.3:compile
[INFO] |  |  +- edu.ucar:httpservices:jar:5.3.3:compile
[INFO] |  |  |  +- org.apache.httpcomponents:httpclient:jar:4.5.9:compile
[INFO] |  |  |  |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  |  |  |  \- commons-codec:commons-codec:jar:1.11:compile
[INFO] |  |  |  \- org.apache.httpcomponents:httpmime:jar:4.5.9:compile
[INFO] |  |  +- joda-time:joda-time:jar:2.10.3:compile
[INFO] |  |  +- org.jdom:jdom2:jar:2.0.6:compile
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java:jar:3.9.1:compile
[INFO] |  |  +- com.google.guava:guava:jar:28.0-jre:compile
[INFO] |  |  |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  |  |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  |  |  +- org.checkerframework:checker-qual:jar:2.8.1:compile
[INFO] |  |  |  +- com.google.errorprone:error_prone_annotations:jar:2.3.2:compile
[INFO] |  |  |  +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] |  |  |  \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.11:compile
[INFO] |  |  \- com.google.re2j:re2j:jar:1.3:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.11.3:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.11.3:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.11.3:compile
[INFO] |  \- org.lasersonlab:jblosc:jar:1.0.1:compile
[INFO] |     \- net.java.dev.jna:jna:jar:4.2.2:compile
[INFO] +- org.mockito:mockito-inline:jar:3.7.7:compile
[INFO] |  \- org.mockito:mockito-core:jar:3.7.7:compile
[INFO] |     +- net.bytebuddy:byte-buddy:jar:1.10.19:compile
[INFO] |     +- net.bytebuddy:byte-buddy-agent:jar:1.10.19:compile
[INFO] |     \- org.objenesis:objenesis:jar:3.1:compile
[INFO] +- org.testng:testng:jar:6.8:compile
[INFO] |  +- junit:junit:jar:4.10:compile
[INFO] |  |  \- org.hamcrest:hamcrest-core:jar:1.1:compile
[INFO] |  +- org.beanshell:bsh:jar:2.0b4:compile
[INFO] |  +- com.beust:jcommander:jar:1.27:compile
[INFO] |  \- org.yaml:snakeyaml:jar:1.6:compile
[INFO] +- xalan:serializer:jar:2.7.2:runtime
[INFO] |  \- xml-apis:xml-apis:jar:1.3.04:runtime
[INFO] +- xalan:xalan:jar:2.7.2:runtime
[INFO] \- ome:formats-api:jar:6.10.0:compile
[INFO]    +- org.openmicroscopy:ome-common:jar:6.0.9:compile
[INFO]    |  +- io.minio:minio:jar:5.0.2:compile
[INFO]    |  |  +- com.google.http-client:google-http-client-xml:jar:1.20.0:compile
[INFO]    |  |  |  +- com.google.http-client:google-http-client:jar:1.20.0:compile
[INFO]    |  |  |  \- xpp3:xpp3:jar:1.1.4c:compile
[INFO]    |  |  +- com.squareup.okhttp3:okhttp:jar:3.7.0:compile
[INFO]    |  |  \- com.squareup.okio:okio:jar:1.12.0:compile
[INFO]    |  +- com.esotericsoftware:kryo:jar:4.0.2:compile
[INFO]    |  |  +- com.esotericsoftware:reflectasm:jar:1.11.3:compile
[INFO]    |  |  |  \- org.ow2.asm:asm:jar:5.0.4:compile
[INFO]    |  |  \- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO]    |  +- ch.qos.logback:logback-core:jar:1.2.9:compile
[INFO]    |  \- ch.qos.logback:logback-classic:jar:1.2.0:compile
[INFO]    +- org.openmicroscopy:ome-xml:jar:6.3.1:compile
[INFO]    |  \- org.openmicroscopy:specification:jar:6.3.1:compile
[INFO]    +- org.openmicroscopy:ome-codecs:jar:0.3.2:compile
[INFO]    |  \- org.openmicroscopy:ome-jai:jar:0.1.3:compile
[INFO]    \- org.slf4j:slf4j-api:jar:1.7.6:compile

@sbesson

sbesson commented Jun 14, 2022

Copy link
Copy Markdown
Member

Using % mvn dependency:tree -Dverbose=true to include the conflicting dependency (resolved by the build system)

[INFO] ome:OMEZarrReader:jar:0.2.1-SNAPSHOT
[INFO] +- com.bc.zarr:jzarr:jar:0.3.4:compile
[INFO] |  +- edu.ucar:cdm-core:jar:5.3.3:compile
[INFO] |  |  +- edu.ucar:udunits:jar:5.3.3:compile
[INFO] |  |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.2:compile - omitted for duplicate)
[INFO] |  |  |  \- (org.slf4j:slf4j-api:jar:1.7.28:compile - omitted for duplicate)
[INFO] |  |  +- edu.ucar:httpservices:jar:5.3.3:compile
[INFO] |  |  |  +- (com.google.guava:guava:jar:28.0-jre:compile - omitted for duplicate)
[INFO] |  |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.2:compile - omitted for duplicate)
[INFO] |  |  |  +- org.apache.httpcomponents:httpclient:jar:4.5.9:compile
[INFO] |  |  |  |  +- (org.apache.httpcomponents:httpcore:jar:4.4.11:compile - omitted for duplicate)
[INFO] |  |  |  |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  |  |  |  \- commons-codec:commons-codec:jar:1.11:compile
[INFO] |  |  |  +- (org.apache.httpcomponents:httpcore:jar:4.4.11:compile - omitted for duplicate)
[INFO] |  |  |  +- org.apache.httpcomponents:httpmime:jar:4.5.9:compile
[INFO] |  |  |  |  \- (org.apache.httpcomponents:httpclient:jar:4.5.9:compile - omitted for duplicate)
[INFO] |  |  |  +- (com.google.re2j:re2j:jar:1.3:compile - omitted for duplicate)
[INFO] |  |  |  \- (org.slf4j:slf4j-api:jar:1.7.28:compile - omitted for duplicate)
[INFO] |  |  +- joda-time:joda-time:jar:2.10.3:compile
[INFO] |  |  +- org.jdom:jdom2:jar:2.0.6:compile
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java:jar:3.9.1:compile
[INFO] |  |  +- com.google.guava:guava:jar:28.0-jre:compile
[INFO] |  |  |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  |  |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.2:compile - omitted for duplicate)
[INFO] |  |  |  +- org.checkerframework:checker-qual:jar:2.8.1:compile
[INFO] |  |  |  +- com.google.errorprone:error_prone_annotations:jar:2.3.2:compile
[INFO] |  |  |  +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] |  |  |  \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[INFO] |  |  +- (com.beust:jcommander:jar:1.78:compile - omitted for conflict with 1.27)
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.11:compile
[INFO] |  |  +- com.google.re2j:re2j:jar:1.3:compile
[INFO] |  |  \- (org.slf4j:slf4j-api:jar:1.7.28:compile - omitted for conflict with 1.7.6)
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.11.3:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.11.3:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.11.3:compile
[INFO] |  \- org.lasersonlab:jblosc:jar:1.0.1:compile
[INFO] |     +- (junit:junit:jar:4.12:compile - omitted for conflict with 4.10)
[INFO] |     \- net.java.dev.jna:jna:jar:4.2.2:compile
[INFO] +- org.mockito:mockito-inline:jar:3.7.7:compile
[INFO] |  \- org.mockito:mockito-core:jar:3.7.7:compile
[INFO] |     +- net.bytebuddy:byte-buddy:jar:1.10.19:compile
[INFO] |     +- net.bytebuddy:byte-buddy-agent:jar:1.10.19:compile
[INFO] |     \- org.objenesis:objenesis:jar:3.1:compile
[INFO] +- org.testng:testng:jar:6.8:compile
[INFO] |  +- junit:junit:jar:4.10:compile
[INFO] |  |  \- org.hamcrest:hamcrest-core:jar:1.1:compile
[INFO] |  +- org.beanshell:bsh:jar:2.0b4:compile
[INFO] |  +- com.beust:jcommander:jar:1.27:compile
[INFO] |  \- org.yaml:snakeyaml:jar:1.6:compile
[INFO] +- xalan:serializer:jar:2.7.2:runtime
[INFO] |  \- xml-apis:xml-apis:jar:1.3.04:runtime
[INFO] +- xalan:xalan:jar:2.7.2:runtime
[INFO] |  \- (xalan:serializer:jar:2.7.2:runtime - omitted for duplicate)
[INFO] \- ome:formats-api:jar:6.10.0:compile
[INFO]    +- org.openmicroscopy:ome-common:jar:6.0.9:compile
[INFO]    |  +- io.minio:minio:jar:5.0.2:compile
[INFO]    |  |  +- com.google.http-client:google-http-client-xml:jar:1.20.0:compile
[INFO]    |  |  |  +- com.google.http-client:google-http-client:jar:1.20.0:compile
[INFO]    |  |  |  |  \- (org.apache.httpcomponents:httpclient:jar:4.0.1:compile - omitted for conflict with 4.5.9)
[INFO]    |  |  |  \- xpp3:xpp3:jar:1.1.4c:compile
[INFO]    |  |  +- (com.google.guava:guava:jar:25.1-jre:compile - omitted for conflict with 28.0-jre)
[INFO]    |  |  +- com.squareup.okhttp3:okhttp:jar:3.7.0:compile
[INFO]    |  |  |  \- (com.squareup.okio:okio:jar:1.12.0:compile - omitted for duplicate)
[INFO]    |  |  +- com.squareup.okio:okio:jar:1.12.0:compile
[INFO]    |  |  +- (joda-time:joda-time:jar:2.7:compile - omitted for conflict with 2.10.3)
[INFO]    |  |  +- (com.fasterxml.jackson.core:jackson-annotations:jar:2.9.6:compile - omitted for conflict with 2.11.3)
[INFO]    |  |  +- (com.fasterxml.jackson.core:jackson-core:jar:2.9.6:compile - omitted for conflict with 2.11.3)
[INFO]    |  |  \- (com.fasterxml.jackson.core:jackson-databind:jar:2.9.6:compile - omitted for conflict with 2.11.3)
[INFO]    |  +- com.esotericsoftware:kryo:jar:4.0.2:compile
[INFO]    |  |  +- com.esotericsoftware:reflectasm:jar:1.11.3:compile
[INFO]    |  |  |  \- org.ow2.asm:asm:jar:5.0.4:compile
[INFO]    |  |  +- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO]    |  |  \- (org.sis:objenesis:jar:2.5.1:compile - omitted for conflict with 3.1)
[INFO]    |  +- (joda-time:joda-time:jar:2.2:compile - omitted for conflict with 2.10.3)
[INFO]    |  +- (com.google.guava:guava:jar:29.0-jre:compile - omitted for conflict with 28.0-jre)
[INFO]    |  +- (org.slf4j:slf4j-api:jar:1.7.6:compile - omitted for conflict with 1.7.28)
[INFO]    |  +- ch.qos.logback:logback-core:jar:1.2.9:compile
[INFO]    |  +- ch.qos.logback:logback-classic:jar:1.2.0:compile
[INFO]    |  |  +- (ch.qos.logback:logback-core:jar:1.2.0:compile - omitted for conflict with 1.2.9)
[INFO]    |  |  \- (org.slf4j:slf4j-api:jar:1.7.22:compile - omitted for conflict with 1.7.28)
[INFO]    |  +- (xalan:serializer:jar:2.7.2:runtime - omitted for duplicate)
[INFO]    |  \- (xalan:xalan:jar:2.7.2:runtime - omitted for duplicate)
[INFO]    +- org.openmicroscopy:ome-xml:jar:6.3.1:compile
[INFO]    |  +- (org.openmicroscopy:ome-common:jar:6.0.9:compile - omitted for duplicate)
[INFO]    |  +- org.openmicroscopy:specification:jar:6.3.1:compile
[INFO]    |  +- (org.slf4j:slf4j-api:jar:1.7.6:compile - omitted for conflict with 1.7.28)
[INFO]    |  \- (joda-time:joda-time:jar:2.2:compile - omitted for conflict with 2.10.3)
[INFO]    +- org.openmicroscopy:ome-codecs:jar:0.3.2:compile
[INFO]    |  +- org.openmicroscopy:ome-jai:jar:0.1.3:compile
[INFO]    |  \- (org.openmicroscopy:ome-common:jar:6.0.9:compile - omitted for duplicate)
[INFO]    +- org.slf4j:slf4j-api:jar:1.7.6:compile
[INFO]    +- (xalan:serializer:jar:2.7.2:runtime - omitted for duplicate)
[INFO]    \- (xalan:xalan:jar:2.7.2:runtime - omitted for duplicate)

So most of the dependencies which fail to resolve in the OMERO build system are defined at the low-level ome-common component and conflict with other components also defined in the the core formats-* components like cdm-core.

It's still not fully clear to me why this is an issue with this repository but not with formats-gpl which effectively pulls the same transitive dependencies and exposes them through the downstream omero-* components. My best guess it that the Ivy conflict resolution fails to resolve the dependency tree injected in https://github.com/ome/openmicroscopy/blob/develop/ivy.xml#L21.

As things stand, we have evidence that although our coupled CI workflow is sufficient to include the development ZarrReader into a development OMERO.server, this causes serious issue when trying to build OMERO in isolation. In addition to the impact on a developer workfow, a more critical impact is that release builds are likely to fail for similar reasons that caused ZarrReader to be excluded in the past. So solving this is a blocker to the inclusion of this reader into an OMERO.server release.

Trying to come up with a list of options to explore/discuss:

  • investigate more specifically what causes the dependency resolution to fail and fing either how to update the POM of this component and/or the Ivy settings file in OMERO to use the current injection of ZarrReader
  • review the low-level Bio-Formats dependencies to unify versions and minimize conflicts so that Maven, Gradle & Ivy resolve dependencies identically
  • update the OMERO build system to drop Ivy as the dependency resolver and use another system e.g. Gradle more compatible with the rest of the stack
  • migrate this reader back into the core Bio-Formats components (see also discussion in Integration with Bio-Formats #32) and release it through omero-model etc
  • migrate this reader as an omero-* component using the Gradle build system

@jburel

jburel commented Jun 23, 2022

Copy link
Copy Markdown
Member

using this PR in openmicroscopy build fails with

		::::::::::::::::::::::::::::::::::::::::::::::
		::              FAILED DOWNLOADS            ::
		:: ^ see resolution messages for details  ^ ::
		::::::::::::::::::::::::::::::::::::::::::::::
		:: org.objenesis#objenesis;2.5.1!objenesis.jar
		::::::::::::::::::::::::::::::::::::::::::::::

https://github.com/ome/openmicroscopy/runs/7022030936?check_suite_focus=true

@jburel

jburel commented Jun 23, 2022

Copy link
Copy Markdown
Member

This time

		::::::::::::::::::::::::::::::::::::::::::::::
		::              FAILED DOWNLOADS            ::
		:: ^ see resolution messages for details  ^ ::
		::::::::::::::::::::::::::::::::::::::::::::::
		:: junit#junit;4.12!junit.jar
		:: org.hamcrest#hamcrest-core;1.3!hamcrest-core.jar
		::::::::::::::::::::::::::::::::::::::::::::::

If I remember correctly a bump to testng should fix that one

@jburel

jburel commented Jun 23, 2022

Copy link
Copy Markdown
Member

@sbesson sbesson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question about overriding vs exclusing the objenesis dependdency.

Otherwise, this seems to fix the downstream build as per #38 (comment) and potentiall unlock the release. Should we try to get this merged and released (proposing 0.3.0 as per the S3 disabling) and try to consume the released version in the openmicroscopy build?

Comment thread pom.xml Outdated
<dependency>
<groupId>org.objenesis</groupId>
<artifactId>objenesis</artifactId>
<version>2.5.1</version>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the version shipped in ome-common (via kryo 4.0.2). From mvn dependency:tree -Dverbose it looks like it is conflicting with the version from org.mockito:mockito-inline:3.7.7.

Declaring it in the POM should suffice although I assume an alternative would be to add and exclude in the mockito dependency?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a quick discussion with @dgault and @jburel, a possible simpler and more effective option would simply be to declare mockito and testng as test dependencies (via scope). This might suffice to exclude these from the openmicroscopy Ivy resolution and prevent the copying of unnecessary JARs into the lib folder

@jburel

jburel commented Jun 23, 2022

Copy link
Copy Markdown
Member

@sbesson sbesson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Barring the removal of the S3 functionality (which will likely need to be reviewed at a later date), the proposed change fix some dependency mismatch in the dependency declaration, classify test-only dependency and remove the usage of commons-io.

I did some minimal testing in our nightly CI server importing a multi-file single image, a WSI and a plate and everything worked as expected.

Under these assumptions, proposing to get this merged, tag the component as 0.3.0 and then bump the version in openmicroscopy in preparation of the release if no objection

@sbesson
sbesson merged commit 3205f7a into ome:main Jun 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants