Test failure for IM6 under debian

[bastien-roucaries]

Hi

packaging a new version for transitioning to IM7 I get the following test failure

1) testRotShearScale(magicktest.TestJMagick)java.lang.RuntimeException: Images had a difference of 337.407 which is bigger than max allowed of 20.0:
  test/actual_output/rotated.jpg test/correct_output/rotated.jpg
        at magicktest.MagickTesttools.compareImage(MagickTesttools.java:131)
        at magicktest.MagickTesttools.writeAndCompare(MagickTesttools.java:161)
        at magicktest.TestJMagick.testRotShearScale(TestJMagick.java:346)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
2) testMontageWithText_failMightBeOk(magicktest.TestJMagick)java.lang.RuntimeException: Images had a difference of 4620.37 which is bigger than max allowed of 20.0:
  test/actual_output/montage_w_text.jpg test/correct_output/montage_w_text.jpg
        at magicktest.MagickTesttools.compareImage(MagickTesttools.java:131)
        at magicktest.MagickTesttools.writeAndCompare(MagickTesttools.java:161)
        at magicktest.TestJMagick.testMontageWithText_failMightBeOk(TestJMagick.java:283)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
3) testConvolve(magicktest.TestJMagick)java.lang.RuntimeException: Images had a difference of 24521.3 which is bigger than max allowed of 20.0:
  test/actual_output/convolved.jpg test/correct_output/convolved.jpg
        at magicktest.MagickTesttools.compareImage(MagickTesttools.java:131)
        at magicktest.MagickTesttools.writeAndCompare(MagickTesttools.java:161)
        at magicktest.TestJMagick.testConvolve(TestJMagick.java:477)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
4) testConstituteDrawTransparent(magicktest.TestJMagick)java.lang.RuntimeException: Images had a difference of 2573.92 which is bigger than max allowed of 20.0:
  test/actual_output/blank.jpg test/correct_output/blank.jpg
        at magicktest.MagickTesttools.compareImage(MagickTesttools.java:131)
        at magicktest.MagickTesttools.writeAndCompare(MagickTesttools.java:161)
        at magicktest.TestJMagick.testConstituteDrawTransparent(TestJMagick.java:402)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
5) testMontage(magicktest.TestJMagick)java.lang.RuntimeException: Images had a difference of 449.157 which is bigger than max allowed of 20.0:
  test/actual_output/montage.jpg test/correct_output/montage.jpg
        at magicktest.MagickTesttools.compareImage(MagickTesttools.java:131)
        at magicktest.MagickTesttools.writeAndCompare(MagickTesttools.java:161)
        at magicktest.TestJMagick.testMontage(TestJMagick.java:264)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
6) testAnnotate_failMightBeOk(magicktest.TestJMagick)java.lang.RuntimeException: Images had a difference of 247.674 which is bigger than max allowed of 20.0:
  test/actual_output/blank_w_text.jpg test/correct_output/blank_w_text.jpg
        at magicktest.MagickTesttools.compareImage(MagickTesttools.java:131)
        at magicktest.MagickTesttools.writeAndCompare(MagickTesttools.java:161)
        at magicktest.TestJMagick.testAnnotate_failMightBeOk(TestJMagick.java:439)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
There were 2 failures:
1) testDiverseOperations(magicktest.TestJMagick)junit.framework.AssertionFailedError: Scaled to  expected:<60> but was:<50>
        at magicktest.TestJMagick.testDiverseOperations(TestJMagick.java:182)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
2) testQuantizeImage_failMightBeOk(magicktest.TestJMagick)junit.framework.AssertionFailedError: Colors  expected:<235> but was:<251>
        at magicktest.TestJMagick.testQuantizeImage_failMightBeOk(TestJMagick.java:378)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)

FAILURES!!!
Tests run: 17,  Failures: 2,  Errors: 6

Can you investigate

pipeline here
https://salsa.debian.org/java-team/jmagick/-/pipelines/724798

[nordfalk]

Hi @bastien-roucaries , if you look at the 'golden' images - i.e. the expected output, you can see that it is 17 years old: https://github.com/techblue/jmagick/tree/master/test/correct_output

Imagemagick has probably developed a lot since then and therefore the 'correct_output' has probably changed.

For the failing Images had a difference of (something) which is bigger than max allowed of 20.0: .... tests, and also the testQuantizeImage_failMightBeOk, please copy 'actual_output' to 'correct_output' and make a PR and we can look thru the changes and visually check that the acutal output is fine.

The error testDiverseOperations(magicktest.TestJMagick)junit.framework.AssertionFailedError: Scaled to expected:<60> but was:<50> in https://github.com/techblue/jmagick/blob/master/test/magicktest/TestJMagick.java#L182 sounds strange, like width and height has been interchanged.

[nordfalk]

You can overwrite the 'golden' testset by setting generate_correct_output = true in https://github.com/techblue/jmagick/blob/master/test/magicktest/MagickTesttools.java#L35

If you look at https://github.com/techblue/jmagick/blob/master/test/magicktest/MagickTesttools.java#L93 you can see that the comparison is based on mean squared error pixel by pixel (done by invoking compare -metric MSE), which means that if something has moved slightly in the image, the test will fail. But that might not be an error, but just a change in the underlying ImageMagick implementation, for example due to an improvement of ImageMagick.