We had some issues with tests not causing builds to fail. After debugging the plugin, I found that the reported test is a combination of the test suite and test method name. The test class name is not used. As a result, if we have multiple test classes with test methods that have identical names, the reported result of one test affects the other.
I've attached a little sample that reproduces the behaviour. I expect the build to fail, but it does not.
[INFO] Scanning for projects...
[INFO]
[INFO] -------------< org.apache.maven:maven-surefire-plugin-bug >-------------
[INFO] Building Archetype - maven-surefire-plugin-bug 1.0-SNAPSHOT
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- resources:3.3.1:resources (default-resources) @ maven-surefire-plugin-bug ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 4 resources from src/main/resources to target/classes
[INFO]
[INFO] --- compiler:3.13.0:compile (default-compile) @ maven-surefire-plugin-bug ---
[INFO] No sources to compile
[INFO]
[INFO] --- resources:3.3.1:testResources (default-testResources) @ maven-surefire-plugin-bug ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Users/jorislambrechts/Desktop/maven-surefire-plugin-bug/src/test/resources
[INFO]
[INFO] --- compiler:3.13.0:testCompile (default-testCompile) @ maven-surefire-plugin-bug ---
[INFO] Recompiling the module because of changed source code.
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 3 source files with javac [debug target 1.8] to target/test-classes
[WARNING] bootstrap class path is not set in conjunction with -source 8
not setting the bootstrap class path may lead to class files that cannot run on JDK 8
--release 8 is recommended instead of -source 8 -target 1.8 because it sets the bootstrap class path automatically
[WARNING] source value 8 is obsolete and will be removed in a future release
[WARNING] target value 8 is obsolete and will be removed in a future release
[WARNING] To suppress warnings about obsolete options, use -Xlint:-options.
[INFO]
[INFO] --- surefire:3.5.4:test (default-test) @ maven-surefire-plugin-bug ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[INFO] Running org.apache.maven.surefire.SuccessTest
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.016 s <<< FAILURE! -- in org.apache.maven.surefire.SuccessTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.011 s <<< FAILURE!
org.opentest4j.AssertionFailedError
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
at org.apache.maven.surefire.FailTest.test(FailTest.java:11)
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 s -- in org.apache.maven.surefire.FailTest
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.028 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
at org.apache.maven.surefire.FailTest.test(FailTest.java:11)
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
at org.apache.maven.surefire.FailTest.test(FailTest.java:11)
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.003 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
at org.apache.maven.surefire.FailTest.test(FailTest.java:11)
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.001 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
at org.apache.maven.surefire.FailTest.test(FailTest.java:11)
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.004 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.003 s <<< FAILURE!
org.opentest4j.AssertionFailedError
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
at org.apache.maven.surefire.FailTest.test(FailTest.java:11)
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 s -- in org.apache.maven.surefire.RunSuite
[INFO]
[INFO] Results:
[INFO]
[WARNING] Flakes:
[WARNING] org.apache.maven.surefire.RunSuite.test
[INFO] Run 1: PASS
[ERROR] Run 2: RunSuite.test
[ERROR] Run 3: RunSuite.test
[ERROR] Run 4: RunSuite.test
[ERROR] Run 5: RunSuite.test
[ERROR] Run 6: RunSuite.test
[ERROR] Run 7: RunSuite.test
[INFO]
[INFO]
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO]
[INFO]
[INFO] --- surefire:3.5.4:test (integration-suite) @ maven-surefire-plugin-bug ---
[INFO] Skipping execution of surefire because it has already been run for this configuration
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.512 s
[INFO] Finished at: 2026-02-16T16:36:47+01:00
[INFO] ------------------------------------------------------------------------
I did not create a PR because I'm not sure whether this is as intended or if I'm configuring the plugin incorrectly.
Affected version
3.5.4
Bug description
Hello,
We had some issues with tests not causing builds to fail. After debugging the plugin, I found that the reported test is a combination of the test suite and test method name. The test class name is not used. As a result, if we have multiple test classes with test methods that have identical names, the reported result of one test affects the other.
I've attached a little sample that reproduces the behaviour. I expect the build to fail, but it does not.
I did not create a PR because I'm not sure whether this is as intended or if I'm configuring the plugin incorrectly.
Thanks,
Joris
maven-surefire-plugin-bug.zip