Allow ordered declarations using WildcardPatternSuite

[jgreffe]

Hello,

when I declare a suite with this, classes are randomly executed:

@RunWith(WildcardPatternSuite.class)
@Suite.SuiteClasses({FirstTest.class, AnotherTest.class})
@SuiteClasses("?*/**/*Test.class")

Maybe using LinkedHashSet would resolve this?

[MichaelTamm]

Hi jgreffe,

I really appreciate that you not only created an issue, but provided a PR too!

But I really wonder, why do you care about the test order in the first place?
I am a little bit hesitant to add more complexity to junit-toolbox without seeing a valid reason for doing so.
Could you please tell me more, why you want that tests are executed in a certain order?

[jgreffe]

Hi jgreffe,

I really appreciate that you not only created an issue, but provided a PR too!

But I really wonder, why do you care about the test order in the first place?
I am a little bit hesitant to add more complexity to junit-toolbox without seeing a valid reason for doing so.
Could you please tell me more, why you want that tests are executed in a certain order?

Hello Michael,

this PR actually has two features:

• replace all HashSet to LinkedHashSet to keep order of files/tests as found
• add ability to use breadth-first or depth-first files scan

This allows to have structured tests always executed in the same order so it's predictable.
Sometimes you may need specific cases to be run before all others.
And sometimes you may need test cases on level 1 to be run before the ones on level 2.

Not sure if it's clear enough?

[MichaelTamm]

Hi again,

depending on tests to be executed in a predictable order sounds like a code smell to me: Tests should be independent from each other.

The best execution order (in my opinion) is: random. (Or maybe: failed first, then random)

I'm still hesitant to merge your PR, because I'm not convinced that it is a good idea.

On the other hand side, predictability isn't such a bad thing. ;)

Would the following compromise be good enough for you: We just replace all HashSets with LinkedHashSets.

[jgreffe]

Hey Michael,

just pointing out this:
http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html#runOrder

failsafe allows Suites being ordered to give more predictability on Suite level.

Would also like having the same predictability when Tests are resolved from

@RunWith(WildcardPatternSuite.class)
@Suite.SuiteClasses({FirstTest.class, AnotherTest.class})
@SuiteClasses("?*/**/*Test.class")

depending on their level in dir/package structure.

But it's up to you :)