Writing Test Runners

Basics

In ToughDay2 the test class expresses how the test looks and the logic of what needs to be executed and the test runner how you want to execute it. Let's look at a simple example directly from the Toughday2 core the SequentialTest and SequentialTestRunner

public abstract class SequentialTest extends AbstractTest  {
    ...

    @Override
    public Class<? extends AbstractTestRunner> getTestRunnerClass() {
        return SequentialTestRunner.class;
    }

    public abstract void test() throws Throwable;
}

public class SequentialTestRunner extends AbstractTestRunner<SequentialTest> {
    public SequentialTestRunner(Class<? extends AbstractTest> testClass) {
        super(testClass);
    }

    @Override
    protected void run(SequentialTest testObject, RunMap runMap) throws Throwable {
            testObject.benchmark().measure(testObject, ()->{
                testObject.test();
            });
    }
}

As you can see, the link is pretty simple. The SequentialTest has a method (getTestRunnerClass()) that returns the SequentialTestRunner. This is how Toughday2 knows, given a test, what runner to use. If you want to change the runner, you just need to override that method.

Creating a Test Runner

Let's implement a simple test runner that instead of running the test once, executes it 10 times. Just for the purpose of the example, we'll not extend the SequentialTest, but also write the test class from scratch. So the code would look as follows:

MyTestBase

import com.adobe.qe.toughday.api.core.AbstractTest;
import com.adobe.qe.toughday.api.core.AbstractTestRunner;

import java.util.ArrayList;
import java.util.List;

public abstract class MyTestBase extends AbstractTest {
    private static final List<AbstractTest> EMPTY_LIST = new ArrayList<>();

    @Override
    public List<AbstractTest> getChildren() {
        return EMPTY_LIST;
    }

    @Override
    public Class<? extends AbstractTestRunner> getTestRunnerClass() {
        return MyTestRunner.class;
    }

    @Override
    public abstract AbstractTest newInstance();

    public abstract void myTest() throws Throwable;
}

MyTestRunner

import com.adobe.qe.toughday.api.core.AbstractTest;
import com.adobe.qe.toughday.api.core.AbstractTestRunner;
import com.adobe.qe.toughday.api.core.RunMap;


public class MyTestRunner extends AbstractTestRunner<MyTestBase> {

    public MyTestRunner(Class<? extends AbstractTest> testClass) { super(testClass); }

    @Override
    protected void run(MyTestBase testObject, RunMap runMap) throws Throwable {
        for(int i = 0; i < 10; i++) {
            testObject.benchmark().measure(testObject, () -> {
                testObject.myTest();
            });
        }
    }
}

As you can see, just to exemplify, I created a new method called myTest (you are not limited to one method) and I created the runner, which iterates ten times executing and benchmarking the test using the benchmark().measure(...) call. You will not use a String argument for the name of the operation, because you are actually benchmarking the test itself, not an operation from it.

DSL Rules:

• The constructor needs to match the super constructor (one argument of type Class<? extends AbstractTest> testClass)
• The constructor needs to be public
• In the test runner you need to take care of benchmarking

TODO: Write a test which checks the Runner constructor.

Composite test runners

Writing runners for a Composite Test is essentially exactly like writing for simple ones. There are only two things that you need to keep in mind as a best practice:

1. Use the child test's runner, don't execute the test in the composite test runner via its methods
2. Don't benchmark the child test, its runner is responsible for taking care of that

We can look at the CompositeTestRunner from the ToughDay2 core to see how would that look like: public class CompositeTestRunner extends AbstractTestRunner {

    public CompositeTestRunner(Class testClass) { super(testClass); }

    @Override
    protected void run(CompositeTest testObject, RunMap runMap) throws Throwable {
        testObject.benchmark().measure(testObject, () -> {
            for (AbstractTest child : testObject.getChildren()) {
                AbstractTestRunner runner = RunnersContainer.getInstance().getRunner(child);
                runner.runTest(child, runMap);
            }
        });
    }
}

Same DSL Rules apply for composite test runners. In fact there is no difference between test/test runners simple or composite from the core's point of view. You can try writing a test runner which executes the children in a random order, instead of one after the other.