Future directions for the test API

[Snaipe]

I am a bit dissatisfied by the current state of the test API. When I started this project, I came from a Java/C++ background where we had annotation-driven (JUnit) & macro-driven (GTest) test frameworks, and was pretty frustrated by how painful to use C frameworks were. This was two and a half years ago, and since then I've seen the design mistakes that I made limit and impact the project in (mostly) negative ways.

I think it's time to address that. I've talked about it a bit in #168, but I think that we need to address a few problems with the current design of the test API

Consistency

We currently have three flavour of tests: regular, parameterized, and theories. It doesn't take much to see that all three are pretty much the same; the only thing that differs is the way parameters are generated. The Test, ParameterizedTest, and Theory macros pretty much roll their own semantics at the moment, and I really hate this.

Hierarchy

The current API defines a two-level hierarchy for tests: on top, test suites, and right below, tests. Criterion defines TestSuite as a mean to define test suites properties, although this can feel quite hack-ish. I always told myself that this hierarchy was good enough for anyone, but now, I think that this might be complexifying the API by always requiring a test suite. Thoughts? It might be better to optionally provide a parent in the test definition.

Compiler-dependency

This one is a bit minor compared to the consistency problem, but as I mentionned in #184, using sections to register tests makes the API compiler-dependent. While nowadays this is less of a problem with the market-share dominance of GCC, LLVM, and MSVC, this is something I'd like to push out of the API and into the implementation.

Proposals

Macro-based

This doesn't change much from the current API, but that doesn't mean we can't improve the semantics of the definition macros.

Turning parents into properties

We could turn the parent of a test into a property: instead of doing:

TestSuite(suite, properties...);
Test(suite, test) { ... }

We could have:

// no parent suite
Test(test) { ... }

// test is in a suite
TestSuite(suite, properties...);
Test(test, .parent = suite) { ... }

// test is nested in a suite, which is itself in a suite
TestSuite(root);
TestSuite(nested, .parent = root);
Test(test, .parent = nested) { ... }

However, this would make the usage of TestSuite to declare a suite mandatory, while its definition is currently inferred by the runner in 2.3.x.

Merging ParameterizedTest with Theory

The semantics of ParameterizedTest and Theory could be merged into one parameterized test macro. The usage of such a macro could be: ParameterizedTest(name, (args), properties...), e.g:

// parameterized test
ParameterizedTest(square, (int bar), .params.generator = my_generator) {
    cr_assert(eq(int, square(bar), bar * bar));
}

// theory
ParameterizedTest(division, (int bar, int baz), .params.matrix = my_theory_args) {
    cr_assume(ne(int, baz, 0));
    cr_assert(eq(int, divide(bar, baz), bar / baz));
}

Unify all the definition macros

We could additionaly go one step further and just make Test follow the above semantics. Regular tests would be declared with (void):

// regular test
Test(multiply, (void)) {
    cr_assert(eq(int, mult(1, 2), 2));
}

// parameterized test
cr_param_list generator_square(void) {
    static int i = 0;
    if (i > 2) {
        return NULL;
    }
    return cr_alloc_params((int, i++));
}

Test(square, (int bar), .params.generator = generator_square) {
    cr_assert(eq(int, square(bar), bar * bar));
}

// theory
cr_param_values division_matrix[] = {
    cr_values(int, INT_MIN, -2, -1, 0, 1, 2, INT_MAX),
    cr_values(int, INT_MIN, -2, -1, 0, 1, 2, INT_MAX),
};

Test(division, (int bar, int baz), .params.matrix = division_matrix) {
    cr_assume(ne(int, baz, 0));
    cr_assert(eq(int, divide(bar, baz), bar / baz));
}

Symbol-lookup

I believe this approach is already done by NovaProva. Basically, instead of relying on a macro to define our test as if it was something special, we just define a function with a specific name:

// regular test
void test_multiply(void) {
    cr_assert(eq(int, mult(1, 2), 2));
}

// using setup/teardown
void setup_multiply(void) {
}
void teardown_multiply(void) {
}

// parameterized w/ generator
cr_param_list generator_square(void) {
    static int i = 0;
    if (i > 2) {
        return NULL;
    }
    return cr_alloc_params((int, i++));
}

void test_square(int n) {
    cr_assert(eq(int, square(n), n * n));
}

// parameterized w/ static list
cr_param_list param_list_square_alt[] = {
    cr_define_params((int, 0)),
    cr_define_params((int, 1)),
    cr_define_params((int, 2)),
};

void test_square_alt(int n) {
    cr_assert(eq(int, square(n), n * n));
}

// theory
cr_param_values param_matrix_divide[] = {
    cr_values(int, INT_MIN, -2, -1, 0, 1, 2, INT_MAX),
    cr_values(int, INT_MIN, -2, -1, 0, 1, 2, INT_MAX),
};

void test_divide(int n, int d) {
    cr_assume(ne(int, d, 0));
    cr_assert(eq(int, divide(n, d), n / d));
}

This makes the API extremely simple; tests are just functions, and you can define other symbols to alter slightly the calling semantics of the test.

As for additional test properties, we have a few ways:

// simple but not convenient for C++
struct cr_test_properties properties_foo = {
    .timeout = 10,
}

// simple, consistent, but maybe more confusing 
double timeout_foo = 10;

// convenience macro
cr_properties(foo, .timeout = 10);

This approach, however, has a drawback: it makes it impossible to do some runtime initialization or exception handling for non-C languages (while declaring through a macro lets us do that since we can wrap the user function in a try/catch for instance); or rather than "impossible", it moves the responsibility back to the runner, so we have to compile some C++ on our side.

Another issue is that in practice, because we need to support C++ and Windows, we'd need to add a cr_register attribute to all of those symbols, which would expand to __declspec(dllexport) and/or extern "C" (the former is needed to actually find the symbol on windows, and the latter is needed to prevent mangling on the symbol).

Discussion

I'd really like some inputs on the matter. Preferably, the final choice will have to compromise between a few issues, but I'd really like to improve simplicity and consistency without shaving too much features off.

Finally, if anyone has other proposals, I'm all ears. There's no rush on the matter since we have some heavy refactor to do on the internals before changing the API, but we need to start thinking about it now.

[DoumanAsh]

Considering how difficult C can be, i think it would be difficult to get rid of macros.
While i find it nice to have just declare normal function, i think it maybe better to let Criterion to go with macros for test definitions.
It, after all, gives more control to Criterion and user do not need to worry about function signature and etc.

It would be though interesting to have possibility to define Suite as structure which could be used as context for running tests(similarly to have Google test uses its fixtures).

The semantics of ParameterizedTest and Theory could be merged into one parameterized test macro.

I'm not really sure it would easy to use as they serve quite different purposes.
It might get confusing.
Will user be able to have both semantics at once?

[Snaipe]

Considering how difficult C can be, i think it would be difficult to get rid of macros.
While i find it nice to have just declare normal function, i think it maybe better to let Criterion to go with macros for test definitions.
It, after all, gives more control to Criterion and user do not need to worry about function signature and etc.

I think that's a fair point, but then I also find having the parameter list in the test macro (i.e. the (int foo, float bar) in Test(name, (int foo, float bar), ...) { ... }) weird on a language standpoint. Making it a simple function makes the API braindead simple, which is a really nice things to have when testing.

On the other hand, as you implied, this means that we lock ourselves in an ABI contract. This discussion will hopefully shine some light on whether this is a bad thing for us considering that I will feature-lock criterion 3.0.

One thing I dislike with the symbol proposal is that I feel that other symbols (setup_, teardown_, properties_, ...) are pretty big gotchas and might not be that convenient once you want to reuse some of the functions. Perhaps only test_* should count as a candidate for registration, and further properties should all go through cr_properties(name, .setup = setup_name, etc...)

I'm not really sure it would easy to use as they serve quite different purposes.
It might get confusing.
Will user be able to have both semantics at once?

That's not really a problem in my opinion once you realize that both are pretty much doing the same thing: calling a function with parameters.
The only thing that differs is how exactly parameters are generated. For the ParameterizedTest macro, it's a sequence of parameter lists that gets called sequentially (sum operation), and for the Theory macro, it's a matrix of parameter values that produces a parameter list for each permutation (product operation). And to be honest, cr_assume() is pretty much a if (!condition) cr_skip_test(), so there is no real reason why we're keeping both interfaces separate, except for the parameter generation as we described above.

[DoumanAsh]

One thing I dislike with the symbol proposal is that I feel that other symbols (setup_, teardown_, properties_, ...) are pretty big gotchas and might not be that convenient once you want to reuse some of the functions. Perhaps only test_* should count as a candidate for registration, and further properties should all go through cr_properties(name, .setup = setup_name, etc...)

Something like setup/teardown would definitely better to go as property of Suite structure (if Criterion would go for explicit suite definition).

Btw how easily symbol look-up can be implemented?
Is there compiler-independent way?

That's not really a problem in my opinion once you realize that both are pretty much doing the same thing: calling a function with parameters.

Hm... well i suppose i may over-think it.
I suppose it would make sense when you can easily distinguish between type of parameterized test.

[Snaipe]

Btw how easily symbol look-up can be implemented?
Is there compiler-independent way?

It's pretty easy. In fact, I've already been doing it for BoxFort and Mimick, so I can probably refactor that logic in a separate library and reuse it in the three projects.
This approach is, in fact, compiler independent, and is rather ABI (or more specifically executable-format) dependent. This isn't really a problem though, since criterion is already executable-format-dependent (because it uses BoxFort), and the market-share is overwhelmingly dominated by PE (Windows), ELF (Linux/BSD), and Mach-O (OS X/macOS/iOS)

[e74b254a966b1c]

If you are going to merge Theories with Parametrised Tests there should still be a way (maybe a parameter) to generate all parameter combinations.
Also, I'd like to have a way to test against random input. I was thinking about requiring the user to provide some random stream generator.

struct stream_state {
    void *state; /* Steam state data. It will be up to the users to choose what they do with it. */
    void *head_value; /* The head value of the stream. */
};

/* A stream generator. This will be up to the user to implement.
 * You could use the current->state to make this function reentrant, or it could be ignored
 * (tt's up to the user how they want it to be)
 */
struct stream_state *next_state(struct stream_state current);

[Snaipe]

If you are going to merge Theories with Parametrised Tests there should still be a way (maybe a parameter) to generate all parameter combinations.
Also, I'd like to have a way to test against random input. I was thinking about requiring the user to provide some random stream generator.

This can probably be done through the proposal to support generators. For instance, to generate the int parameters 0 through 2:

cr_param_list generator_square(void) {
    static int i = 0;
    if (i > 2) {
        return NULL;
    }
    return cr_alloc_params((int, i++));
}

(which pretty much is a more atomic version of the current ParameterizedTest interface)

And for random number generation (e.g., generating 10 random ints):

cr_param_list random_int_generator(void) {
    static int i = 0;
    if (i++ > 10) {
        return NULL;
    }
    return cr_alloc_params((int, random_int()));
}

However I have a few gripes with this. I don't like having to do memory allocation, type safety is missing, and the control flow of a regular function isn't really convenient for a generator.

struct stream_state {
   void *state; /* Steam state data. It will be up to the users to choose what they do with it. */
   void *head_value; /* The head value of the stream. */
};

I'd probably like the generation to be somewhat typesafe. The current Theory implementation claims UB when parameter types doesn't match the function prototype, without any compile-time warning. Plus, the way it infers which type is which when printing the parameters on a failure is broken.

I may have a proposal to address all these:

struct context {
    int32_t seed;
};

int32_t random_int32(int32_t *seed);

// declare a generator for the foo Test
Generator(foo, (i32 bar)) {
    cr_gen_enter(struct context, {
        .seed = ...
    });
    // generate 10 random numbers
    for (size_t i = 0; i < 10; ++i) {
        cr_gen_yield(random_int32(&seed));
    }
    cr_gen_leave();
}

Test(foo, (int32_t bar)) { ... }

Basically, Generator would declare a coroutine that gets called repeatedly until no more values are yielded. The i32 here would be a tag as described in the new assertion API, which translates to int32_t, so we can use this information to get some pseudo type introspection on the parameters, and statically assert at compile-time that both signatures match.

By the way, I think that criterion could also provide a simple implementation of a mersenne twister for convenience since rand() is garbage as a PRNG. Thoughts?

[e74b254a966b1c]

Starting from your proposal, I would like to change generators to make them able to create infinite streams.
So, instead of having a generator that gives us N values, we should have a generator that, when called, gives a single value and a new state.

/* Generator receives the types of the state and generated value.
 * It has 3 pre-defined variables:
 *     state      - const pointer to the current state
 *     next_state - location to store next state (generator should fill this)
 *     value      - location to store generated value (generator should fill this)
 * In the case below, state type = int32_t, value type = char.
 */
Generator((int32_t, char), char_generator) {
    int32_t number = random_int32(state);
    *next_state = number;
    *value = number % 0xFF;
}

int32_t seed1 = 42, seed2 = 7, seed3 = 13;

/* Similar to a theory, but instead of providing theory data points for c1, c2, c3,
 * you give them generators + seeds.
 * The last parameter is the number of iterations.
 */
StreamTheory((char_generator &seed1 c1, char_generator &seed2 c2, char_generator &seed3 c3),
             suite_name, test_name, 1232445) {
    test_something_with(c1, c2, c3);
    /* Please note that nowhere, do we need to explicitly call char_generator.
     * It will be up to Criterion to call char_generator for each iteration.
     * In the first iteration, the generators will receive seed1, seed2 & seed3, but
     * in the next iterations they will receive the states generated at the previous step.
     * Passing the state as function parameters ensures that generating c1 will not affect
     * generating c2 and c3.
     */
}

By the way, I think that criterion could also provide a simple implementation of a mersenne twister for convenience since rand() is garbage as a PRNG. Thoughts?

Don't know about rand()'s RNG qualities, but if it's bad, sure! Having this + stream generators will make the tool great for testing code against bad formatted input..
Also, would love some parallelized theories :)

[Snaipe]

Starting from your proposal, I would like to change generators to make them able to create infinite streams.

Note that the proposal I made defined a generator to be a coroutine, so an infinite stream can simply be represented as follows:

Generator(foo, (i32 bar)) {
    cr_gen_enter(void);
    // infinite generator of 1
    while (true) {
        cr_gen_yield(1);
    }
    cr_gen_leave();
}

The idea of making generators reusable is interesting. If that's the case, maybe the best course of action would be to embrace the fact that it's a function and get rid of the Generator macro?

out_type foo_generator(cr_gen_ctx ctx, state_type *state) {
    cr_gen_enter(ctx);
    // do something with the state, yield some values, ...
    cr_gen_leave(ctx);
}

StreamTheory((char_generator &seed1 c1, char_generator &seed2 c2, char_generator &seed3 c3),
             suite_name, test_name, 1232445) {

If the generators can be specified orthogonally from each others, this raises a few questions that are going to be hard to solve. For instance, what happens if you pass an infinite generator for c1, but a finite generator for c2? what happens if the iteration number is greater than the number of times the finite generator can be called?

Also the fact that you base your example as an extension of the Theory semantics makes the orthogonality of generators difficult. I don't even think we can have generators at all for Theories, because by nature Theories are called with the n-dimensional matrix products of all of its possible parameters. For instance:

If the possibles values of c1 are (1, 2); c2 (3, 4); and c3 (5, 6), then the theory will be called with the following parameters: (1, 3, 5), (1, 3, 6), (1, 4, 5), (1, 4, 6), (2, 3, 5), (2, 3, 6), (2, 4, 5), (2, 4, 6). More generally, for N parameters with M values each, you have M^N possible combinations.

You see what kind of problem we're going to have for generators: for each new value produced by a generator, we'll have to test it against every previous values ever produced, which becomes impractical for very large values as we have to store things as we go and we can only guess the upper bound; and while you may be able to be warned for ressource exhaustion at compile time with a static array, this is not possible with a generator.

I realize that this can be solved by doing some careful dynamic allocation, but it's hard to make this fast, and it might blow up in the developer's face when not used with care. Plus, Theories really aren't made for random testing, but careful testing with hand-picked "interesting" values (or ranges of values, which should definitely be something to support in the future). Testing against random values isn't theory testing, but fuzzy testing, and should be used in parallel to theories.

On another note, maybe we're taking an approach to the parameterization problem that is too complicated. How about not parameterizing tests, but introducing subtests:

Test(foo) {
    int32_t state = 42;
    Subtest("testing against 10 random values", .repeat = 10) {
        int32_t val = random_int32(&state);
        cr_assert(eq(do_thing(val), expected_val)); // failing the assert fails the subtest, but the test continues
    }
}

However, I have two issues with this:

The first issue I see there is that this syntax doesn't let us define nested functions (and while it might be accepted as a GNU extension, MSVC doesn't allow it) so we need to put this in a for loop. This means that we can't really spawn threads ourselves to parallelize this, but this can be worked around by the usage of OpenMP pragmas, which all of the compilers we target support.

The second issue is that if we parallelize this, then we'll be throwing multithreading into subtests, which might confuse developers (since the function they call might be racy) unless they explicitely want it. We could add a .parallel = true option for Subtest, though. A big plus of this approach however is that threads are significantly cheaper than spawning a new process like we do for parameterized tests, so this is going to be very fast.

[a1lu]

Turning parents into properties

I like this approach. It would be nice to be able do define the Test - TestSuite relation like this:

Test(foo) {...}

TestSuite(bar, ...)
{ // or another separation
    Test(foo2){...}
}

I'm afraid that's not possible in C though.

Unify all the definition macros

We should do this do simplify the API. But I don't like the signature ( Test(multiply, (void)) ) for regular tests.

[e74b254a966b1c]

If the generators can be specified orthogonally from each others, this raises a few questions that are going to be hard to solve. For instance, what happens if you pass an infinite generator for c1, but a finite generator for c2? what happens if the iteration number is greater than the number of times the finite generator can be called?

I was thinking about cycling through that finite amount of data points so that c2 acts as an infinite generator. On an abstract level, all that these infinite generators need to do is take a state and return a pair of state and value (State -> (State, Value)). This condition alone is enough for them to act like infinite streams.

Also the fact that you base your example as an extension of the Theory semantics makes the orthogonality of generators difficult. I don't even think we can have generators at all for Theories, because by nature Theories are called with the n-dimensional matrix products of all of its possible parameters.

You are right, it was a mistake to associate them with theories.

Testing against random values isn't theory testing, but fuzzy testing, and should be used in parallel to theories.

Yes, fuzzing is what I really want to do with generators. Also the fact that generators are independent and they pass the states as arguments makes them perfect to track the cause of errors and also make it possible to resume from a known state, after fixing a bug (this is mostly useful when the generators aren't random, but instead they simulate some kind of communication protocol that has states).

Maybe a good approach is to make a Fuzzing function, different from the Theory function.

[Snaipe]

@a1lu

I'm afraid that's not possible in C though.

Right, it's not. However, we can have subtests that follow the same hierarchy (as mentionned a few posts up).

We should do this do simplify the API. But I don't like the signature ( Test(multiply, (void)) ) for regular tests.

I was about to propose something like this:

Test(foo) {
}

Test(bar, params(i32 i, u64 u)) {
}

But I realize that this is also impossible to do as the macro concatenation of some identifier with . (as in .foo = bar) is not valid. Do you perhaps have something in mind?

---

@mirror3000

I was thinking about cycling through that finite amount of data points so that c2 acts as an infinite generator.

Interesting, I was thinking of stopping altogether with a finite generator finished, but this is a sensible option too. This also means that we effectively have to store all of the previous values, which becomes a problem if the generated set is large (and it usually is considering generators are precisely used when the context may become too large).
And we can't really not store previous values either since some generators may include one-time context and may not be callable more than once. In fact we can't assume anything from the contents of generator functions since they are technically user code.

I'm still not sure whether having separate generators is simpler to implement and use though. One could still use one generator for a particular test, that calls itself sub-generators (like a PRNG) in the manner they want. This might let users have more control over what values go through the generator, as some of them will definitely use them for values that aren't random.

make it possible to resume from a known state

That's an interesting idea. I think this could be nice to have especially with the feature request described in #202.

Maybe a good approach is to make a Fuzzing function, different from the Theory function.

I wonder if introducing yet another macro is a good idea on the long term.

I've been thinking a bit about how theories & parameterized tests are being implemented, and I definitely think that we could leverage some of the techniques I used in Mimick to run special tests like theories (or even fuzzing if we do implement this). Maybe something like the following:

int div_ints(int a, int b) {
    return a / b;
}

CR_CONTRACT_PREPARE(div_ints);

bool nonzero(int val) {
    return !!val;
}

Test(theory) {
    // create a contract over div_ints, called with any int as a first parameter, and any nonzero int as a second parameter
    cr_contract c = cr_contract_from(div_ints(cr_any(int), cr_that(int, nonzero)));

    // check for the theory using the contract with the specified datapoints
    int int_dp[] = {INT_MIN, -1, 0, 1, INT_MAX};
    bool ok = cr_theory_check(c, .datapoints = {int_dp, int_dp});

    // assert that the check passed
    cr_assert(ok, "checking theory for div_ints");
}

Test(fuzz) {
    // create a contract over div_ints, called with any int as a first parameter, and any nonzero int as a second parameter
    cr_contract c = cr_contract_from(div_ints(cr_any(int), cr_that(int, nonzero)));

    // fuzz the contract over 100 iterations
    bool ok = cr_fuzz(c, 100);

    // assert that the check passed
    cr_assert(ok, "checking fuzzing for div_ints");
}