Collaboration with t-testing-devex

[epage]

Following up on our reddit discussion

As you mentioned, our short-term goals are likely not aligned. I'm assuming your focus is on providing a high quality custom test harness to use right now. Our current official goal isn't even custom test harness related but for stabilizing json output; I'm working on a custom test harness for the sake of exploring potential needs of the schema. Long term, we'd like to have a high quality custom test harness but also provide good interop between custom test harnesses and ensure we have a stable API for an ecosystem of plugins to be tied into. This means I'm doing a lot of experimentation which might not be too chaotic for general end-users.

Some potential routes for collaboration

• You are ok with our experimental nature and sharing ownership with others and we fully merge in your project
• We collaborate on information, like end-user needs, techniques, etc, and maybe we fork your project so rustest can still meet its goals while we jump start the macro side of our project.

[epage]

btw a similar thread with the test-r maintainer is on zulip

[mgautierfr]

Some potential routes for collaboration

• You are ok with our experimental nature and sharing ownership with others and we fully merge in your project

It may be a good thing to list what features you want to experiment with.

But for json I think we can enable it behind a feature flag, with documentation telling this is an experiment and user feedback are welcomed. You could "play" with json while keeping the root features (running tests, fixtures system...) functional and relatively stable.

To do so I think we would have to stop using/fork libtest_mimic to be able to patch this part.
I've already thought about that as I was kind of feel that I was duplicated some part of the runner, but I haven't done it as I'm not sure it worth the work. If libtest_mimic is "attacked" by both sides we may have to.

• We collaborate on information, like end-user needs, techniques, etc, and maybe we fork your project so rustest can still meet its goals while we jump start the macro side of our project.

In any case, we can of course collaborate on information, whatever we decide about the code collaboration.

---

From the thread on zulip, here some complements/corrections:

test-r uses an enable!() macro to generate main while rustest uses a #[main] attribute. For me, the attribute implies something more is possible since it leaves you a main body but I don't see it mentioned in the docs, so I'm not sure

Indeed, the #[main] attribute leave the main body free to do "something". However, I don't know what yet, so the current body of the main function is simply drop for now and replaced with rustest main code.

Also, I think it would be nice if cargo automatically deactivate harness if there is a main function already present. With the attribute, you actually have a main function in your code. It may help cargo to detect it if such change is made.

test-r registers fixtures as #[test_dep] and accesses it using the return type of the fixture, requiring a tag if there are two fixtures with the same return type. rustestregisters fixtures as #[fixture] and accesses them using the fixture's name (like pytest)

There is two things here:

• Indeed, rustest use the name of the function marked with #[fixture] as the name of the fixture.
• But internally, it create a new type and so fixtures are referenced (in tests and dependent fixtures) by the type (which is also the name of the fixture).

rustest is in the middle of rstest and test-r:

• As rstest, we use the name of the setup function as the fixture name.
• As test-r, we use the argument type to request a fixture

Internally, I "simply" use type inference to create the correct fixture. This avoid me some complex parsing of function signatures in the proc macro.

test-r uses fixture matrices for parametrized tests while rustest uses parametrized tests

rustest uses both. You can parametrize tests and fixtures.
Internally it is always a fixture matrix (Param is just an ad hoc fixture put in the matrix).
Currently, parametrized test/fixture is the only way to have a matrix with more than one value, but I would like to allow fixture setup to return a Vec<T> instead of T

test-r use the common should_panic attribute while rustest uses xfail (xfail also covers the return type)

This part will probably change soon. I have to extend the xfail to be able to specify how we expect the test to fail.