Storing Test configuration

[knolleary]

We need to properly specify how test configurations are stored in the flow.

There are two options:

1. Each node/flow gains a new property (t for example) that contains all of the test configuration needed for that node/flow.
2. A new config node is defined that contains all of the test configuration information - nothing is added to individual nodes. Or multiple config nodes are used, one per test case/test suite.

---

Some parts of the developer experience that will by affected by this choice:

1. A user creates a new flow by copying existing nodes. Should the test cases by duplicated as well automatically? What if the test case spans multiple nodes, but the user only copies one of them? It probably makes sense to not automatically copy the test case, but have a way the user can copy and existing test case entirely and then modify it.
2. Same point again, but this time exporting the flow (either to Clipboard or Library). If the tests are in a config node, there will be no direct relationship between the nodes and the config node - so the export logic won't know to include them.

---

To begin with, it probably makes sense to use a config node. But should that be one per test-suite, one per test-case or one for everything?

"One for everything" becomes awkward when the user imports another flow that includes tests - we'd have to merge the config nodes somehow.

Having thought it through writing this, I think the right approach will be two config nodes: test-suite and test-case. That would make it easy to create copies of existing test cases/suites.

Thoughts?

[HiroyasuNishiyama]

Thank you for notifying me about this page.

As I wrote at last week's slack meeting, I'm now working on a simple prototype implementation to confirm the design concept of flow-testing.
Its implementation currently uses one type of configuration node. The configuration node corresponds to each test suite and also contains information about the test cases it contains.

As you suggest, I think an approach that splits this into two configuration nodes is also possible.
The reason I put them together is because I was worried that there would be too many configuration nodes related to testing.
However, considering copying and moving test cases between test suites, these may have to be separate nodes.

[knolleary]

How much are you prototyping? The goal with listing out the tasks was to have small pieces of work that can be PR'd/reviewed/discussed, rather than trying to implement the whole thing in one PR.

For example, getting the UI right will take a lot of work without doing anything about running the tests themselves. There are different pieces of the UI that can be delivered one at a time. Lets make sure we get the basic pieces right as we go.

[HiroyasuNishiyama]

I understand your opinion.

The aim of current prototype is not about implementing everything, but about exploring the direction of implementation.
Currently, it contains mockup implementation of basic UI parts and partial implementation of just a few event & actions to check the functionality of plugin mechanisms and pluggable message router.

2021-07-27.23.17.43.mov