[project] Integration plugin testing

[UncleGoogle]

Extracted from GOG-Nebula/galaxy-integration-steam#1 (comment)

This could be integrated in build action or separate action.
Or maybe in the form of a testing framework which emulates Galaxy behavior and calls the plugin in the similar manner as Galaxy? That would be super useful to automate acceptance tests and avoid a lot of manual testing.

This will also make new plugins version more stable.

To be planned & splinted into smaller tasks

[ABaumher]

I believe it's possible to do this as part of continuous integration and block PRs based on failed cases. I'm glad we don't have that atm because i still need to fix all the cases (I've been hoping someone else would do it lol) but I'm not sure what that entails.

[UncleGoogle]

Yes, at the begging a simple job in CI that instantiates Plugin by using main function as Galaxy does would be enough. Then listen to list of features and maybe calls authenticate. If plugin don't crash test pass. This is actually a lot, because there can be a lot of mistakes that can lead to Python syntax error or import error even if locally everything looks good. That would be actually a test mostly for build process so that can and should be fully automated.

Similar approach would be taken to create framework to run such live tests in dev environment -- that that's a different story