Automated testing

[Gerharddc]

To better support extensive refactoring and large feature additions, we desperately need to invest in proper automated testing. The reason this has been avoided thus far is because this is because it is rather complex to set up in the case of Litterbox. There are two major challenges:

1. Since Litterbox needs to interact with Podman and create many files in the home directory, the only reasonable way to test Litterbox would be inside a container or a VM (otherwise testing will quickly "pollute" the host machine and might also be affected by its state). This is already beyond the scope of what is easily supported by the standard testing framework in Rust.

2. Since the interface to Litterbox is an interactive CLI, it is also very difficult to interact with it programmatically. This is another area where the standard testing framework is not particularly useful.

I haven't looked into it much, but Robot Framework and/or regular Python might be more suitable for this type of complex "integration testing" than the standard testing framework in Rust.

Running tests inside a container will also make it impossible (or at least very difficult) to support development of Litterbox itself inside a litterbox. I really wanted this to be possible but I think it is something worth giving up in the name of simplicity.

[Gerharddc]

@thund-x if you have the time and motivation for a large project, this would be extremely useful to work on. Currently large refactors are very difficult and time consuming (both to implement and review) given the need to manually review that they do not break anything.

[thund-x]

I'd love to give a helping hand, however first I want to polish up the code internally (e.g. separate interactive from non-interactive code) and implement any missing features that will allow me to use Litterbox in its fullest capacity. One of those features which I believe will help with testing in the short term is a non-interactive CLI for creating containers for scripting purposes.

Red Hat has a blog post for creating nested podman containers and documentation for verifying nested virtualization support for when/if the time to use crosvm comes. The blog post consequentially describes an approach to develop Litterbox within a litterbox.

FWIW podman has a REST API (podman system service) if you are worried about CLI stability.

Also, what do we test within a container? Do we ensure each litterbox command exhibits the expected behavior? Do we ensure no semver breaking changes have been made to the public interface? In that case verifying clap derived structs accept certain arguments should be enough. Do we treat podman as a black box and assume it works perfectly or do we verify its actions (e.g. created an image with X, Y, Z attributes, created a container, and so forth)?

DX-wise, I'd love if I can call a single function (e.g. run_isolated) within a unit test and have that function prepare the entire environment, copy the source code within a container, run the unit test there, tear down the container, and return to the host with the test result. Theoretically all we will have to do is detect whether the host or container is running the test and conditionally run cargo test [...OPTIONS] and redirect its output to the host's unit test.

That container should have R/W volumes for target (per cargo test on the host) and the podman image cache (permanently) to minimize download and build times.

[Gerharddc]

I'd love to give a helping hand, however first I want to polish up the code internally (e.g. separate interactive from non-interactive code) and implement any missing features that will allow me to use Litterbox in its fullest capacity. One of those features which I believe will help with testing in the short term is a non-interactive CLI for creating containers for scripting purposes.

Thanks! And yeah, a non-interactive CLU would definitely simplify testing a lot.

Red Hat has a blog post for creating nested podman containers and documentation for verifying nested virtualization support for when/if the time to use crosvm comes. The blog post consequentially describes an approach to develop Litterbox within a litterbox.

I have also seen this blog post from RedHat but have not tried it out yet. At first glance, it looked very complex and a bit fragile which is why I have not tried it yet. But it could maybe work. I've also considered VMs as a possible solution, but that is too far off to be useful in the short term.

My feeling these days is that the easiest way to get started is by just developing either directly on the host system or inside a regular VM. I think Zed has support for remote development via SSH now, so it could be an option to run Zed inside a litterbox and then use that for remote development inside a regular VM. In theory it should be able to forward the Litterbox SSH agent too.

Something else to consider is that we need a solution that works on GitHub Actions runners so it can be part of the CI/CD. If I recall correctly, they are just VMs.

FWIW podman has a REST API (podman system service) if you are worried about CLI stability.

No, I'm not overly worried about the CLI stability (at least not right now) so I don't think we need to bother with the REST API.

Also, what do we test within a container? Do we ensure each litterbox command exhibits the expected behavior? Do we ensure no semver breaking changes have been made to the public interface? In that case verifying clap derived structs accept certain arguments should be enough. Do we treat podman as a black box and assume it works perfectly or do we verify its actions (e.g. created an image with X, Y, Z attributes, created a container, and so forth)?

Ideally we want to test as much as possible+practical of course. But we don't have to solve everything all at once. The most valuable would just be to verify that key commands run without returning errors and that they have the expected results.

As a very first step it would be great to just verify that the primary workflow works. I.e. you can build and enter a litterbox and run a command inside it. This should already catch most of the potential breakage possibilities.

DX-wise, I'd love if I can call a single function (e.g. run_isolated) within a unit test and have that function prepare the entire environment, copy the source code within a container, run the unit test there, tear down the container, and return to the host with the test result. Theoretically all we will have to do is detect whether the host or container is running the test and conditionally run cargo test [...OPTIONS] and redirect its output to the host's unit test.

Yeah that would be very nice if it is possible!

[Gerharddc]

My feeling these days is that the easiest way to get started is by just developing either directly on the host system or inside a regular VM. I think Zed has support for remote development via SSH now, so it could be an option to run Zed inside a litterbox and then use that for remote development inside a regular VM. In theory it should be able to forward the Litterbox SSH agent too.

I've now confirmed that SSH forwarding for remote development works quite well with the Litterbox agent. So it seems this could be a viable approach, also in cases where one needs more than what is possible inside a container (such as mounting a USB device in some cases)