Image Garden, Spread integration tests as GitHub Action

The image-garden GitHub action allows running full-system integration tests, using spread, on vanilla images of Ubuntu, Debian, Fedora, CentOS, openSUSE, Alma, Arch Linux, Rocky Linux, Oracle Linux and more as prepared by image-garden.

You can test your application in the real environment it would be deployed in, very rapidly, entirely under your control. The same test suite can run locally on your computer and in CI, making the push-style development a thing of the past.

Image Garden and Spread are a perfect for testing packaging, system integration scripts, interaction with sandbox mechanism and anything else that requires a full system with a real Linux kernel used by the target platform.

Disclaimer

The action is in pre-release mode. You can see it in action at https://github.com/canonical/snapd-smoke-tests/

Inputs

spread-subdir

Optional sub-directory where spread.yaml is located.

spread-artifacts

Optional directory where spread saves task artifacts. A non-empty value causes the both collection of artifacts and uploading collected files to GitHub.

spread-artifacts-suffix

Optional suffix used to disambiguate GitHub artifacts that store spread artifacts. When missing the garden system name is used instead.

spread-variant

Optional variant of spread to use. Consult spread documentation for available variants.

spread-live

Optional boolean flag to enable live mode for spread. When set to "true", the -live flag is passed to the spread command, allowing interactive control of the test execution.

Constraint: spread-live can only be used when spread-variant is set to "plus". The action will fail with an error if this constraint is violated.

Default: "false"

Usage

Follow the steps to get started:

1. Create a branch of your project where you plan to introduce integration testing.

   At the end of this process we will open a pull request that introduces both the defintion of the test and the CI/CD workflow that runs it.

2. Create the .image-garden/README.md and .image-garden.mk files.

   The presence of the .image-garden/ directory instructs image-garden to store all temporary files in a sub-directory. This makes it much more tidy but we also need to create a file for git to keep the directory around.

   Please link to the documentation for both tools (image-garden and spread) for the benefit of your developers.

   # Image Garden State Directory
   
   This directory is used by `image-garden` to store large temporary files.
   It should be ignored in `.gitignore` apart from this `README.md` file
   itself.
   
   Consult documentation of [spread](https://github.com/canonical/spread) and
   [image-garden](https://gitlab.com/zygoon/image-garden).

   .image-garden.mk file is used to expand the cloud-init user data template defined on image-garden. It can be left empty if no extra configuration is needed.

3. Create the spread.yaml file.

   This file defines how spread should allocate and discard virtual machines used for testing, which systems we are interested in testing and where to find the tests.

   Put the following content inside. Pay attention to formatting as YAML requires spaces for indentation.

   project: demo
   path: /root/demo
   exclude:
     - .image-garden/
     - .git
   backends:
     garden:
       type: adhoc
       allocate: |
           if [ -n "${SPREAD_HOST_PATH-}" ]; then
               PATH="$SPREAD_HOST_PATH"
           fi
           exec image-garden allocate "$SPREAD_SYSTEM"."$(uname -m)"
       discard: |
           if [ -n "${SPREAD_HOST_PATH-}" ]; then
               PATH="$SPREAD_HOST_PATH"
           fi
           exec image-garden discard "$SPREAD_SYSTEM_ADDRESS"
       systems:
         - debian-cloud-12:
             username: root
             password: root
         - fedora-cloud-42:
             username: root
             password: root
   suites:
     tests/:
       summary: Integration test

4. Create the tests/hello/task.yaml file.

   This file defines the first test in our test suite.

   summary: hello-world
   execute: |
       echo "Here you will invoke your program"
       # TODO: actually call my program

   Spread uses directory structure where test suites contain one or more tasks (tests and tasks are used interchangeably). The tests/ directory here is the suite and tests/hello is the task.

5. Install the image-garden snap.

   sudo snap install --edge image-garden

   Your system must support snap packages. In general see https://snapcraft.io/image-garden for installation instructions tailored to specific Linux distribution.

6. Use spread built into image-garden

   Image-garden snap contains a bundled copy of upstream spread. The snap sandbox makes using a bundled copy much easier than having to install and maintain a separate program.

   sudo snap alias image-garden.spread spread

   This will allow you to run spread to run the integration tests. Spread reads spread.yaml and the referenced task.yaml files and arranges to prepare, execute and restore each task.

7. Ensure you can use hardware-assisted virtualisation

   You must have the right to open /dev/kvm. On most systems this file is owned by the kvm group. You must be a member of this group. Use groups to see if you are already a member. If you are not run sudo usermod -aG kvm $LOGNAME. For simplicity logout and log back in again and re-check that you are a member of the group.

   If your system does not have the /dev/kvm device then it does not offer hardware assisted virtualisation. Image garden will still work but everything will be much, much slower. This is typically encountered when system BIOS disables or does not allow enabling hardware virtualisation, inside virtual machines that do not support nested virtualisation and on specific CPU architectures where virtualisation is not yet supported (e.g. RISC-V and some older aarch64 systems).

8. Run spread locally

   Run spread -v to run the test for the first time. You will see that spread will allocate one instance of the two systems we've specified, connect to them, copy our project across, prepare each system by running project-specific logic, prepare each suite, prepare the task, execute the task and restore the task, suite and system.

   First run will be much slower, as image-garden will have to download pristine images for the operating systems you've selected for your tests, boot them once for preparation and save the result. Subsequent runs will be much faster.

   At this stage you can spend time to expand the test so that it does something useful. Spread is typically used to build, install and see the software working in each system. Your project-wide prepare may install compilation tools, configure, build and install your software. Your tasks may exercise distinct functionality of your software.

   Spread is heavily used within the Canonical/Ubuntu ecosystem, you may find projects using it to learn more this way.

9. Add the image-garden-action workflow.

   Create the file .github/workflows/spread.yaml with the following content:

   name: spread
   on:
       push:
           branches: ["main"]
       pull_request:
   jobs:
       debian-12:
           runs-on: ubuntu-latest
           steps:
               - name: Checkout code
                 uses: actions/checkout@v4
                 # This is essential for git restore-mtime to work correctly.
                 with:
                   fetch-depth: 0
               - name: Run integration tests
                 uses: zyga/image-garden-action@v0
                 with:
                   snapd-channel: latest/edge
                   image-garden-channel: latest/edge
                   garden-system: debian-cloud-12
       fedora-42:
           runs-on: ubuntu-latest
           steps:
               - name: Checkout code
                 uses: actions/checkout@v4
                 with:
                   fetch-depth: 0
               - name: Run integration tests
                 uses: zyga/image-garden-action@v0
                 with:
                   snapd-channel: latest/edge
                   image-garden-channel: latest/edge
                   garden-system: fedora-cloud-42

10. Add new files to git

    git add .github/workflows/spread.yaml .image-garden/README.md spread.yaml tests/hello/task.yaml

    Note that this workflow is very basic. You most likely want to use matrix jobs to avoid repetition and test on more systems. You should also plan caching strategy and choose workers that are large enough to run your test workflows. The default workers from GitHub come with four cores and 16GB of RAM and support for kvm and should work great for small to medium sized projects.

    As you add use more and more test systems or run more and more tests, you will consume your quota of GitHub CI minutes. You may need to purchase more, or deploy and pay for self-hosted runners.

    Commit, push and open a pull request.

Please leave a GitHub star if you found this useful. Please feel free to open issues, either on image-garden-action, image-garden or spread.

Troubleshooting

This section lists common errors and how to resolve them.

error: nothing matches provider filter

Cause: The system specified in the workflow input does not exist in the spread.yaml configuration file.

Resolution: Verify that the value of the garden-system input matches one of the systems defined in spread.yaml.

Cannot allocate garden:<system-name>: backend "garden" allocate must print ADDRESS=<SSH address> to stdout, got: ""

Cause: The requested operating system and architecture combination is not available in image-garden.

Resolution: Confirm that the specified OS and architecture are supported by image-garden, and update your inputs as needed.

Trouble accessing repositories using apt

Cause: Image-garden bundles apt-cacher-ng, which acts as a caching proxy for apt repositories. This improves performance by caching downloaded packages across workflow runs, but may mask issues related to accessing said repositories in environments with strict policies for outgoing network connections.

Resolution: Disable apt-cacher-ng by setting the cache-deb-packages input to false in your workflow:

- name: Run integration tests
  uses: zyga/image-garden-action@v0
  with:
    garden-system: debian-cloud-12
    cache-deb-packages: "false"

This will stop the apt-cacher-ng service and remove its socket, allowing direct access to apt repositories.

Further reading

Consult documentation of spread and image-garden.