NaaVRE flavors

NaaVRE flavors are sets of Docker images tailor-made for a specific use-case. They bring together NaaVRE Jupyter extensions and a Conda environment. Each flavor consists of the three images:

• naavre-fl-{myflavor}-jupyter, the NaaVRE Jupyter Lab image. This is the image for the users Jupyter Lab instances, containing dependencies for notebook execution.
• naavre-fl-{myflavor}-cell-runtime, the base images for the build stage of NaaVRE cells. This image contains conda dependencies for containerized cells execution.
• naavre-fl-{myflavor}-cell-build, the base images for the runtime stage of NaaVRE cells. This image contains other dependencies (system, manually added, etc.) for containerized cells execution.

Each flavor corresponds to a directory ./flavors/{myflavor}, with the following structure:

{myflavor}
├── environment.yaml           # Conda environment with `name: {myflavor}`
├── flavor_config.yaml         # Build configuration
├── tests
│    └── tests.sh              # Test script run in naavre-{myflavor}-cell
├── [cell-build.Dockerfile]    # Optional override to docker/cell-build.Dockerfile
├── [cell-runtime.Dockerfile]  # Optional override to docker/cell-runtime.Dockerfile
└── [jupyter.Dockerfile]       # Optional override to docker/jupyter.Dockerfile

Build and Run Containers Locally

Build

Use the helper script:

$ ./build-local.sh -h
Usage: ./build-local.sh [-n] [-t target] flavor

-h,--help           print help and exit
-n,--dry-run        print the commands that would be executed and exit
-t,--target target  build target (options: jupyter, cell-build, cell-runtime,
                    cell-test, cell-all, all; default: all)
flavor              flavor name

Example: all images for the vanilla flavor:

$ ./build-local.sh vanilla -t all
Building images...
docker build . -f ./docker/jupyter.Dockerfile --build-arg CONDA_ENV_FILE=./flavors/vanilla/environment.yaml -t naavre-fl-vanilla-jupyter:local
[+] Building 0.3s (11/11) FINISHED                                                             docker:default
 => ...
docker build . -f ./docker/cell-build.Dockerfile --build-arg CONDA_ENV_FILE=./flavors/vanilla/environment.yaml -t naavre-fl-vanilla-cell-build:local
[+] Building 0.3s (9/9) FINISHED                                                               docker:default
 => ...
docker build . -f ./docker/cell-runtime.Dockerfile --build-arg CONDA_ENV_FILE=./flavors/vanilla/environment.yaml -t naavre-fl-vanilla-cell-runtime:local
[+] Building 0.7s (5/5) FINISHED                                                               docker:default
 => ...
docker build . -f ./docker/cell-test.Dockerfile --build-arg BUILD_IMAGE=naavre-fl-vanilla-cell-build:local --build-arg RUNTIME_IMAGE=naavre-fl-vanilla-cell-runtime:local -t naavre-fl-vanilla-cell-test:local
[+] Building 0.3s (11/11) FINISHED                                                             docker:default
 => ...

Built images:
naavre-fl-vanilla-jupyter:local
naavre-fl-vanilla-cell-build:local
naavre-fl-vanilla-cell-runtime:local
naavre-fl-vanilla-cell-test:local

Example: jupyter image for the vanilla flavor:

$ ./build-local.sh vanilla -t jupyter
Building images...
docker build . -f ./docker/jupyter.Dockerfile --build-arg CONDA_ENV_FILE=./flavors/vanilla/environment.yaml -t naavre-fl-vanilla-jupyter:local
[+] Building 0.3s (11/11) FINISHED                                                             docker:default
 => ...

Built images:
naavre-fl-vanilla-jupyter:local

Example: all cell images for the vanilla flavor:

$ ./build-local.sh vanilla -t cell-all
Building images...
docker build . -f ./docker/cell-build.Dockerfile --build-arg CONDA_ENV_FILE=./flavors/vanilla/environment.yaml -t naavre-fl-vanilla-cell-build:local
[+] Building 0.3s (9/9) FINISHED                                                               docker:default
 => ...
docker build . -f ./docker/cell-runtime.Dockerfile --build-arg CONDA_ENV_FILE=./flavors/vanilla/environment.yaml -t naavre-fl-vanilla-cell-runtime:local
[+] Building 1.0s (5/5) FINISHED                                                               docker:default
 => ...
docker build . -f ./docker/cell-test.Dockerfile --build-arg BUILD_IMAGE=naavre-fl-vanilla-cell-build:local --build-arg RUNTIME_IMAGE=naavre-fl-vanilla-cell-runtime:local -t naavre-fl-vanilla-cell-test:local
[+] Building 0.3s (11/11) FINISHED                                                             docker:default
 => ...

Built images:
naavre-fl-vanilla-cell-build:local
naavre-fl-vanilla-cell-runtime:local
naavre-fl-vanilla-cell-test:local

Note: the cell-test target requires cell-build and cell-runtime. While the helper script allows to build only cell-test, it will fail if its dependencies haven’t previously been built. Use cell-all to build all three targets.

Run

Jupyter Lab

Example for the vanilla flavor:

docker run -it -p 8888:8888 naavre-fl-vanilla-jupyter:local

Terminal

Example for the vanilla flavor:

docker run --rm -it naavre-fl-vanilla-jupyter:local /bin/bash

Cell tests

Tests can be run with the helper script:

./build-local.sh vanilla -t cell-all --run-tests

Alternatively, you can run the tests with the following command:

docker run -v ./flavors/vanilla/tests/:/tests/ naavre-fl-vanilla-cell-test:local /bin/bash /tests/tests.sh
echo $?

Successful tests should output 0.

Tip

NaaVRE images are large and can quickly fill up your machine's storage. If needed, free-up space with docker image prune [-a] (documentation).

Resolving Conda-Pip Conflicts in Flavor Building

When building environments (or "flavors") with both Conda and pip, dependency conflicts can cause builds to fail. Follow Conda's recommendation of installing as many packages as possible with Conda.

If flavor build failure persists, install the packages only available on pip in a clean environment and list the dependencies:

!pip install pipdeptree
!pipdeptree -p <package-name>

Add as much of the dependencies as possible in conda.

Retry building the flavor.

If the flavor doesn't build, build only the conda part of the flavor. When the flavor only contains Conda packages and still doesn't build, the problem is not caused by a conda-pip conflict.

Run a terminal with the built flavor.

Activate the conda environment:

conda activate <flavor-name>

Install the packages that could not be installed in conda using pip:

!pip install <package-name>

You will get a report of installed packages that hopefully lists some packages where the version has been changed.

Add these version to the conda packages.

Retry building the flavor. Hopefully, the flavor builds successfully.