Kantele

Core facility

Core facility has its own top level page, where projects are plotted that are of core facility type. These plots are populated with data selected from a project table, which functions as in the home page, i.e. one can filter on user, name, dates, type, activity, expiration, etc. It is possible to close projects in this page, as well as phasing out of projects (i.e. closes and sets an expiration date). In the default table all open CF projects are shown. Closing or phasing out a project will mark them with a ✅ in the plots.

Auto-deletion

A cron job exists (TODO, currently not running) for a custom Django command to delete expired files. This can be run by hand as well, like this:

docker compose [options] exec web python manage.py delete_expired_files --help

Using --dry-run will only show the potential files to be deleted, otherwise one can also specify e.g. --analysis, --datasets, etc. The command will queue files for deletion:

• files in INBOX which are also in a dataset
• files in datasets on shares with an expiry time (e.g. sensitive data, space-constrained analysis cluster)
• Also other files in shares with an expiry time, e.g. analysis output, library files, web-served report files
• mzML converted files which have a configured lifespan in settings.py

Development, before starting

Prerequisites:

• docker
• npm 18

First you build the containers for running the Django backend with postgres and nginx:

# Create an .env file:
cp .env.example .env

# Edit the new .env file!

# Build containers, with static files and database skeleton
bash src/docker/init_develop.sh

# First, ensure that docker containers are running
docker compose up

# You can access a local development page http://localhost or http://${your-machine-url}/
# The admin login = test & password = test

Development:

The backend Django code is mounted in the running web container while developing. If you change the code, restart with:

docker compose restart web

In production, the frontend JS is built and stored inside the nginx container, but in development you do the frontend building yourself, so you can quickly rebuild a frontend app when changing it. To change a single app (datasets, home, analysis, etc), go to the folder and build:

cd src/frontend/datasets
npm run build

# You can run continuous frontend rebuilding with live reload like this:
npm run dev

# For refreshing ALL frontend apps at the same time
# Also does Django static file collection
bash src/docker/create_static.sh

Note that changes can be viewed at http://localhost (using default ports - as seen in docker-compose.yml).

If you want to rebuild the containers:

# Set user and group id for the file ownership to the same ids the user has.
# I am not sure how to do this on windows or if it is necessary
export USER_ID=$(id -u)
export GROUP_ID=$(id -g)
docker compose build

DB content

Kantele uses a postgres database, and the development container has local folders bind-mounted in the container for persistence, so if the container is removed, the database will persist. There is also a backup folder mounted (defined in .env as BACKUP_PATH), through which you can deliver SQL dump files.

Creating a database dump:

# If you have the containers up:
docker compose exec db pg_dump -U kanteleuser -d kantele -f /pgbackups/file_to_dump_to.sql

# If the containers are down:
docker compose run db pg_dump -U kanteleuser -d kantele -f /pgbackups/file_to_dump_to.sql

Installing a new database dump:

# First delete the old DB - if there is none, first start the containers 
# with "docker compose up" to initialize postgres:
# If the containers are down:
docker compose run db dropdb -U postgres kantele
docker compose run db createdb -U postgres kantele
# Now install the new data
docker compose run db psql -U postgres -d kantele -f /pgbackups/file_with_dump.sql

# If instead you have the containers up:
docker compose exec db dropdb -U postgres kantele
docker compose exec db createdb -U postgres kantele
docker compose exec db psql -U postgres -d kantele -f /pgbackups/file_with_dump.sql

Upgrading postgres major version, first set the old and new version in src/docker/postgres-update-compose.yml, adjust the paths to the data dir in those (likely different in production), then do the following (first shutdown application to prevent writes):

# Backup data:
mv path/to/data/db path/to/data/olddb
# Start postgres container with old version:
docker compose -f src/docker/postgres-update-compose.yml --env-file .env up olddb
# Create a dump:
docker compose -f src/docker/postgres-update-compose.yml --env-file .env exec olddb pg_dumpall -U postgres > dumpold.sql
# Stop old container, inspect dump file, then start new:
docker compose -f src/docker/postgres-update-compose.yml --env-file .env up newdb
# Load old dump in new container (`-T` is to disable tty so we can pipe the local dump file:
docker compose -f src/docker/postgres-update-compose.yml --env-file .env exec -T newdb psql -U postgres < dumpold.sql

When going from postgres 14 to 15 or higher you will also need to add:
docker compose -f src/docker/postgres-update-compose.yml --env-file .env exec -T newdb psql -U postgres -d kantele
GRANT ALL ON SCHEMA public TO kanteleuser; 
# Somehow in testing my migrations after this crash with "no permission" until I do it again
# in the same compose environment as the migrations, which is weird since the web UI runs fine

Testing

To run the Django tests (python unittests):

# Run all the tests:
bash run_tests.sh

# Or run a specific test
bash run_tests.sh analysis.tests.TestGetAnalysis.test_ok

Tests run in docker compose containers and are also running on github actions inside the same containers.

Repo

There is a data directory for persistent data between container rebuilds in development and testing. The directory contains:

• analysisfiles - analysis results go here, currently only work in prod
• db - database persistence
• test - contain filesystem fixtures for tests
• teststorage - empty, gitignored, used as a fresh fixture workdir in testing, populated by shutil.copytree in base test
• storage - used in develop/stage as a storage unit
• newstorage - for simulating a second storage unit
• uploads - where uploads go in the nginx container