MIP-Engine

Prerequisites

1. Install python3.8

2. Install poetry It is important to install poetry in isolation, so follow the recommended installation method.

Setup

Environment Setup

1. Install dependencies

   poetry install
   

2. Activate virtual environment

   poetry shell
   

3. Optional To install tab completion for invoke run (replacing bash with your shell)

   source <(poetry run inv --print-completion-script bash)
   

4. Optional pre-commit is included in development dependencies. To install hooks

   pre-commit install
   

Local Deployment

1. Create a deployment configuration file .deployment.toml using the following template

   ip = "172.17.0.1"
   log_level = "DEBUG"
   framework_log_level ="INFO"
   monetdb_image = "madgik/mipenginedb:dev"
   rabbitmq_image = "madgik/mipengine_rabbitmq:dev"
   
   algorithm_folders = "./mipengine/algorithms,./tests/algorithms"
   
   node_landscape_aggregator_update_interval = 30
   celery_tasks_timeout = 20
   celery_run_udf_task_timeout = 120
   
   [privacy]
   minimum_row_count = 10
   
   [cleanup]
   nodes_cleanup_interval=10
   contextid_release_timelimit=3600 #an hour
   
   [smpc]
   enabled=false
   optional=false
   get_result_interval = 10
   get_result_max_retries = 100
   smpc_image="gpikra/coordinator:v6.0.0"
   db_image="mongo:5.0.8"
   queue_image="redis:alpine3.15"
   
   [[nodes]]
   id = "globalnode"
   role = "GLOBALNODE"
   monetdb_port=50000
   rabbitmq_port=5670
   
   [[nodes]]
   id = "localnode1"
   role = "LOCALNODE"
   monetdb_port=50001
   rabbitmq_port=5671
   smpc_client_port=9001
   
   [[nodes]]
   id = "localnode2"
   role = "LOCALNODE"
   monetdb_port=50002
   rabbitmq_port=5672
   smpc_client_port=9002
   
   

   and then run the following command to create the config files that the node services will use

   inv create-configs
   

2. Install dependencies, start the containers and then the services with

   inv deploy
   

3. Optional Load the data into the db with (It is compulsory if you want to run an algorithm)

   inv load-data
   

4. Attach to some service's stdout/stderr with

   inv attach --controller
   

   or

   inv attach --node <NODE-NAME>
   

5. Restart all the node/controller services and keep the same containers with

   inv start-node --all && inv start-controller --detached
   

Local Deployment (without single configuration file)

1. Create the node configuration files inside the ./configs/nodes/ directory following the ./mipengine/node/config.toml template.

2. Install dependencies, start the containers and then the services with

   inv deploy --monetdb-image madgik/mipenginedb:dev1.2 --celery-log-level info
   

Start monitoring tools

1. Start Flower monitoring tool

   by choosing a specific node to monitor

   inv start-flower --node <NODE-NAME>
   

   or start a separate flower instance for all of the nodes with

   inv start-flower --all
   

   Then go to the respective address on your browser to start monitoring the nodes.

2. Kill all flower instances at any point with

   inv kill-flower
   

Algorithm Run

1. Make a post request, e.g.

   python example_post_request.py