Monitoring Tutorial

Overview

This repository contains information on potential monitoring solutions for the N5GEH platform. These are just suggestions for specific solutions for different use cases. The repository contains sections on:

• General Notes
  • Usage of Ansible
  • Install Docker with Ansible
  • Creating a Keycloak-Client
• Documentation of the Monitoring Systems
  • Status Monitoring
  • IoT Monitoring
  • Platform Monitoring
  • Security Scan

The objectives and possibilities of the individual sub-areas are presented in the following overview. There is some overlap, as the various approaches are partly redundant.

General Notes

Using Ansible

• Easy setup via Ansible Playbook
• Installation of Ansible

  apt update
  apt install ansible

• Required files
  • playbook.yml: Definition of the necessary steps
  • hosts.ini: Definition of the hosts on which the playbook should be executed
    • Example for local host see ./ansible_basic/hosts.ini
• Running Ansible

  ansible-playbook -i path/to/hosts.ini path/to/playbook_status.yml

• Extension for using a .env
  • Using a bash script ./ansible_basic/ansible-playbook.sh
  • Call:

    ./ansible-playbook.sh -i path/to/hosts.ini path/to/playbook_status.yml

Docker

• Docker is used for deploying the individual services
• Using an Ansible Playbook to install Docker is possible

Creating a Client in Keycloak

In order to use this setup with Keycloak, you will need to set up a client in Keycloak. See the tutorial for this Keycloak connection for more information.

Documentation of the Monitoring Systems

Status Monitoring

• Using Uptime-Kuma:

  • to check the availability of various services
  • to check server certificates
  • to visualize the system status
  • to notify in case of service failures

• You can adjust the language and add your own components. The tool gives you a status page, like this:
  

• It is also possible to use Uptime-Kuma in a simple Docker Compose setup. See the project's documentation for details.

• Securing via oauth2-proxy, so that a login via Keycloak is required.

  • Setting up the connection to Keycloak is necessary
  • Use of a defined group for release is required

• Using Traefik as a reverse proxy

  • Setting up certificates via an ACME HTTP CA-Server (DFN-PKI) (An adaptation is required for other Certresolvers)

• Deployment via Ansible Playbook and Docker Compose (through Ansible):

  • Configuration of Traefik is equipped with environment variables by Ansible

• Using environment variables (.env) in the root directory:

  #Uptime-Kuma
  UPTIME_KUMA_PATH=               # Path to the files of Uptime-Kuma
  UPTIME_KUMA_HOST=               # DNS name / host of Uptime-Kuma
  
  # OAuth Plugin
  OAUTH_PROVIDER_URL=             # Keycloak URL from Keycloak
  OAUTH2_PROXY_CLIENT_ID=         # Client ID from Keycloak
  OAUTH2_PROXY_CLIENT_SECRET=     # Client Secret from Keycloak
  OAUTH2_PROXY_COOKIE_NAME=       # Name for the cookies
  OAUTH2_PROXY_COOKIE_SECRET=     # Self-chosen cookie secret (seed string for secure cookies (optionally base64 encoded))
  OAUTH2_PROXY_ALLOWED_GROUP=     # Allowed groups from Keycloak
  OAUTH2_PROXY_BASIC_DOMAIN=      # Parent domain of UPTIME_KUMA_HOST
  OAUTH2_PROXY_FOOTER=            # Option to add a footer (optional)
  OAUTH2_TEMPLATES_PATH=          # Path for the templates to adjust the login flow (automatic redirect to the Provider)
  OAUTH2_PROVIDER_DISPLAY_NAME=   # Name of the Lofin povider (optional - only a text on the button, if there is no automatic redirect)
  
  # TRAEFIK
  TRAEFIK_PATH=                   # Path to the files of Traefik
  TRAEFIK_ACME_EAB_KID=           # EAB_KID from Sertigo
  TRAEFIK_ACME_EAB_HMACENCODED=   # EAB_HMACENCODED from Sertigo
  TRAEFIK_ACME_EMAIL=             # Email for ACME account
  TRAEFIK_ACME_HTTP_CASERVER=     # Certresolver if the http_resolver is used

IoT-Monitoring

• Monitoring of IoT-Devices via Checkmk with notifications via Rocket.Chat or Matrix

• The tool provides an overview of hardware and software details such as memory usage and load. The first overview shows whether all the checks are OK or if there are any problems. This image shows an example.
  

• Monitoring tool for various IoT devices and systems - see documentation docs.checkmk.com

  • You need to add the devices yourself. See the official documentation for this.
  • The ways in which you can add devices depend on the licence type you use.

• Individual extensions can be added manually, or like the Rocket.Chat Plugin via Ansible Playbook. A Documentation is avaiable under the checkmk Docs. In the Community Edition, plugins could be installed from the command line (in the Docker Container):

  • omd su ${CMK_SITE_ID}
  • mkp add ${path_to_plugin.mkp}
  • mkp enable $ {name_of_plugin}

• The following ENVS are required for deploying:

  CMK_PATH=       # Path where CheckMk stores the monitoring data and configuration.
  CMK_SITE_ID=    # ID of your monitoring site
  CMK_PASSWORD=   # Set an initial password for the cmkadmin user.
  CMK_URL=        # URL / DNS-Name for the CMK-Host
  

• Deployment is possible via Docker Compose or Ansible Playbook: ./ansible_basic/ansible-playbook.sh -i ./ansible_basic/hosts.ini /${PATH_TO}/ansible_service_monitoring/playbook_cmk.yml

• Notifications through Rocket.Chat must be configured through the GUI. For more information (including other notification methods), see the Checkmark Docs.

• The plugin for Notifications via Matrix is deployed via the ansible notebook and needs to be configured via the GUI.

Platform Monitoring

• Monitoring the status of the FIWARE platform components in implementation with Docker Swarm

  • It can also be used to collect data from IoT devices, but implementation is not as straightforward as it is shown above.

• Monitoring by usage of Prometheus with multiple data collectors:

  • cAdvisor:
    • Container Advisor provides information about the resource usage of the running containers.
    • This data is collected for each container deployed in the system.
    • Needed to run on each docker host
  • Node exporter
    • Provides information about the hardware
  • CrateDB:
    • Using the Crate JMX HTTP Exporter
    • Needs to be implemented in the Docker-Container of CreateDB
    • Build your image with the attached dockerfile - see platform_monitoring/cratedb_exporter/

• More data collectors could be used, like:

  • json_exporter
    • Collect data from json http apis, e.g. orion or iot-agent
  • MongoDB exporter
    • The MongoDB exporter provides the metrics exposed by MongoDB monitoring commands
  • A overview about more exporters and intagrations gives the prometheus doc

• Deployment via compose

  • Example for a docker swarm architecure, each data collecor is deployed on each host. Adapt the implementation in your system.
  • Configuration with the prometheus.yml
    • In docker swarm, the configuration could be provided via the external configs
    • For more Information see doc
  • The monitoring data of prometheus should be stored in a volume to keep them outside of the container itself

• Grafana visualisation

  • Grafana could visualize the data from prometheus, like
  • Prometheus could be added as datasource - see prometheus doc
  • A lot of dashboard templates could be used, like:
    • Grafana Cadvisor exporter
    • Grafana CrateDB Monitoring
    • Grafana Node Exporter Full

Security scan

• Information about possible solutions for a security scan are avaiable in this repo.

License

This tutorial is licensed under the MIT License MIT License.

Further project information

Acknowledgments

We gratefully acknowledge the financial support of the Federal Ministry
for Federal Ministry for Economic Affairs and Energy, promotional reference 03EN1030A.