Setup with Graylog, OpenSearch, and Filebeat all running in Docker containers.
The webinterfaces of Graylog and OpenSearch-Dashboard will be available through the reverse proxy Traefik.
The goal is to gather logs from various servers on a central Logging server. In particular, logs from docker containers and pods in a Kubernetes cluster should be available in a web dashboard.
In this section we set up the central logging server. It serves following services:
- Graylog
- OpenSearch
- OpenSearch-Dashboard
- Mongo
- cert-extract - make acme certificates from Traefik available in Graylog
- Traefik - reverse proxy
The setup is tested using Ubuntu 22.04 - but it should work on other Linux as well.
- Subdomain for Graylog
- Subdomain for OpenSearch-Dashboard
- Server
Open ports mentioned in docker-compose.host.yml and docker-compose.traefik.yml in your firewall. For more details, have a look at the section VM Security Groups.
If you are using a simple software firewall like Firewalld or UFW, keep in mind that docker bypasses all firewall rules in default configuration.
Create DNS Entries for subdomains for graylog and opensearch-dashboard service.
Clone this repository on your node and cd into the directory.
Start reverse proxy Traefik if it is not already up and running on the node:
docker compose -f docker-compose.traefik.yml up -dReplace placeholder domain for graylog service with your subdomain in all files:
find . -type f -exec sed -i 's/logs.placeholderdomain.com/logs.mydomain.com/g' {} +The domain for graylog is used in multiple files such as
filebeat.ymldocker-compose.host.ymlandkubernetes-deploy/filebeat-daemonset.yaml
Replace placeholder domain for opensearch-dashboard service with your subdomain in all files:
find . -type f -exec sed -i 's/os.placeholderdomain.com/os.mydomain.com/g' {} +Copy .env.graylog.template and rename to .env.graylog:
cp .env.graylog.template .env.graylogSet variables GRAYLOG_ROOT_PASSWORD_SHA2 and GRAYLOG_PASSWORD_SECRET in .env.graylog.
Copy .env.opensearch.template and rename to .env.opensearch:
cp .env.opensearch.template .env.opensearchSet variables OPENSEARCH_INITIAL_ADMIN_PASSWORD in .env.opensearch.
Set path to acme.json (letsencrypt dir in traefik) in cert-extract service in docker-compose.host.yml. To do so you may search for CHANGE PATH TO acme.json dir
Start services:
docker compose -f docker-compose.host.yml up -dThe Graylog and OpenSearch-Dashboard webinterface should be available within 2 minutes.
Set auth in OpenSearch-Dashboard webinterface.
The default username and password is admin.
In order to restrict access to the VM security groups can be used. To allow web browsers and log agents like filebeat to access Graylog, some security groups have to be added to host VM:
- 80 (http)
- 443 (https)
- 5044
- 1514
- 12201
You also may want to restrict access for certain subnet or hosts
To enable TLS for Inputs, certificates are needed.
In this setup, these can be extracted from Traefik.
For that following image is used in docker-compose.host.yml: https://hub.docker.com/r/soerenmetje/acme-certs-extract
The volume ./certs is used for sharing certificates.
In Graylog, go to Input configuration and set TLS cert file to /usr/share/graylog/certs/flat/logs.placeholderdomain.com_full.crt
and TLS private key file to /usr/share/graylog/certs/flat/logs.placeholderdomain.com.key.
Graylog plugins can be used too.
Put the plugin .jar files in the ./plugins directory.
The directory ./plugins is mapped by a volume into the Graylog container.
- Graylog plugin for Telegram alert notifications: TelegramAlert
- System Logs: Syslog
- Docker Logs: FileBeat
Syslog can be used to gather System logs.
When starting with plain Graylog, create a Input for Syslog first. This is an input type not a single machine.
Configure the machine to push the logs to the configured Graylog input
Create file /etc/rsyslog.d/graylog.conf and add following content:
*.* @@logs.placeholderdomain.com:1514;RSYSLOG_SyslogProtocol23Format
# @@ = TCP
# currently unsecure
Reload config:
sudo systemctl restart rsyslog.service
sudo systemctl status rsyslog.service Fluent Bit is a log-shipper and can be used to gathering logs from f.e. Docker and Kubernetes.
First, in Graylog, create an Input for GELF TCP first. This input can be used by multiple GELF services.
TODO
Change host in kubernetes-deploy/fluentbit-helm-values.yaml to your graylog subdomain, if not already done.
Also, you may want to check the Fluent Bit configuration in kubernetes-deploy/fluentbit-helm-values.yaml under tag config.
Deploy Fluent Bit:
kubectl create namespace fluentbit
helm repo add fluent https://fluent.github.io/helm-charts
helm upgrade --install fluent-bit fluent/fluent-bit -f kubernetes-deploy/fluentbit-helm-values.yaml -n fluentbitFileBeat is a log-shipper and can be used to gathering logs from f.e. Docker.
When starting with plain Graylog, create a Input for FileBeat first. This is a general input for multiple FileBeat services.
Change hosts in filebeat.yml to your graylog subdomain, if not already done.
Start FileBeat:
docker compose -f docker-compose.filebeat.yml up -dChange hosts in kubernetes-deploy/filebeat-daemonset.yaml to your graylog subdomain, if not already done.
Start FileBeat:
kubectl apply -f ./kubernetes-deployNamespace is set to
kube-systemin yaml-files
- Graylog Docs Docker https://go2docs.graylog.org/5-0/downloading_and_installing_graylog/docker_installation.htm
- Graylog & co @ Docker https://docs.graylog.org/docs/docker
- Fluent Bit @ Kubernetes https://docs.fluentbit.io/manual/installation/kubernetes
- Graylog OpenSearch: https://go2docs.graylog.org/5-0/planning_your_deployment/upgrading_to_opensearch_-_installation.htm