Skip to content

[Chore] 모니터링 알림을 디스코드로 연결하고 대시보드의 빈 패널과 틀린 값을 고친다 #336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
KiSeungMin merged 1 commit into from
Sep 21, 2026
Merged

[Chore] 모니터링 알림을 디스코드로 연결하고 대시보드의 빈 패널과 틀린 값을 고친다 #336

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/ci-dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,13 +158,13 @@ jobs:
- name: Deploy Monitoring Stack
run: |
docker-compose -f docker-compose.dev.yml --env-file .env.dev --profile monitoring pull \
mysql-exporter-dev prometheus-dev loki-dev promtail-dev grafana-dev
docker-compose -f docker-compose.dev.yml --env-file .env.dev --profile monitoring stop prometheus-dev grafana-dev loki-dev promtail-dev mysql-exporter-dev || true
docker-compose -f docker-compose.dev.yml --env-file .env.dev --profile monitoring rm -f prometheus-dev grafana-dev loki-dev promtail-dev mysql-exporter-dev || true
mysql-exporter-dev redis-exporter-dev prometheus-dev loki-dev promtail-dev grafana-dev
docker-compose -f docker-compose.dev.yml --env-file .env.dev --profile monitoring stop prometheus-dev grafana-dev loki-dev promtail-dev mysql-exporter-dev redis-exporter-dev || true
docker-compose -f docker-compose.dev.yml --env-file .env.dev --profile monitoring rm -f prometheus-dev grafana-dev loki-dev promtail-dev mysql-exporter-dev redis-exporter-dev || true
docker-compose -f docker-compose.dev.yml --env-file .env.dev --profile monitoring up -d --force-recreate \
mysql-exporter-dev prometheus-dev loki-dev promtail-dev grafana-dev
mysql-exporter-dev redis-exporter-dev prometheus-dev loki-dev promtail-dev grafana-dev
docker-compose -f docker-compose.dev.yml --env-file .env.dev ps \
mysql-exporter-dev prometheus-dev loki-dev promtail-dev grafana-dev
mysql-exporter-dev redis-exporter-dev prometheus-dev loki-dev promtail-dev grafana-dev
shell: bash

- name: Verify Monitoring Integration
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/ci-prod.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,13 +219,13 @@ jobs:
- name: Deploy Monitoring Stack
run: |
docker-compose -f docker-compose.prod.yml --env-file .env.prod --profile monitoring pull \
mysql-exporter-prod prometheus-prod loki-prod promtail-prod grafana-prod
docker-compose -f docker-compose.prod.yml --env-file .env.prod --profile monitoring stop prometheus-prod grafana-prod loki-prod promtail-prod mysql-exporter-prod || true
docker-compose -f docker-compose.prod.yml --env-file .env.prod --profile monitoring rm -f prometheus-prod grafana-prod loki-prod promtail-prod mysql-exporter-prod || true
mysql-exporter-prod redis-exporter-prod alertmanager-prod prometheus-prod loki-prod promtail-prod grafana-prod
docker-compose -f docker-compose.prod.yml --env-file .env.prod --profile monitoring stop prometheus-prod grafana-prod loki-prod promtail-prod mysql-exporter-prod redis-exporter-prod alertmanager-prod || true
docker-compose -f docker-compose.prod.yml --env-file .env.prod --profile monitoring rm -f prometheus-prod grafana-prod loki-prod promtail-prod mysql-exporter-prod redis-exporter-prod alertmanager-prod || true
docker-compose -f docker-compose.prod.yml --env-file .env.prod --profile monitoring up -d --force-recreate \
mysql-exporter-prod prometheus-prod loki-prod promtail-prod grafana-prod
mysql-exporter-prod redis-exporter-prod alertmanager-prod prometheus-prod loki-prod promtail-prod grafana-prod
docker-compose -f docker-compose.prod.yml --env-file .env.prod ps \
mysql-exporter-prod prometheus-prod loki-prod promtail-prod grafana-prod
mysql-exporter-prod redis-exporter-prod alertmanager-prod prometheus-prod loki-prod promtail-prod grafana-prod
shell: bash

- name: Verify Monitoring Integration
Expand Down
9 changes: 5 additions & 4 deletions .github/workflows/restart-monitoring.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ jobs:
RABBITMQ_PASSWORD=${{ secrets.RABBITMQ_PASSWORD }}
GRAFANA_ADMIN_USER=${{ secrets.GRAFANA_ADMIN_USER }}
GRAFANA_ADMIN_PASSWORD=${{ secrets.GRAFANA_ADMIN_PASSWORD }}
DISCORD_WEBHOOK_URL=${{ secrets.DISCORD_WEBHOOK_URL }}
GRAFANA_PROD_ROOT_URL=https://ono-prod.seungminki.shop/grafana/
GRAFANA_PROD_SERVE_FROM_SUB_PATH=true
PROMETHEUS_PROD_EXTERNAL_URL=https://ono-prod.seungminki.shop/prometheus/
Expand All @@ -56,16 +57,16 @@ jobs:
if: ${{ github.event.inputs.target == 'dev' }}
run: |
docker compose -f docker-compose.dev.yml --env-file .env.dev --profile monitoring up -d --force-recreate \
mysql-exporter-dev prometheus-dev loki-dev promtail-dev grafana-dev
mysql-exporter-dev redis-exporter-dev prometheus-dev loki-dev promtail-dev grafana-dev
docker compose -f docker-compose.dev.yml --env-file .env.dev ps \
mysql-exporter-dev prometheus-dev loki-dev promtail-dev grafana-dev
mysql-exporter-dev redis-exporter-dev prometheus-dev loki-dev promtail-dev grafana-dev
shell: bash

- name: Restart monitoring stack (prod)
if: ${{ github.event.inputs.target == 'prod' }}
run: |
docker compose -f docker-compose.prod.yml --env-file .env.prod --profile monitoring up -d --force-recreate \
mysql-exporter-prod prometheus-prod loki-prod promtail-prod grafana-prod
mysql-exporter-prod redis-exporter-prod alertmanager-prod prometheus-prod loki-prod promtail-prod grafana-prod
docker compose -f docker-compose.prod.yml --env-file .env.prod ps \
mysql-exporter-prod prometheus-prod loki-prod promtail-prod grafana-prod
mysql-exporter-prod redis-exporter-prod alertmanager-prod prometheus-prod loki-prod promtail-prod grafana-prod
shell: bash
13 changes: 13 additions & 0 deletions docker-compose.dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,19 @@ services:
profiles:
- monitoring

redis-exporter-dev:
image: oliver006/redis_exporter:v1.67.0
container_name: ono-redis-exporter-dev
restart: unless-stopped
environment:
REDIS_ADDR: redis://redis-dev:6379
depends_on:
- redis-dev
networks:
- ono-network
profiles:
- monitoring

# Grafana - DEVELOPMENT (Profile: monitoring)
grafana-dev:
image: grafana/grafana:latest
Expand Down
37 changes: 37 additions & 0 deletions docker-compose.prod.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,41 @@ services:
profiles:
- monitoring

redis-exporter-prod:
image: oliver006/redis_exporter:v1.67.0
container_name: ono-redis-exporter-prod
restart: unless-stopped
environment:
REDIS_ADDR: redis://redis-prod:6379
depends_on:
- redis-prod
networks:
- ono-network
profiles:
- monitoring

# Alertmanager - PRODUCTION (Profile: monitoring)
# Prometheus 알림을 디스코드로 보낸다. 웹훅 주소를 설정 파일에 적지 않으려고
# 뜰 때 환경변수를 파일로 옮기고 alertmanager.prod.yml 의 webhook_url_file 로 읽는다.
alertmanager-prod:
image: prom/alertmanager:v0.28.1
container_name: ono-alertmanager-prod
restart: unless-stopped
environment:
DISCORD_WEBHOOK_URL: ${DISCORD_WEBHOOK_URL}
entrypoint: ["/bin/sh", "-c"]
command:
- |
printf '%s' "$$DISCORD_WEBHOOK_URL" > /tmp/discord_webhook_url
exec /bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager --cluster.listen-address=
volumes:
- ./monitoring/alertmanager.prod.yml:/etc/alertmanager/alertmanager.yml:ro
- alertmanager_prod_data:/alertmanager
networks:
- ono-network
profiles:
- monitoring

# Grafana - PRODUCTION (Profile: monitoring)
grafana-prod:
image: grafana/grafana:latest
Expand Down Expand Up @@ -320,3 +355,5 @@ volumes:
driver: local
grafana_prod_data:
driver: local
alertmanager_prod_data:
driver: local
19 changes: 19 additions & 0 deletions monitoring/alertmanager.prod.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Prometheus 가 평가한 알림(monitoring/alerts/ono-alerts.yml)을 디스코드로 보낸다.
# 웹훅 주소는 앱 에러 알림과 같은 DISCORD_WEBHOOK_URL 을 쓰고, 컨테이너가 뜰 때
# 환경변수를 파일로 옮겨 둔다(docker-compose.prod.yml 의 alertmanager-prod 참고).
# 설정 파일에 비밀값을 적지 않기 위해 webhook_url 대신 webhook_url_file 을 쓴다.

route:
receiver: discord
group_by: [alertname, job]
# 같은 알림이 여러 개 동시에 뜨면 30초 모아서 한 번에 보낸다.
group_wait: 30s
group_interval: 5m
# 해결되지 않은 알림은 4시간마다 다시 알린다.
repeat_interval: 4h

receivers:
- name: discord
discord_configs:
- webhook_url_file: /tmp/discord_webhook_url
send_resolved: true
Loading
Loading