feat: enrich monitoring with structured logs and new exporters - #538
Merged
Merged
Conversation
Extract unit, priority, and app labels from the systemd journal on all hosts (glyph, spore, zeta) so Loki queries can be scoped by service unit, syslog severity, and identifier without full-text search. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Entire-Checkpoint: d908b189c88f
postgres_exporter covers query throughput, connections, and vacuum stats for the five databases on glyph. smartctl_exporter surfaces disk health, temperature, and reallocated sectors — important for a NAS. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Entire-Checkpoint: f616080a78e1
Enables the built-in /metrics endpoint in navidrome and adds a Prometheus scrape job for it. Covers library size, play counts, active streams, and scan duration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Entire-Checkpoint: 56d6f0a22081
Adds a localhost-only nginx stub_status endpoint on port 9080 and the prometheus nginx_exporter on port 9113. Glyph's Prometheus scrapes it for request rate, active connections, and error metrics. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Entire-Checkpoint: 514b9becae1a
Adds a Monitoring Stack section covering Loki label schema, Prometheus jobs/exporters, and common LogQL/PromQL patterns. Instructs agents to use the grafana MCP proactively instead of journalctl or SSH. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Entire-Checkpoint: 61096eefaeb4
stackptr
force-pushed
the
feat-monitoring-enrichment
branch
from
August 26, 2026 22:19
35f965a to
9d5329d
Compare
The pipeline approach (forward_to loki.relabel) strips __journal__* meta-labels before they reach the relabel component. relabel_rules on loki.source.journal applies rules during the read phase while those labels are still available; discovery.relabel provides the correct RelabelRules export type for that argument. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Entire-Checkpoint: 72d76598b42b
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Entire-Checkpoint: 0cfc43778db5
Fields without a leading underscore (PRIORITY, SYSLOG_IDENTIFIER) map to __journal_<field> with a single underscore, not double. Only fields with a leading underscore like _SYSTEMD_UNIT produce the double underscore form __journal__systemd_unit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Entire-Checkpoint: e1dd3242a97b
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Entire-Checkpoint: 05efbccc8f54
stackptr
added a commit
that referenced
this pull request
Sep 2, 2026
Resolve conflict in navidrome.nix: keep Prometheus.Enabled (added on main by #538) alongside the apple-music/nd-lyrics plugin config. Entire-Checkpoint: c5af4ce18fcb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
unit,priority, andapplabels from the systemd journal on glyph, spore, and zeta, enabling scoped LogQL queries like{host="glyph", unit="navidrome.service", priority=~"[0-3]"}without full-text searchpostgres_exporter(port 9187) andsmartctl_exporter(port 9633) cover DB health across the five local databases and disk SMART data for the NAS/metricsendpoint and adds a Prometheus scrape job for library size, play counts, and scan durationnginx_exporteron port 9113; Prometheus on glyph scrapes it for request rate and connection statsDeploy order
Test plan
unit,priority, andapplabelsglyph:9099/targets—postgres,smartctl, andnavidromejobs should show UPcurl -s localhost:9187/metrics | grep pg_up→ should returnpg_up 1curl -s localhost:9633/metrics | grep smartctl_device→ should list drivescurl -s localhost:4533/metrics | grep navidrome_→ should list library statscurl -s http://127.0.0.1:9080/nginx_status→ should show active connectionsnginxjob in Prometheus targets →spore.note-iwato.ts.net:9113should show UPunit/priority/applabels are missing from Loki after deploy, checkjournalctl -u alloy— Alloy will log warnings for any unresolved source label names🤖 Generated with Claude Code