fix(monitoring): make a deploy apply Loki and tautulli-exporter config edits - #422
Merged
Conversation
…g edits Two more single-file bind mounts were missing from the DEP-6 hash list, so ansible-playbook deploy.yml never applied an edit to either. Same shape as #419 fixed for Alloy and #328 for Alertmanager: a git pull replaces the file by rename, the container keeps the old inode, and plain `up -d` no-ops, because Compose reacts to image, env and mount definition changes but never to the bytes of a mounted file. The play still reports changed and nothing errors, so forgetting the hand-run force-recreate looks exactly like success. loki-config.yml. Loki has no reload endpoint at all. Only a separate -runtime-config.file reloads live, and only for the limits and KV store components, which we do not use, so every setting in this file is read once at process start. It holds retention_period and reject_old_samples_max_age, so a retention edit that silently never loaded either deletes logs we believed we were keeping or fills the disk. Both look like a healthy system right up until the damage is done. tautulli-exporter/main.py. It is bind-mounted over /app/main.py and Python reads the source once when the process starts. It is a vendored patch of upstream carrying five local behaviour fixes, two of which define metrics we dashboard on, so an edit that never loaded leaves the exporter serving the old wrong numbers while every health signal stays green. Recreating either is cheap. Loki is down for seconds, well inside LokiDown's 5m and LokiLogIngestionStalled's 20m; Alloy's loki.write retries a refused connection, the ingester WAL sits on the persistent ${APPDATA}/loki mount and replays anything unflushed, and nothing in the project depends_on loki. The exporter loses a few seconds of scrape, covered twice over by its patched 120s startup grace and by ProbeDown's 5m window. monitoring/socket-proxy/haproxy.cfg is deliberately NOT included. It is startup-only too, but DEP-6 recreates one service and that one needs two: restart-socket-consumers.service recreates socket-proxy-monitoring and then runs `docker restart alloy`, because Alloy's docker tailers have to reconnect through the fresh proxy, and Compose depends_on orders startup without restarting dependents. Wiring the label alone would reproduce the first half of that unit and skip the half that keeps Loki ingesting, on the exact path that once went dark for 29h with a 20m detection lag. Proving it either way means recreating socket-proxy on the live server and watching ingestion recover, which needs its own session. The exposure left open is small: that file is a vendored copy of the image's own template, its documented edit trigger is an image bump, and an image pin change already recreates the container by itself. Proof of this PR needs two deploys. The labels: blocks are new, so the first deploy after this merges recreates both services whatever the digest does, and proves only that Compose noticed a new label. A follow-up content-only edit is what tests the fingerprint path, the way #420 did for config.alloy.
PBernaerts
added a commit
that referenced
this pull request
Aug 21, 2026
Loki reads its config once at startup and has no reload endpoint, and Python reads main.py once when the process starts. Both are single-file bind mounts, so a git pull swaps the inode under a running container and a plain `docker compose up -d` applies nothing. Since #422 the DEP-6 hash label does apply an edit, and these headers say so where somebody is standing when they edit the file rather than only in the compose labels. Both files fail quietly if an edit never loads. loki-config.yml holds the retention settings, so a change that did not take either deletes logs you believed you were keeping or fills the disk. main.py carries the exporter's own scrape-health metrics, so stale code keeps reporting healthy while serving the old numbers. This is also the content-only change that proves the fingerprint path: the previous deploy recreated both services because the label block itself was new, which proves nothing about the digest. The next deploy changes bytes only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Finishes the sweep #419/#420 started. Two more single-file bind mounts in the monitoring stack were missing from the DEP-6 hash list, so
ansible-playbook deploy.ymlnever applied an edit to either.Same shape as #419 (Alloy) and #328 (Alertmanager): a
git pullreplaces the file by rename, the container keeps the old inode, and plainup -dno-ops, because Compose reacts to image, env and mount definition changes but never to the bytes of a mounted file. The play still reportschangedand nothing errors, so forgetting the hand-run force-recreate looks exactly like success.Why each one is in
monitoring/loki/loki-config.yml- Loki has no reload endpoint at all. Only a separate-runtime-config.filereloads live (every 10s, and only for the limits and multi-KV-store components), which we do not use; compose passes-config.fileonly and the config declares noruntime_config. So every setting here is read once at process start. It holdsretention_periodandreject_old_samples_max_age, so a retention edit that silently never loaded either deletes logs we believed we were keeping or fills the disk, and both look like a healthy system until the damage is done.monitoring/tautulli-exporter/main.py- bind-mounted over/app/main.py; Python reads the source once when the process starts. It is a vendored patch of upstream carrying five local behaviour fixes, two of which define metrics we dashboard on (tautulli_exporter_last_successful_scrape_timestamp_seconds,tautulli_exporter_scrape_failures_total), so an edit that never loaded leaves the exporter serving the old wrong numbers while every health signal stays green.What a recreate costs
LokiDownisfor: 5m,LokiLogIngestionStalledisfor: 20m; Alloy'sloki.writeretries a refused connection; the ingester WAL is on the persistent${APPDATA}/lokimount and replays anything unflushed; nothing in the projectdepends_onloki/ready, andProbeDownisfor: 5manywayWhy
socket-proxy/haproxy.cfgis deliberately NOT hereIt is startup-only too (a template the linuxserver init renders at container start), but it does not belong in this PR:
restart-socket-consumers.servicerecreatessocket-proxy-monitoringand then runsdocker restart alloy, on its own stated grounds that Alloy's docker tailers must reconnect through the fresh proxy. Composedepends_onorders startup, it does not restart dependents. A hash label alone would reproduce the first half of that unit and skip the half that keeps Loki ingesting.LokiLogIngestionStalledneeding 20 minutes to notice.Proving it needs two deploys, and the first one proves nothing
Neither service had a
labels:key before, so adding one is itself a service-definition change: the first deploy after this merges recreates both whatever the digest does. That deploy proves only that Compose noticed a new label, and it looks exactly like success. Only a later content-only edit tests the fingerprint path. This is why #419 and #420 were split, and a follow-up PR will do the same here.Createdmoves andhomelab.config-shachanges with nothing else differentchanged=0,CreatedunchangedEvidence at each step is
docker inspect -f '{{.Created}}', thehomelab.config-shalabel read back off the container, andsha256sumof the repo file, checking the last two match. Never by reading the config back out of the container: that reads the mount, which is current whether or not the process re-read it.Checks run
docker compose --env-file monitoring/.env.example -f monitoring/compose.yaml config --quiet- OK; all fivehomelab.config-shalabels render, with the:-unsetfallback outside the playbookansible-playbook --syntax-checkon all three entrypoints - OKansible-lintfromansible/- 0 failures, profileproductionpassedbash tests/run.sh- no failurespre-commiton both changed files - all hooks passedLoose end found, not fixed here
main.py's header says it is vendored from upstream v0.2.6 and names the image pin as 0.2.6, butmonitoring/compose.yamlpinsmm404/tautulli-exporter:0.2.7. Its own header says a re-sync is required when the pin changes, so the vendored patch has not been re-diffed against the running image. Not a deploy-application bug, out of scope here, raised for a separate look.