Skip to content

fix(monitoring): make a deploy apply Loki and tautulli-exporter config edits - #422

Merged
PBernaerts merged 1 commit into
mainfrom
fix/dep6-loki-tautulli-hashes
Aug 21, 2026
Merged

fix(monitoring): make a deploy apply Loki and tautulli-exporter config edits#422
PBernaerts merged 1 commit into
mainfrom
fix/dep6-loki-tautulli-hashes

Conversation

@PBernaerts

Copy link
Copy Markdown
Owner

Finishes the sweep #419/#420 started. Two more single-file bind mounts in the monitoring stack were missing from the DEP-6 hash list, so ansible-playbook deploy.yml never applied an edit to either.

Same shape as #419 (Alloy) and #328 (Alertmanager): a git pull replaces the file by rename, the container keeps the old inode, and plain up -d no-ops, because Compose reacts to image, env and mount definition changes but never to the bytes of a mounted file. The play still reports changed and nothing errors, so forgetting the hand-run force-recreate looks exactly like success.

Why each one is in

monitoring/loki/loki-config.yml - Loki has no reload endpoint at all. Only a separate -runtime-config.file reloads live (every 10s, and only for the limits and multi-KV-store components), which we do not use; compose passes -config.file only and the config declares no runtime_config. So every setting here is read once at process start. It holds retention_period and reject_old_samples_max_age, so a retention edit that silently never loaded either deletes logs we believed we were keeping or fills the disk, and both look like a healthy system until the damage is done.

monitoring/tautulli-exporter/main.py - bind-mounted over /app/main.py; Python reads the source once when the process starts. It is a vendored patch of upstream carrying five local behaviour fixes, two of which define metrics we dashboard on (tautulli_exporter_last_successful_scrape_timestamp_seconds, tautulli_exporter_scrape_failures_total), so an edit that never loaded leaves the exporter serving the old wrong numbers while every health signal stays green.

What a recreate costs

Service Cost Covered by
loki seconds of downtime LokiDown is for: 5m, LokiLogIngestionStalled is for: 20m; Alloy's loki.write retries a refused connection; the ingester WAL is on the persistent ${APPDATA}/loki mount and replays anything unflushed; nothing in the project depends_on loki
tautulli-exporter a few seconds of scrape gap the local patch's bounded 120s startup grace on /ready, and ProbeDown is for: 5m anyway

Why socket-proxy/haproxy.cfg is deliberately NOT here

It is startup-only too (a template the linuxserver init renders at container start), but it does not belong in this PR:

  1. DEP-6 recreates one service; this one needs two. restart-socket-consumers.service recreates socket-proxy-monitoring and then runs docker restart alloy, on its own stated grounds that Alloy's docker tailers must reconnect through the fresh proxy. Compose depends_on orders startup, it does not restart dependents. A hash label alone would reproduce the first half of that unit and skip the half that keeps Loki ingesting.
  2. The failure it could introduce is the silent one - the exact path that went dark for ~29h on 2026-07-18, with LokiLogIngestionStalled needing 20 minutes to notice.
  3. Proving it either way is a live test of the most fragile path in the stack: deliberately recreate socket-proxy on the server and watch whether ingestion recovers without the alloy bounce. That needs its own session, plan and rollback.
  4. The exposure left open is small. The file is a vendored copy of the image's own template, its documented edit trigger is "after an image bump, diff against the new tag's template and re-apply", and an image pin change already recreates the container by itself. A standalone edit is possible but rare, and its failure mode (the 10-minute EOF warnings returning to the anomaly panel) is visible rather than silent.

Proving it needs two deploys, and the first one proves nothing

Neither service had a labels: key before, so adding one is itself a service-definition change: the first deploy after this merges recreates both whatever the digest does. That deploy proves only that Compose noticed a new label, and it looks exactly like success. Only a later content-only edit tests the fingerprint path. This is why #419 and #420 were split, and a follow-up PR will do the same here.

Deploy Change since previous What it proves
1 (this PR) label block added nothing about the digest path; both recreate on the definition change
2 (follow-up, content-only) file bytes only the fingerprint works: Created moves and homelab.config-sha changes with nothing else different
3 (re-run, no change) none idempotence: changed=0, Created unchanged

Evidence at each step is docker inspect -f '{{.Created}}', the homelab.config-sha label read back off the container, and sha256sum of the repo file, checking the last two match. Never by reading the config back out of the container: that reads the mount, which is current whether or not the process re-read it.

Checks run

  • docker compose --env-file monitoring/.env.example -f monitoring/compose.yaml config --quiet - OK; all five homelab.config-sha labels render, with the :-unset fallback outside the playbook
  • ansible-playbook --syntax-check on all three entrypoints - OK
  • ansible-lint from ansible/ - 0 failures, profile production passed
  • bash tests/run.sh - no failures
  • pre-commit on both changed files - all hooks passed

Loose end found, not fixed here

main.py's header says it is vendored from upstream v0.2.6 and names the image pin as 0.2.6, but monitoring/compose.yaml pins mm404/tautulli-exporter:0.2.7. Its own header says a re-sync is required when the pin changes, so the vendored patch has not been re-diffed against the running image. Not a deploy-application bug, out of scope here, raised for a separate look.

…g edits

Two more single-file bind mounts were missing from the DEP-6 hash list, so
ansible-playbook deploy.yml never applied an edit to either. Same shape as
#419 fixed for Alloy and #328 for Alertmanager: a git pull replaces the file
by rename, the container keeps the old inode, and plain `up -d` no-ops,
because Compose reacts to image, env and mount definition changes but never
to the bytes of a mounted file. The play still reports changed and nothing
errors, so forgetting the hand-run force-recreate looks exactly like success.

loki-config.yml. Loki has no reload endpoint at all. Only a separate
-runtime-config.file reloads live, and only for the limits and KV store
components, which we do not use, so every setting in this file is read once
at process start. It holds retention_period and reject_old_samples_max_age,
so a retention edit that silently never loaded either deletes logs we
believed we were keeping or fills the disk. Both look like a healthy system
right up until the damage is done.

tautulli-exporter/main.py. It is bind-mounted over /app/main.py and Python
reads the source once when the process starts. It is a vendored patch of
upstream carrying five local behaviour fixes, two of which define metrics we
dashboard on, so an edit that never loaded leaves the exporter serving the
old wrong numbers while every health signal stays green.

Recreating either is cheap. Loki is down for seconds, well inside LokiDown's
5m and LokiLogIngestionStalled's 20m; Alloy's loki.write retries a refused
connection, the ingester WAL sits on the persistent ${APPDATA}/loki mount
and replays anything unflushed, and nothing in the project depends_on loki.
The exporter loses a few seconds of scrape, covered twice over by its
patched 120s startup grace and by ProbeDown's 5m window.

monitoring/socket-proxy/haproxy.cfg is deliberately NOT included. It is
startup-only too, but DEP-6 recreates one service and that one needs two:
restart-socket-consumers.service recreates socket-proxy-monitoring and then
runs `docker restart alloy`, because Alloy's docker tailers have to
reconnect through the fresh proxy, and Compose depends_on orders startup
without restarting dependents. Wiring the label alone would reproduce the
first half of that unit and skip the half that keeps Loki ingesting, on the
exact path that once went dark for 29h with a 20m detection lag. Proving it
either way means recreating socket-proxy on the live server and watching
ingestion recover, which needs its own session. The exposure left open is
small: that file is a vendored copy of the image's own template, its
documented edit trigger is an image bump, and an image pin change already
recreates the container by itself.

Proof of this PR needs two deploys. The labels: blocks are new, so the first
deploy after this merges recreates both services whatever the digest does,
and proves only that Compose noticed a new label. A follow-up content-only
edit is what tests the fingerprint path, the way #420 did for config.alloy.
@PBernaerts
PBernaerts merged commit ae8f626 into main Aug 21, 2026
9 checks passed
@PBernaerts
PBernaerts deleted the fix/dep6-loki-tautulli-hashes branch August 21, 2026 13:09
PBernaerts added a commit that referenced this pull request Aug 21, 2026
Loki reads its config once at startup and has no reload endpoint, and Python
reads main.py once when the process starts. Both are single-file bind mounts,
so a git pull swaps the inode under a running container and a plain
`docker compose up -d` applies nothing. Since #422 the DEP-6 hash label does
apply an edit, and these headers say so where somebody is standing when they
edit the file rather than only in the compose labels.

Both files fail quietly if an edit never loads. loki-config.yml holds the
retention settings, so a change that did not take either deletes logs you
believed you were keeping or fills the disk. main.py carries the exporter's own
scrape-health metrics, so stale code keeps reporting healthy while serving the
old numbers.

This is also the content-only change that proves the fingerprint path: the
previous deploy recreated both services because the label block itself was new,
which proves nothing about the digest. The next deploy changes bytes only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant