docs(monitoring): say in config.alloy why an edit needs a deploy - #420
Merged
Conversation
Alloy reads this file once at startup and it is a single-file bind mount, so a git pull replaces it by rename and the container keeps the old inode. Plain `docker compose up -d` applies nothing. Since #419 the DEP-6 hash label does apply it, and this header says so at the place somebody is standing when they edit the file, rather than only in the role defaults. It matters here more than for most configs: these stages decide which log lines reach Loki at all, so a stage that silently never loaded keeps discarding lines we meant to keep, and the empty query reads as health.
PBernaerts
added a commit
that referenced
this pull request
Aug 21, 2026
…g edits (#422) Two more single-file bind mounts were missing from the DEP-6 hash list, so ansible-playbook deploy.yml never applied an edit to either. Same shape as #419 fixed for Alloy and #328 for Alertmanager: a git pull replaces the file by rename, the container keeps the old inode, and plain `up -d` no-ops, because Compose reacts to image, env and mount definition changes but never to the bytes of a mounted file. The play still reports changed and nothing errors, so forgetting the hand-run force-recreate looks exactly like success. loki-config.yml. Loki has no reload endpoint at all. Only a separate -runtime-config.file reloads live, and only for the limits and KV store components, which we do not use, so every setting in this file is read once at process start. It holds retention_period and reject_old_samples_max_age, so a retention edit that silently never loaded either deletes logs we believed we were keeping or fills the disk. Both look like a healthy system right up until the damage is done. tautulli-exporter/main.py. It is bind-mounted over /app/main.py and Python reads the source once when the process starts. It is a vendored patch of upstream carrying five local behaviour fixes, two of which define metrics we dashboard on, so an edit that never loaded leaves the exporter serving the old wrong numbers while every health signal stays green. Recreating either is cheap. Loki is down for seconds, well inside LokiDown's 5m and LokiLogIngestionStalled's 20m; Alloy's loki.write retries a refused connection, the ingester WAL sits on the persistent ${APPDATA}/loki mount and replays anything unflushed, and nothing in the project depends_on loki. The exporter loses a few seconds of scrape, covered twice over by its patched 120s startup grace and by ProbeDown's 5m window. monitoring/socket-proxy/haproxy.cfg is deliberately NOT included. It is startup-only too, but DEP-6 recreates one service and that one needs two: restart-socket-consumers.service recreates socket-proxy-monitoring and then runs `docker restart alloy`, because Alloy's docker tailers have to reconnect through the fresh proxy, and Compose depends_on orders startup without restarting dependents. Wiring the label alone would reproduce the first half of that unit and skip the half that keeps Loki ingesting, on the exact path that once went dark for 29h with a 20m detection lag. Proving it either way means recreating socket-proxy on the live server and watching ingestion recover, which needs its own session. The exposure left open is small: that file is a vendored copy of the image's own template, its documented edit trigger is an image bump, and an image pin change already recreates the container by itself. Proof of this PR needs two deploys. The labels: blocks are new, so the first deploy after this merges recreates both services whatever the digest does, and proves only that Compose noticed a new label. A follow-up content-only edit is what tests the fingerprint path, the way #420 did for config.alloy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge this AFTER #419, not before. It is deliberately a content-only change
to
monitoring/alloy/config.alloy, and that ordering is what turns it into theproof that #419 works.
What it adds
A header comment on
config.alloyexplaining that Alloy reads the file once atstartup, that it is a single-file bind mount whose inode a
git pullswaps, andthat what actually applies an edit is the DEP-6
homelab.config-shalabel addedin #419. The warning belongs where somebody is standing when they edit the file,
not only in the Ansible role defaults.
The consequence is worth stating there too: these stages decide which log lines
reach Loki at all, so a stage that silently never loaded keeps discarding lines
we meant to keep, and the empty query reads as health.
Why it exists as its own PR
#419 adds a
labels:block to a service that had none. That is a change to theservice definition, so the first deploy after #419 recreates alloy no matter
what the digest does. Proving the fix with that deploy proves only that Compose
noticed a new label.
The digest path needs a deploy where the ONLY thing that changed is the bytes of
config.alloy. That is exactly this PR. Before #419 this change would haverecreated nothing at all.
Proof sequence on the server, after both are merged
docker inspect alloy --format '{{.Created}}'and thehomelab.config-shalabel.deploy.yml. Alloy recreates (new label block).deploy.ymlagain with no changes:changed=0(DEP-4).deploy.yml, no--force-recreate. Alloymust recreate and the label digest must differ from step 3. That is the
proof.
deploy.ymlagain:changed=0.Re-read the baseline immediately before step 4 rather than trusting an older
reading. On a shared checkout another session's correct, unrelated deploy can
move it, and reasoning after the fact about which deploy caused a recreate is
exactly the ambiguity this sequence exists to avoid.
Checks
alloy validateagainst the pinnedgrafana/alloy:v1.18.1, exit 0, so thecomment does not break the config that carries every log line to Loki
pre-commit run --all-filesgreen