fix(monitoring): make a deploy actually apply an Alloy config change - #419
Conversation
monitoring/alloy/config.alloy was missing from the DEP-6 hash list, so ansible-playbook deploy.yml never applied an edit to it. The file is a single-file bind mount: a git pull replaces it by rename, the container keeps the old inode, and plain `up -d` no-ops. Every Alloy change needed a hand-run `docker compose up -d --force-recreate alloy`, and forgetting it looked exactly like success, because the play reports changed, nothing errors, and the old pipeline keeps running. That matters more here than for most configs. This file decides which log lines reach Loki at all, so a drop stage that silently never loaded means the lines we meant to keep are still being discarded, and the query that returns nothing reads as health. Same fix #328 used for alertmanager.yml: one entry in compose_deploy_config_hashes plus the injected digest as a service label, so a content change recreates just alloy. A recreate is cheap here because read positions live in the alloy data volume, so no old container logs are replayed into Loki.
Alloy reads this file once at startup and it is a single-file bind mount, so a git pull replaces it by rename and the container keeps the old inode. Plain `docker compose up -d` applies nothing. Since #419 the DEP-6 hash label does apply it, and this header says so at the place somebody is standing when they edit the file, rather than only in the role defaults. It matters here more than for most configs: these stages decide which log lines reach Loki at all, so a stage that silently never loaded keeps discarding lines we meant to keep, and the empty query reads as health.
Deployed and proven on the server, 2026-08-21Sequence run exactly as written above, baseline re-read immediately before each step.
Step 4 is the proof. The only difference in that pull was the bytes of The injected digest is genuinely the file's hash, not just a value that changes: Step 2's recreate is the service-definition change, as predicted, and proves Correction to this PR bodyThe "Does this make deploys recreate alloy more often?" section above claims a Every alloy recreate produces one This is pre-existing and not caused by this PR. The identical error, same What it drops is only entries older than Loki's acceptance window, which could Ingestion verified healthy after the final deploy: 1048 lines in the last minute |
…g edits (#422) Two more single-file bind mounts were missing from the DEP-6 hash list, so ansible-playbook deploy.yml never applied an edit to either. Same shape as #419 fixed for Alloy and #328 for Alertmanager: a git pull replaces the file by rename, the container keeps the old inode, and plain `up -d` no-ops, because Compose reacts to image, env and mount definition changes but never to the bytes of a mounted file. The play still reports changed and nothing errors, so forgetting the hand-run force-recreate looks exactly like success. loki-config.yml. Loki has no reload endpoint at all. Only a separate -runtime-config.file reloads live, and only for the limits and KV store components, which we do not use, so every setting in this file is read once at process start. It holds retention_period and reject_old_samples_max_age, so a retention edit that silently never loaded either deletes logs we believed we were keeping or fills the disk. Both look like a healthy system right up until the damage is done. tautulli-exporter/main.py. It is bind-mounted over /app/main.py and Python reads the source once when the process starts. It is a vendored patch of upstream carrying five local behaviour fixes, two of which define metrics we dashboard on, so an edit that never loaded leaves the exporter serving the old wrong numbers while every health signal stays green. Recreating either is cheap. Loki is down for seconds, well inside LokiDown's 5m and LokiLogIngestionStalled's 20m; Alloy's loki.write retries a refused connection, the ingester WAL sits on the persistent ${APPDATA}/loki mount and replays anything unflushed, and nothing in the project depends_on loki. The exporter loses a few seconds of scrape, covered twice over by its patched 120s startup grace and by ProbeDown's 5m window. monitoring/socket-proxy/haproxy.cfg is deliberately NOT included. It is startup-only too, but DEP-6 recreates one service and that one needs two: restart-socket-consumers.service recreates socket-proxy-monitoring and then runs `docker restart alloy`, because Alloy's docker tailers have to reconnect through the fresh proxy, and Compose depends_on orders startup without restarting dependents. Wiring the label alone would reproduce the first half of that unit and skip the half that keeps Loki ingesting, on the exact path that once went dark for 29h with a 20m detection lag. Proving it either way means recreating socket-proxy on the live server and watching ingestion recover, which needs its own session. The exposure left open is small: that file is a vendored copy of the image's own template, its documented edit trigger is an image bump, and an image pin change already recreates the container by itself. Proof of this PR needs two deploys. The labels: blocks are new, so the first deploy after this merges recreates both services whatever the digest does, and proves only that Compose noticed a new label. A follow-up content-only edit is what tests the fingerprint path, the way #420 did for config.alloy.
* ci(monitoring): parse config.alloy before it can merge No gate ever asked Alloy whether config.alloy is valid. tests/run.sh greps the drop expressions out of the file as text, so a stray brace or a forward_to pointing at a component that does not exist passed every check, merged, and failed at container start. Since #419 a config.alloy edit recreates the alloy container, so that failure stops log ingestion, and it presents as silence: every metric stays healthy and only LokiLogIngestionStalled notices, twenty minutes later. `alloy validate` provisions the component graph, so it catches the dangling reference as well as the syntax error. The image tag is read out of monitoring/compose.yaml so a Renovate bump carries the gate along rather than leaving a second pin to maintain by hand. Task CV-1. * PROOF ONLY, reverted in the next commit: dangling component reference CV-4 wants the new gate watched failing, not assumed. This points the docker source at loki.process.typo_containers, which does not exist. * Revert "PROOF ONLY, reverted in the next commit: dangling component reference" This reverts commit 889f9ff.
1 participant
What was wrong
monitoring/alloy/config.alloywas not in the DEP-6 hash list(
compose_deploy_config_hashes), soansible-playbook playbooks/deploy.ymlnever applied an edit to it. It is a single-file bind mount: a
git pullreplaces the file by rename, the container keeps the old inode, and plain
up -dno-ops. Every Alloy change needed a hand-rundocker compose up -d --force-recreate alloy, and forgetting it looked exactlylike success: the play reports changed, nothing errors, and the old pipeline
keeps running.
Why it is worse than a normal stale config
What this file controls is which log lines reach Loki at all. A drop stage that
silently never loaded means the lines we meant to keep are still being
discarded, and the query that returns nothing reads as health. Same failure
shape as #401 (a month of alerts going nowhere while the channel looked quiet),
with a deploy-time cause instead of an HTTP one.
The fix
Exactly what #328 did for
alertmanager.yml, which had this bug first:compose_deploy_config_hashes(ALLOY_CONFIG_SHA),homelab.config-shalabel on thealloyservice,so a content change flips the label and Compose recreates just that service.
The list's own comment invites this ("Extend this list as other services adopt
the same label").
Does this make deploys recreate alloy more often?
Yes, and that is already designed for.
monitoring/compose.yamlpasses--storage.path=/var/lib/alloy/datawith a comment saying why: without it everyrecreate replays old container logs into Loki and earns 400 "too far behind".
Read positions persist across recreates, so the extra recreate costs a seconds-
long ingestion gap and nothing else.
One way this could have been quietly wrong, checked
The
alloyservice had nolabels:key, and thex-hardeninganchor it mergescarries only
security_opt. So there is no merge collision in either direction:the new labels block cannot silently drop the hardening, and the hardening
cannot silently drop the label. Verified, not assumed.
Checks run locally
bash tests/run.shgreendocker compose config --quieton the monitoring project with the renderedsite overlay
unsetwith no env var and the digest with one setansible-playbook --syntax-checkon all three entrypoints,ansible-lint(production profile, 0 failures)
pre-commit run --all-files(gitleaks, OPSEC, storage guard) greenHow this gets proven, and what does NOT prove it
This PR alone cannot be proven by a deploy. It adds a
labels:block to aservice that had none, which is a change to the service definition, so the
first deploy after merging it recreates alloy whether or not the hash mechanism
works. Watching that recreate and calling it proof would be a false positive.
The digest path needs a deploy whose only difference is the bytes of
config.alloy. That is #420, a content-only header comment on that file, whichmust be merged AFTER this one.
Full sequence on the server, re-reading the baseline immediately before each
step rather than trusting an older reading:
docker inspect alloy --format '{{.Created}}'plus thehomelab.config-shalabel.
deploy.yml. Alloy recreates, because the definitionchanged. Proves nothing about the digest.
deploy.ymlagain:changed=0(DEP-4).deploy.yml, no--force-recreate. Alloy must recreateand the digest must differ from step 3. This is the proof.
deploy.ymlagain:changed=0.Anyone reading the deploy log later will see two recreates. Only the second one
means anything.
Negative control, observed free of charge on 2026-08-21: an unrelated deploy of
#418 changed neither
config.alloynor the alloy service definition, reportedchanged=4, and left alloy'sCreatedat2026-08-21T12:25:12.498Z. That isthe "before" behaviour this PR changes.
Not in scope
The monitoring stack has three more single-file bind mounts that are likely
startup-only and not in the list:
loki/loki-config.yml,socket-proxy/haproxy.cfg, andtautulli-exporter/main.py. Each needs its ownthink about what a recreate costs, and socket-proxy especially: it is the
service whose stale inode silently 503s Alloy and stops all Loki ingestion, with
a
restart-socket-consumersunit built around that interaction. Separatesession, not padding on this one.