ci(monitoring): parse config.alloy before it can merge - #426
Merged
Conversation
No gate ever asked Alloy whether config.alloy is valid. tests/run.sh greps the drop expressions out of the file as text, so a stray brace or a forward_to pointing at a component that does not exist passed every check, merged, and failed at container start. Since #419 a config.alloy edit recreates the alloy container, so that failure stops log ingestion, and it presents as silence: every metric stays healthy and only LokiLogIngestionStalled notices, twenty minutes later. `alloy validate` provisions the component graph, so it catches the dangling reference as well as the syntax error. The image tag is read out of monitoring/compose.yaml so a Renovate bump carries the gate along rather than leaving a second pin to maintain by hand. Task CV-1.
CV-4 wants the new gate watched failing, not assumed. This points the docker source at loki.process.typo_containers, which does not exist.
…eference" This reverts commit 889f9ff.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an
alloy validatestep to themonitoring-configjob, so CI parsesmonitoring/alloy/config.alloyinstead of only grepping strings out of it.Task CV-1 of
homelab-ci-config-validation. CV-2 (Loki) and CV-3 (Gatus)are not in this PR.
Why
tests/run.shpulls the drop expressions out ofconfig.alloywith awk andmatches them against fixture log lines. Nothing ever asked Alloy whether the
file it is grepping is a valid config, so a misplaced brace or a
forward_tonaming a component that does not exist passed every gate and only failed when
the container started.
Since #419 a
config.alloychange recreates the alloy container, so a badedit stops log ingestion outright, and that presents as silence: every
metric stays healthy, the anomaly panel goes quiet, and quiet reads as fine.
LokiLogIngestionStalledatfor: 20mis the only thing that would catch it.validaterather thanfmt: fmt only parses, validate provisions thecomponent graph, so it also catches the dangling component reference.
The tag is read out of
monitoring/compose.yaml, the same way the caddy jobdoes it, so a Renovate bump moves the gate with the service instead of
leaving a fourth hand-maintained pin. That pattern is only honest because
alloy is a pulled image, so the thing CI validates with is the thing that
runs.
Proof the gate catches something (CV-4)
Watched failing in CI, not assumed. The middle commit on this branch pointed
the docker source at
loki.process.typo_containers.receiver, and the revertput it back:
monitoring-configtestsThe red run failed with:
Note the
testsjob stayed green through the broken commit. That is the gapthis PR closes, shown rather than argued: the text-level drop-expression check
does not notice a config Alloy will refuse to start on.
Locally, against the same pinned
grafana/alloy:v1.18.1, a stray}failsthe same way (
expected identifier, got }at 263:1). That casefmtwouldalso have caught; the dangling reference is the one it would not.
Not covered
The head of this branch is the real config, so the merged state is the green
one.
monitoring/loki/loki-config.ymlandmonitoring/gatus/config.yaml.j2still have no gate, tracked as CV-2 and CV-3.