Skip to content

ci(monitoring): parse config.alloy before it can merge - #426

Merged
PBernaerts merged 3 commits into
mainfrom
ci/alloy-validate
Aug 21, 2026
Merged

ci(monitoring): parse config.alloy before it can merge#426
PBernaerts merged 3 commits into
mainfrom
ci/alloy-validate

Conversation

@PBernaerts

@PBernaerts PBernaerts commented Aug 21, 2026

Copy link
Copy Markdown
Owner

What

Adds an alloy validate step to the monitoring-config job, so CI parses
monitoring/alloy/config.alloy instead of only grepping strings out of it.

Task CV-1 of homelab-ci-config-validation. CV-2 (Loki) and CV-3 (Gatus)
are not in this PR.

Why

tests/run.sh pulls the drop expressions out of config.alloy with awk and
matches them against fixture log lines. Nothing ever asked Alloy whether the
file it is grepping is a valid config, so a misplaced brace or a forward_to
naming a component that does not exist passed every gate and only failed when
the container started.

Since #419 a config.alloy change recreates the alloy container, so a bad
edit stops log ingestion outright, and that presents as silence: every
metric stays healthy, the anomaly panel goes quiet, and quiet reads as fine.
LokiLogIngestionStalled at for: 20m is the only thing that would catch it.

validate rather than fmt: fmt only parses, validate provisions the
component graph, so it also catches the dangling component reference.

The tag is read out of monitoring/compose.yaml, the same way the caddy job
does it, so a Renovate bump moves the gate with the service instead of
leaving a fourth hand-maintained pin. That pattern is only honest because
alloy is a pulled image, so the thing CI validates with is the thing that
runs.

Proof the gate catches something (CV-4)

Watched failing in CI, not assumed. The middle commit on this branch pointed
the docker source at loki.process.typo_containers.receiver, and the revert
put it back:

Run Head monitoring-config tests
32492528481 gate only, config untouched success success
32492555900 deliberate dangling reference failure success
32492759927 revert, back to the real config success success

The red run failed with:

Error: /c.alloy:43:17: component "loki.process.typo_containers.receiver" does not exist or is out of scope
Error: validation failed
##[error]Process completed with exit code 1.

Note the tests job stayed green through the broken commit. That is the gap
this PR closes, shown rather than argued: the text-level drop-expression check
does not notice a config Alloy will refuse to start on.

Locally, against the same pinned grafana/alloy:v1.18.1, a stray } fails
the same way (expected identifier, got } at 263:1). That case fmt would
also have caught; the dangling reference is the one it would not.

Not covered

The head of this branch is the real config, so the merged state is the green
one. monitoring/loki/loki-config.yml and monitoring/gatus/config.yaml.j2
still have no gate, tracked as CV-2 and CV-3.

No gate ever asked Alloy whether config.alloy is valid. tests/run.sh greps
the drop expressions out of the file as text, so a stray brace or a
forward_to pointing at a component that does not exist passed every check,
merged, and failed at container start.

Since #419 a config.alloy edit recreates the alloy container, so that
failure stops log ingestion, and it presents as silence: every metric stays
healthy and only LokiLogIngestionStalled notices, twenty minutes later.

`alloy validate` provisions the component graph, so it catches the dangling
reference as well as the syntax error. The image tag is read out of
monitoring/compose.yaml so a Renovate bump carries the gate along rather
than leaving a second pin to maintain by hand.

Task CV-1.
CV-4 wants the new gate watched failing, not assumed. This points the
docker source at loki.process.typo_containers, which does not exist.
@PBernaerts
PBernaerts merged commit a8135b6 into main Aug 21, 2026
9 checks passed
@PBernaerts
PBernaerts deleted the ci/alloy-validate branch August 21, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant