Skip to content

feat: follow loki and tempo to their new home, and notice next time - #31

Merged
stxkxs merged 1 commit into
mainfrom
feat/chart-provenance
Aug 8, 2026
Merged

feat: follow loki and tempo to their new home, and notice next time#31
stxkxs merged 1 commit into
mainfrom
feat/chart-provenance

Conversation

@stxkxs

@stxkxs stxkxs commented Aug 8, 2026

Copy link
Copy Markdown
Member

Mirrors the catalog change in eks-gitops#190.

Two moves, one announced

tempo is deprecated: true and its README names grafana-community as the destination. The fork carries the full chart history, so it's the same chart with the same single-binary topology, renumbered at the fork: 1.24.42.2.3, app 2.9.02.10.7.

tempo-distributed is not the fallback the old comment in this repo suggested — it was deprecated in the same move. That comment is gone.

loki is the quiet one. The OSS chart moved to the same place, forked at 6.55.0; what stayed behind is the Grafana Enterprise Logs chart, and it sets no deprecation flag. The pin resolved, the chart installed, the render gate passed, Renovate kept offering 7.x patches. 7.2.018.7.5.

The breakage rendering could not see

With persistence off, the new loki chart mounts its own emptyDir at /var/loki — precisely the gap this repo's extraVolumes workaround existed to fill. Keeping both gives the loki container two volumeMounts on one path:

StatefulSet/loki container loki: 2 volumes mounted at /var/loki (storage, data)

Kubernetes rejects that outright; the pod never starts. helm template renders it and exits 0, and render-check.sh only ever inspected the exit status — so this would have shipped green.

The workaround is removed, and the render gate now pipes each slice through scripts/check-rendered-mounts.py instead of discarding the output. Its self-test covers the case in both directions:

  caught    two volumes on one path
  caught    the duplicate is in an initContainer
  allowed   same path in two DIFFERENT containers is legal
  allowed   a container with no volumeMounts at all
  caught    a CronJob's nested pod spec is reached

It also refuses a stream that parses to zero manifests, so it can't pass by examining nothing. And it needs a constructor for YAML 1.1's = value tag — prometheus-operator-crds ships a CRD containing one and SafeLoader raises on it; without that the choice was failing the gate on an unrelated chart or swallowing the document whole.

The check

scripts/check-chart-deprecation.py, the kx side of the check in eks-gitops. Split by what is and isn't a function of this commit:

  • offline, in ci.yml, blocking — every pinned chart has a provenance record, every record names a chart still pinned.
  • --live, weekly in chart-provenance.yml — fetches all 32 pins, fails on deprecated: true or a description that no longer matches.

Recording the description is what catches the loki class, where nothing else moved. Recording the repository also closes a gap in mirror-check, which compares chart→version and never looks at where a chart came from — the two repos could pull one version from two different repositories and it would pass. That matters most for the seven kx-only slices mirror-check doesn't cover at all; nothing upstream watches those.

Separate files from the eks-gitops copy rather than a shared module, because the two repos state their pins in different languages (install.sh vs ApplicationSet YAML), and a repo that can't gate itself without cloning another one isn't really gated.

Two things the pin move surfaced

mirror-check.py sync mangled its own manifest. It wrote JSON with json.dump defaults, so every em-dash in the divergence reasons came back as a escape — moving a one-line ref rewrote prose across the whole file. Fixed with ensure_ascii=False.

An undeclared divergence. Moving the pin to the merged catalog commit surfaced that eks-gitops began pinning the eks-agent-platform operator chart after kx's last sync. Both sides run the operator; only eks-gitops pins a version, because kx installs the chart from the sibling checkout and kind-loads an image built from that tree. Declared in stack/upstream.json rather than papered over — an omission and a decision shouldn't look the same. Pre-existing, not caused here.

Verification

gate result
render-check (33 slices) all OK, exit 0
renovate coverage 32 pins, all matched
chart provenance (offline) 32 pins, all recorded
chart provenance self-test 5/5 breaks rejected
mount check self-test 5/5 cases correct
mirror-check 32 match @614560a, 12 declared divergences
yamllint / shellcheck clean

Config carry-over was checked by rendering old and new with the same values: receivers, storage backend, schema v13, retention and replication all identical. tempo's whole manifest diff is the image bump, one newly-explicit default, and upstream dropping the OpenCensus receiver port nothing here enabled.

Both observability charts moved out of grafana/helm-charts, matching the catalog
change in eks-gitops. One announced it; the other did not.

─── The two moves ───

tempo is flagged `deprecated: true` and its README names grafana-community as
the destination. The fork carries the full history, so it is the same chart with
the same single-binary topology, renumbered at the fork: 1.24.4 -> 2.2.3, app
2.9.0 -> 2.10.7. tempo-distributed is not the fallback the old comment in this
repo suggested — it was deprecated in the same move, and that comment is gone.

loki is the quieter one. The OSS chart moved to grafana-community, forked at
6.55.0; what stayed at grafana/helm-charts is the Grafana Enterprise Logs chart.
It sets no deprecation flag. The pin resolved, the chart installed, the render
gate passed, and Renovate kept offering 7.x patches for a chart that had changed
product. 7.2.0 -> 18.7.5, the OSS lineage.

─── The one real breakage, which rendering could not see ───

With persistence off, the new loki chart mounts its own emptyDir at /var/loki.
That is precisely the gap this repo's `extraVolumes` workaround existed to
fill — loki mkdir'ing storage and ruler dirs onto a read-only rootfs and
crashing. Keeping both gives the loki container two volumeMounts on one
mountPath, which Kubernetes rejects outright: the pod never starts.

`helm template` renders it and exits 0. render-check.sh only ever inspected the
exit status, so it would have shipped green.

So the workaround is removed, and the render gate now pipes each slice through
scripts/check-rendered-mounts.py instead of discarding it. That check has a
self-test covering the case that matters in both directions — two volumes on one
path in one container fails, the same path across two different containers is
legal — plus initContainers, nested CronJob pod specs, and a null volumeMounts.
It also refuses a stream that parses to zero manifests, so it cannot pass by
examining nothing.

It needs a constructor for YAML 1.1's `=` value tag: prometheus-operator-crds
ships a CRD containing one, and SafeLoader raises on it. Without that the choice
was failing the gate on an unrelated chart or swallowing the document whole.

─── The check ───

scripts/check-chart-deprecation.py, the kx side of the same check now in
eks-gitops. Split by what is and is not a function of this commit:

  offline, in ci.yml, BLOCKING — every pinned chart has a provenance record and
      every record names a chart still pinned.

  --live, weekly in chart-provenance.yml — fetches all 32 pins and fails on a
      `deprecated: true` or a description that no longer matches its record.

Recording the description is what catches the loki class, where nothing else
moved. Recording the repository also closes a gap in mirror-check, which
compares chart to version and never looks at where a chart came from: the two
repos could pull one version from two different repositories and it would pass.
Changing where a chart comes from now has to be re-recorded deliberately.

That matters most for the seven kx-only slices mirror-check does not cover at
all. Nothing upstream watches those.

─── Two things the pin move surfaced ───

`mirror-check.py sync` wrote its manifest with json.dump defaults, so every
em-dash in the divergence reasons came back as a — escape and moving a
one-line ref rewrote prose across the file. Fixed with ensure_ascii=False.

Moving the pin to the merged catalog commit also surfaced a divergence that had
been invisible: eks-gitops began pinning the eks-agent-platform operator chart
after kx's last sync. Both sides run the operator; only eks-gitops pins a
version, because kx installs the chart from the sibling checkout and kind-loads
an image built from that tree. Declared in stack/upstream.json rather than
papered over — an omission and a decision should not look the same.
@stxkxs
stxkxs merged commit 6ba4148 into main Aug 8, 2026
9 checks passed
@stxkxs
stxkxs deleted the feat/chart-provenance branch August 8, 2026 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant