Skip to content

Give tempo's S3 backend the endpoint its client requires - #204

Merged
stxkxs merged 1 commit into
mainfrom
tempo-s3-endpoint
Aug 10, 2026
Merged

Give tempo's S3 backend the endpoint its client requires#204
stxkxs merged 1 commit into
mainfrom
tempo-s3-endpoint

Conversation

@stxkxs

@stxkxs stxkxs commented Aug 10, 2026

Copy link
Copy Markdown
Member

The tempo ApplicationSet injected a bucket and a region and no endpoint. Tempo reaches S3 through the minio-go client, which validates the endpoint before it looks at anything else and refuses an empty one:

level=error msg="error running Tempo" err="failed to init module services: error initialising
module: optional-store: failed to create store: unexpected error creating core: failed to
create minio client: Endpoint:  does not follow ip address or domain name standards."

Tempo exits non-zero at startup, so the StatefulSet crashloops and the Application sits Progressing forever.

What it cost

On a fresh install this is the single thing holding up catalog convergence. Observed live: 46 Applications reached Healthy and this one did not, and the installer waited its full 30 minutes on it before giving up.

...574s:  46 applications, 3 not Healthy
...636s:  46 applications, 1 not Healthy
...1765s: 46 applications, 1 not Healthy     <- tempo, for the next 19 minutes

loki is not the same story

Its AWS client derives an endpoint from the region, so bucket + region is a complete configuration there. That is why the two blocks look asymmetric and have to stay that way — copying loki's shape to tempo is what produced a config that renders correctly and cannot start.

Why the gate missed it

The render test asserted the bucket and region are injected under the annotation, and that nothing is injected without it. Both were true the whole time. It asked whether the manifest was well-formed, never whether the configuration it produced was complete enough for the process to run.

It now asserts the endpoint, against the rendered value rather than a literal so a region change carries. Against the previous template it fails with the rendered block printed:

                trace:
                  backend: s3
                  s3:
                    bucket: alpha-111111111111-us-west-2-tempo
                    region: us-west-2
FAIL

That failure is the only evidence the assertion catches anything.

The tempo ApplicationSet injected a bucket and a region and no endpoint. Tempo
reaches S3 through the minio-go client, which validates the endpoint before it
looks at anything else and refuses an empty one:

    failed to create minio client: Endpoint:  does not follow ip address or
    domain name standards.

Tempo exits non-zero at startup, so the StatefulSet crashloops and the
Application sits Progressing forever. On a fresh install that is the single
thing holding up catalog convergence: forty-six Applications reached Healthy and
this one did not, and the installer waited its full thirty minutes on it before
giving up.

loki is not the same story, which is why the two blocks look asymmetric and have
to stay that way: its AWS client derives an endpoint from the region, so bucket
plus region is a complete configuration there. Copying loki's shape to tempo is
what produced a config that renders correctly and cannot start.

WHY THE GATE MISSED IT

The render test asserted the bucket and the region are injected under the
annotation, and that nothing is injected without it. Both were true the whole
time. It asked whether the manifest was well-formed, never whether the
configuration it produced was complete enough for the process to run — the same
distinction between a valid artifact and a working one that this catalog keeps
meeting.

It now asserts the endpoint, against the rendered value rather than against a
literal, so a region change carries. It fails on the previous template with the
rendered block printed, which is the only evidence it catches anything.
@github-actions

Copy link
Copy Markdown

CI Results

Check Status
Zero-placeholder gate ✅ success
YAML Lint ✅ success
Dashboard gate (live grafana.com ids + AMG-saveable) ✅ success
Kyverno policy tests (+ verify-images contract) ✅ success
Fork-safety gate (no hardcoded org, blocking) ✅ success
Helm-render (every addon × every env) ✅ success
Policy-admission (Enforce-tier Kyverno vs the fleet) ✅ success
ApplicationSet schema + sync waves ✅ success
Appset render (Karpenter subnet selector) ✅ success
Secret scan (gitleaks) ✅ success
Render + assert + schema + misconfig (all environments) ✅ success

All checks passed.

@stxkxs
stxkxs merged commit 41030d9 into main Aug 10, 2026
23 checks passed
@stxkxs
stxkxs deleted the tempo-s3-endpoint branch August 11, 2026 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant