Give tempo's S3 backend the endpoint its client requires - #204
Merged
Conversation
The tempo ApplicationSet injected a bucket and a region and no endpoint. Tempo
reaches S3 through the minio-go client, which validates the endpoint before it
looks at anything else and refuses an empty one:
failed to create minio client: Endpoint: does not follow ip address or
domain name standards.
Tempo exits non-zero at startup, so the StatefulSet crashloops and the
Application sits Progressing forever. On a fresh install that is the single
thing holding up catalog convergence: forty-six Applications reached Healthy and
this one did not, and the installer waited its full thirty minutes on it before
giving up.
loki is not the same story, which is why the two blocks look asymmetric and have
to stay that way: its AWS client derives an endpoint from the region, so bucket
plus region is a complete configuration there. Copying loki's shape to tempo is
what produced a config that renders correctly and cannot start.
WHY THE GATE MISSED IT
The render test asserted the bucket and the region are injected under the
annotation, and that nothing is injected without it. Both were true the whole
time. It asked whether the manifest was well-formed, never whether the
configuration it produced was complete enough for the process to run — the same
distinction between a valid artifact and a working one that this catalog keeps
meeting.
It now asserts the endpoint, against the rendered value rather than against a
literal, so a region change carries. It fails on the previous template with the
rendered block printed, which is the only evidence it catches anything.
CI Results
All checks passed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The tempo ApplicationSet injected a bucket and a region and no endpoint. Tempo reaches S3 through the minio-go client, which validates the endpoint before it looks at anything else and refuses an empty one:
Tempo exits non-zero at startup, so the StatefulSet crashloops and the Application sits
Progressingforever.What it cost
On a fresh install this is the single thing holding up catalog convergence. Observed live: 46 Applications reached Healthy and this one did not, and the installer waited its full 30 minutes on it before giving up.
loki is not the same story
Its AWS client derives an endpoint from the region, so bucket + region is a complete configuration there. That is why the two blocks look asymmetric and have to stay that way — copying loki's shape to tempo is what produced a config that renders correctly and cannot start.
Why the gate missed it
The render test asserted the bucket and region are injected under the annotation, and that nothing is injected without it. Both were true the whole time. It asked whether the manifest was well-formed, never whether the configuration it produced was complete enough for the process to run.
It now asserts the endpoint, against the rendered value rather than a literal so a region change carries. Against the previous template it fails with the rendered block printed:
That failure is the only evidence the assertion catches anything.