[SS-74] Don't try creating a Kafka progress topic if it already exists.#35252
Open
ublubu wants to merge 2 commits intoMaterializeInc:mainfrom
Open
[SS-74] Don't try creating a Kafka progress topic if it already exists.#35252ublubu wants to merge 2 commits intoMaterializeInc:mainfrom
ublubu wants to merge 2 commits intoMaterializeInc:mainfrom
Conversation
|
Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone. PR title guidelines
Pre-merge checklist
|
ublubu
commented
Feb 27, 2026
Comment on lines
+209
to
+218
| // First, check whether the topic already exists. | ||
| let metadata = client | ||
| .inner() | ||
| // N.B. It is extremely important not to ask specifically | ||
| // about the topic here, even though the API supports it! | ||
| // Asking about the topic will create it automatically... | ||
| // with the wrong number of partitions. Yes, this is | ||
| // unbelievably horrible. | ||
| .fetch_metadata(None, Some(Duration::from_secs(10)))?; | ||
| let already_exists = metadata.topics().iter().any(|t| t.name() == new_topic.name) || { |
Contributor
Author
There was a problem hiding this comment.
This is the new part. The rest is just indented (inside the || { ... })
Contributor
Author
There was a problem hiding this comment.
Copypaste from https://github.com/ublubu/materialize/blob/588547c24336fc41d5039afce1f5e414a28356d5/src/kafka-util/src/admin.rs#L251-L263
let metadata = client
.inner()
// N.B. It is extremely important not to ask specifically
// about the topic here, even though the API supports it!
// Asking about the topic will create it automatically...
// with the wrong number of partitions. Yes, this is
// unbelievably horrible.
.fetch_metadata(None, Some(Duration::from_secs(10)))?;
let topic = metadata
.topics()
.iter()
.find(|t| t.name() == new_topic.name)
.ok_or_else(|| anyhow!("unable to fetch topic metadata after creation"))?;5f5f328 to
fe8bce7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
It is an accepted pattern to create the topics needed for a new sink (e.g. as superuser) before actually creating the sink (as a less-privileged user).
Before this fix, Materialize always tries creating the progress topic, even if it already exists, leading to an authorization failure.
After this fix, Materialize sees that the progress topic exists, so it continues without attempting the operation that requires the "Create" ACL.