[pip] PIP-455: Support Namespace Bundle Lookup and Topic Preloading

[zhaizhibo]

Motivation

Modifications

Start a PIP: Support Namespace Bundle Lookup and Topic Preloading

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[github-actions]

@zhaizhibo Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

[lhotari]

@zhaizhibo Please ensure that you have subscribed to the dev mailing list (instructions). I released your email from the moderation queue.

[zhaizhibo]

@zhaizhibo Please ensure that you have subscribed to the dev mailing list (instructions). I released your email from the moderation queue.

@lhotari Thanks, I have subscribed the dev mail.

[codelipenghui]

We don't need to support V1 API since we are working on removing the V1 endpoints from the repo.

[codelipenghui]

Could you please also define the response format of this API, it will be a map which mapping the service URL for each bundle?

[shibd]

One concern I have with the namespace-level lookup / preloading flow is bundle placement fairness before traffic actually arrives.

For pulsar-admin namespaces lookup my-tenant/my-ns, if the implementation just iterates bundles and triggers lookup/ownership acquisition one by one, I’m not sure this will produce an even bundle distribution across brokers in the cold-start case. At preload time, these bundles may have little or no traffic, so the load manager may not have enough signal to place them in a way that reflects the future workload. In the worst case, a single broker could end up owning many preloaded bundles, and then when traffic arrives later we would need another rebalance cycle, which brings us back to the same lazy-loading / cold-start problem that this PIP is trying to avoid.

Could the PIP clarify this part of the design a bit more?

How should namespace-level preload avoid concentrating too many bundles on one broker?
Is the expected behavior to reuse the normal bundle assignment logic as-is, or should there be a more explicit “spread-first” strategy for this admin-triggered warm-up path?

Should the API return the final bundle -> broker mapping so operators can verify that the warm-up actually achieved a balanced placement?