feat: add TrafficExtension for multi-tenancy by NohaIhab · Pull Request #502 · canonical/mlflow-operator

NohaIhab · 2026-09-25T13:19:29Z

Closes #457

Add an Istio TrafficExtension to derive MLflow user identity, this is for the use case when reaching MLflow from inside the user/profile namespace.

* [WIP] feat: add custom authentication for multi-tenancy * feat: add custom authentication for multi-tenancy * test: adapt integration tests to work-in-progress authentication setup * style: fix typos * test: restore accidentally removed import * style: sort imports or skip linting * fix: switch from peer relation to juju secret * test: update unit tests * fix: reset unit status to active when bucket exists * fix: reset unit status to active after bucket created * ci: print debugging information * Revert "ci: print debugging information" This reverts commit b12e607. * fix: provision auth database via data platform libs * fix: use separate endpoints for the two relational databases * refactor: set the auth database URI via env var as well * style: fix line breaks * refactor: make relation status messages precise * test: fix typos * test: fix typos * test: fix typos * fix: reuse same database (same URI) for both backend store and auth database * fix: reuse same database (same URI) for both backend store and auth database * fix: reuse same database (same URI) for both backend store and auth database * fix: use correct parameter name in jinja template * fix: change logging level * ci: allow for debugging * fix: have proxy mode on by default * fix: restore proxy mode off by default * fix: restore the real problematic situation * fix: debug with a single uvicorn worker * fix: exclude jobs unsuitable to tracking server * fix: correct typo * fix: switch from mysql to postgresql for auth tables not to undergo usptream bugs * style: do not break line * test: test minor upgrades instead of major ones * test: propagate header also in ingress-reachability tests * style: break line * test: fix test file name * test: fix typos * test: add integration tests for user identity and tenant * test: fix typo * test: rename minor-upgrade file * test: fix user role fetching * test: add test case for no grants to newly seen identity * style: fix spelling of variable name * fix: switch from PostgreSQL to MySQL in latest/edge release bundle * feat: stop the tracking server when the backend store is removed * test: increase coverage * style: fix comment grammar * style: fix comment grammar (again)

* feat: allow user aliasing * test: cover new charm logic * style: add newlines * test: add integration tests for identity aliasing * test: fix typo * refactor: make status messages shorter and redirect to logs * refactor: restart the workload on config changes * style: remove newlines * style: make code coherent * style: fix spelling

Add Istio authorization policies for MLflow 3 multi-tenancy and restrict workload access to the following Identities: * the Waypoint in the platform namespace * the Ingress gateway(s) * the Unit related on the `metrics-endpoint` relation (only for the metrics port)

* feat: add mlflow_client provider logic * chore: update dependencies * chore: switch from charm library to Python package for istio beacon * Revert "chore: switch from charm library to Python package for istio beacon" This reverts commit 605f32d. * chore: fix charm library manually * chore: bump chisme * style: make code maintainable * style: make code maintainable * style: make code maintainable * style: make code maintainable * style: fix conflicts among linters * style: fix noqa syntax * style: fix noqa syntax * chore: switch from service-mesh charm lib to dpcharmlibs python package * chore: fix dependencies * test: adjust provider according to requirer * test: cover granting via data-integrator * style: break lines properly * style: break lines properly * test: update missing test file too * test: fix typo * test: fix typo * test: fix typos * refactor: make grant-reconcile logic clearer * test: exclude URL schema for MinIO * refactor: make grant-reconcile script clearer * style: fix comma * test: fix import * test: fix import * style: fix line break * test: fix typos * docs: improve mlflow-client reconciliation docstring * docs: improve mlflow-client reconciliation docstring * test: fix typos * refactor: make grant-reconcile script clearer * refactor: make grant-reconcile script clearer * refactor: make grant-reconcile script clearer * docs: fix docstring typo * refactor: make grant-reconcile script clearer * text: deploy data-integrator from the requirer's pull request * chore: sort our dependencies after rebasing with conflicts * chore: sort our imports after rebasing with conflicts * chore: sort our imports after rebasing with conflicts * style: fix line break * test: refactor test names * test: fix awaiting * test: fix typos * test: assert native grants * style: add newline * test: fix typos * test: reintroduce acidentally removed line * fix: set backend store URI explicitly in workspace store utility * test: fix search endpoint path * test: refactor constant names * test: test relation removal only at the very end * test: add missing query parameter * test: do not add relation that is already in place * test: retry for live grant changes to take effect * test: debug * test: fix role assertion * test: set target workspaces in requests * test: fix role assertion * test: fix typo * docs: make comment less verbose * fix: ensure old permissions for existing roles are removed * test: cover role update in same workspace * test: poll instead of waiting for port-forwarding * test: cast prot to int * test: switch to tenacy for port-forwarding retries * test: add missing import * docs: add paragraph on why script necessary

MattiaSarti

Excellent work!

deusebio

The business logic looks fine and neat! I only have a comment on the tests, providing a suggestion of a pattern that we have used in Spark when spawning new pods and executing things on them (which is up to you if you like the pattern and you want to adopt it), but at the very least I would try avoiding a bit the repetition between test_charm_ambient and test_charm_s3 and factor out common methods. It is not super-critical, but I feel it would reduce the number of code lines to be reviews and changes in various PRs

Apart from that , all good!

deusebio · 2026-09-29T07:21:44Z

        return "".join(choices(ascii_lowercase, k=length))

+    @staticmethod
+    def _run_script_from_pod(namespace: str, script: str) -> str:


suggestion I'm thinking that maybe one could create a fixture to span and teardown the pod, and the use the pod to make curl requests as one pleases during test.

here is an example and how it is used here

good idea, updated in ff3a36c, I extracted this one to helpers.py and we can continue by moving things there when needed

deusebio · 2026-09-29T07:24:11Z

        experiment_name = f"{TEST_EXPERIMENT_NAME}-{self.generate_random_string(6)}"
-        logs_result = None
+        tracking_uri = f"http://{CHARM_NAME}.{ops_test.model_name}.svc.cluster.local:{mlflow_port}"
+        curl_script = (


suggestion I would really try to break this one up a bit, and get a "exec" => "output" with single API calls. It is not a biggie but this may connect with the previous point to create a Pod, allow to exec commands and retrieve their outputs more programatically

same as #502 (comment)

deusebio · 2026-09-29T07:25:25Z

        return "".join(choices(ascii_lowercase, k=length))

+    @staticmethod
+    def _run_script_from_pod(namespace: str, script: str) -> str:


todo this seems a bit of repetition, it is probably sane to abstract this in either helpers.py

as discussed in the call, we'll be deprecating other integration test files in an upcoming PR

deusebio · 2026-09-29T07:26:41Z

        )

+    @pytest.mark.abort_on_fail
+    async def test_configure_profile_identity_alias(


todo also this seems a bit of repetition. Could this be abstracted away in a single place?

as discussed in the call, we'll be deprecating other integration test files in an upcoming PR

deusebio

Great stuff @NohaIhab ! Thank you for adding the fixture for the curl pod. I really like it a lot more!

deusebio · 2026-09-29T10:38:36Z

+        tracking_uri = f"http://{CHARM_NAME}.{ops_test.model_name}.svc.cluster.local:{mlflow_port}"
+        workspace_header = f"{UPSTREAM_WORKSPACE_HEADER_NAME}: {WORKSPACE_WITH_ADMIN_ACCESS_FINAL}"
+
+        curl_pod.curl(


praise I really really like this a lot more! Well done!

MattiaSarti and others added 4 commits September 10, 2026 17:35

feat: remove NodePort config (#486)

3a47667

NohaIhab added this pull request to stack #503 September 25, 2026 13:19

NohaIhab force-pushed the KF-8906/feat/add-traffic-extenstion-multitenancy branch from a032e5c to d2f0f95 Compare September 25, 2026 13:22

Base automatically changed from KF-9002/feat/add-mlflow_client-provider-logic to feat/add-multi-tenancy September 28, 2026 05:40

feat: add TrafficExtension for multi-tenancy

634d424

MattiaSarti force-pushed the KF-8906/feat/add-traffic-extenstion-multitenancy branch from d2f0f95 to 634d424 Compare September 28, 2026 05:40

Merge feat/add-multi-tenancy into traffic extension

62db39a

NohaIhab marked this pull request as ready for review September 28, 2026 07:18

NohaIhab requested review from MattiaSarti and deusebio September 28, 2026 07:22

MattiaSarti reviewed Sep 28, 2026

View reviewed changes

Comment thread src/service_mesh/traffic-extension.yaml.j2

Comment thread tests/unit/test_operator.py

Comment thread tests/integration/test_charm_ambient.py Outdated

Comment thread tests/integration/test_charm_ambient.py

NohaIhab added 2 commits September 28, 2026 12:52

review: add unit test for relation broken

72db660

review: use tenacity for wait

9a55701

MattiaSarti approved these changes Sep 28, 2026

View reviewed changes

Comment thread src/service_mesh/traffic-extension.yaml.j2

deusebio reviewed Sep 29, 2026

View reviewed changes

NohaIhab added 2 commits September 29, 2026 12:11

review: run curl requests from a reusable CurlPod helper

ff3a36c

test: double postgresql wait timeout in integration tests

3c337c2

NohaIhab requested a review from deusebio September 29, 2026 09:19

deusebio approved these changes Sep 29, 2026

View reviewed changes

NohaIhab merged commit 76e214d into feat/add-multi-tenancy Sep 29, 2026
15 checks passed

NohaIhab deleted the KF-8906/feat/add-traffic-extenstion-multitenancy branch September 29, 2026 11:22

NohaIhab mentioned this pull request Sep 29, 2026

Custom TrafficExtension for Multi-Tenancy #457

Closed

Conversation

NohaIhab commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MattiaSarti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

deusebio left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deusebio left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

NohaIhab commented Sep 25, 2026 •

edited

Loading