[Kafka] Change Kafka installation - multi namespace support

[shay79il]

Migrating bitnami kafka to Strimzi Kafka operator

Add Strimzi Kafka operator configuration
Update values for Kafka deployment

JIRA

[GiladShapira94]

Looks good, can you edit the admin and the non-admin values files with the values that need to be use.
it does not need to support ingress and user that use NodePort will need to change the ports manually

[Copilot]

Pull request overview

This PR migrates the Kafka deployment from Bitnami Kafka to the Strimzi Kafka Operator, enabling multi-namespace support and modernizing the Kafka infrastructure with KRaft mode (ZooKeeper-less operation).

Key Changes:

• Replaced Bitnami Kafka chart dependency with Strimzi Kafka Operator (version 0.48.0)
• Introduced new Kubernetes custom resources for Kafka deployment including KafkaNodePool, Kafka cluster, RBAC resources, and network policies
• Configured single-node Kafka cluster with KRaft mode for simplified deployment

Reviewed changes

Copilot reviewed 8 out of 10 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
charts/mlrun-ce/values.yaml	Replaced Bitnami Kafka configuration with Strimzi operator values, including storage, resources, listeners, and RBAC settings; removed unrelated minio image config
charts/mlrun-ce/templates/kafka/kafka-cluster.yaml	Added Kafka custom resource definition for Strimzi operator with listener and config management
charts/mlrun-ce/templates/kafka/kafka-nodepool.yaml	Added KafkaNodePool resource for KRaft-mode Kafka cluster management
charts/mlrun-ce/templates/kafka/kafka-rbac.yaml	Created RBAC resources (ServiceAccount, Role, RoleBinding) for cross-namespace Kafka access
charts/mlrun-ce/templates/kafka/kafka-network-policy.yaml	Added NetworkPolicy to control egress traffic to Kafka cluster across namespaces
charts/mlrun-ce/templates/kafka/kafka-bootstrap-alias.yaml	Created service alias for simplified Kafka bootstrap server naming
charts/mlrun-ce/requirements.yaml	Updated chart dependency from bitnami/kafka to strimzi-kafka-operator
charts/mlrun-ce/requirements.lock	Updated lock file with new Strimzi operator dependency and digest
charts/mlrun-ce/Chart.yaml	Bumped chart version from 0.10.0-rc5 to 0.10.0-rc6
.gitignore	Added comprehensive .DS_Store file patterns for macOS

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The NetworkPolicy allows unrestricted egress to all namespaces and pods (lines 56-58), which defeats the purpose of the network policy. This rule effectively allows traffic to anywhere, making the specific Kafka access rules redundant. Consider removing this overly permissive rule or adding a comment explaining why unrestricted egress is necessary for this use case.

Suggested change

	# Allow egress to the internet/other services (optional)
	# Comment out the next section if you want to restrict to Kafka only
	- to:
	- namespaceSelector: {}
	- podSelector: {}

[shay79il]

MLRun CE Install Guide

Prerequisites

• Docker Desktop installed and running (Kubernetes enabled)
• Pick a kube context:

kubectl config get-contexts
kubectl config use-context docker-desktop

1) Install ingress-nginx (for docker-desktop)

If you want ingress URLs to work on Docker Desktop:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.3/deploy/static/provider/cloud/deploy.yaml
kubectl rollout status -n ingress-nginx deployment/ingress-nginx-controller --timeout=2m

2) Add /etc/hosts entries (per namespace)

Add these for each namespace you deploy (example for mlrun + docker-desktop):

sudo /bin/sh -c 'printf "\n# mlrun-hosts:mlrun:docker-desktop BEGIN\n127.0.0.1 jupyter.mlrun.docker-desktop.lab.iguazeng.com\n127.0.0.1 mlrun-ui.mlrun.docker-desktop.lab.iguazeng.com\n127.0.0.1 mlrun-api.mlrun.docker-desktop.lab.iguazeng.com\n127.0.0.1 nuclio.mlrun.docker-desktop.lab.iguazeng.com\n127.0.0.1 s3.mlrun.docker-desktop.lab.iguazeng.com\n127.0.0.1 minio-console.mlrun.docker-desktop.lab.iguazeng.com\n127.0.0.1 tdengine.mlrun.docker-desktop.lab.iguazeng.com\n# mlrun-hosts:mlrun:docker-desktop END\n" >> /etc/hosts'

To remove:

sudo /bin/sh -c "sed -i '' '/# mlrun-hosts:mlrun:docker-desktop BEGIN/,/# mlrun-hosts:mlrun:docker-desktop END/d' /etc/hosts"

3) Single namespace (no controller)

helm upgrade --install mlrun \
  -n mlrun --create-namespace \
  --insecure-skip-tls-verify \
  ce/charts/mlrun-ce \
  -f ce/charts/mlrun-ce/values.yaml

4) Multi namespace (controller + per‑namespace)

4.1) Controller (once)

helm upgrade --install mlrun-ce-controller \
  -n controller --create-namespace \
  --insecure-skip-tls-verify \
  ce/charts/mlrun-ce \
  -f ce/charts/mlrun-ce/admin_installation_values.yaml

4.2) Namespace install (repeat per namespace)

Replace <ns> and <cluster>.

helm upgrade --install <ns> \
  -n <ns> --create-namespace \
  --insecure-skip-tls-verify \
  ce/charts/mlrun-ce \
  -f ce/charts/mlrun-ce/values.yaml \
  --set nuclio.crd.create=false \
  --set mpi-operator.enabled=false \
  --set mpi-operator.crd.create=false \
  --set mpi-operator.rbac.clusterResources.create=false \
  --set spark-operator.enabled=false \
  --set kube-prometheus-stack.enabled=false \
  --set pipelines.crd.enabled=false \
  --set pipelines.priority_class.enabled=false \
  --set pipelines.service.type=ClusterIP \
  --set mlrun.api.service.type=ClusterIP \
  --set mlrun.ui.service.type=ClusterIP \
  --set jupyterNotebook.service.type=ClusterIP \
  --set minio.service.type=ClusterIP \
  --set minio.consoleService.type=ClusterIP \
  --set nuclio.dashboard.nodePort= \
  --set global.nuclio.dashboard.nodePort= \
  --set mlrun.api.ingress.enabled=true \
  --set mlrun.ui.ingress.enabled=true \
  --set jupyterNotebook.ingress.enabled=true \
  --set minio.ingress.enabled=true \
  --set minio.consoleIngress.enabled=true \
  --set nuclio.dashboard.ingress.enabled=true \
  --set tdengine.ingress.enabled=true \
  --set-string=mlrun.api.ingress.hosts[0].host=mlrun-api.<ns>.<cluster>.lab.iguazeng.com \
  --set-string=mlrun.ui.ingress.hosts[0].host=mlrun-ui.<ns>.<cluster>.lab.iguazeng.com \
  --set-string=jupyterNotebook.ingress.hosts[0].host=jupyter.<ns>.<cluster>.lab.iguazeng.com \
  --set-string=nuclio.dashboard.ingress.hosts[0]=nuclio.<ns>.<cluster>.lab.iguazeng.com \
  --set-string=minio.ingress.hosts[0]=s3.<ns>.<cluster>.lab.iguazeng.com \
  --set-string=minio.consoleIngress.hosts[0]=minio-console.<ns>.<cluster>.lab.iguazeng.com \
  --set-string=tdengine.ingress.hosts[0]=tdengine.<ns>.<cluster>.lab.iguazeng.com \
  --set-string=mlrun.api.ingress.hosts[0].paths[0].path=/ \
  --set-string=mlrun.api.ingress.hosts[0].paths[0].pathType=Prefix \
  --set-string=mlrun.ui.ingress.hosts[0].paths[0].path=/ \
  --set-string=mlrun.ui.ingress.hosts[0].paths[0].pathType=Prefix \
  --set mlrun.api.ingress.ingressClassName=nginx \
  --set mlrun.ui.ingress.ingressClassName=nginx \
  --set jupyterNotebook.ingress.ingressClassName=nginx \
  --set minio.ingress.ingressClassName=nginx \
  --set minio.consoleIngress.ingressClassName=nginx \
  --set nuclio.dashboard.ingress.ingressClassName=nginx \
  --set tdengine.ingress.ingressClassName=nginx

---

Verify

kubectl get pods -n <ns>
kubectl get ing -n <ns>