Summary
When cert-manager and/or OpenDataHub CRDs are installed on the cluster but the operator's ServiceAccount lacks RBAC to list/watch their resources, the SharedTrust and MLflow operand controllers register cache watchers that fail indefinitely. Because controller-runtime blocks all controllers until every cache is synced, this prevents the entire operator from reconciling — including unrelated controllers like clientregistration, agentruntime, and the authbridge webhook.
Symptoms
clientregistration controller starts EventSources but never starts workers — no Keycloak client registration happens, no kagenti-keycloak-client-credentials-* secrets are created- Pods stuck in
ContainerCreating waiting for credential secrets that never appear
- Repeated
cache.UnhandledError / Failed to watch log entries for:
*v1.Certificate (cert-manager): certificates.cert-manager.io is forbidden*mlflow.DataScienceCluster (OpenDataHub): datascienceclusters.datasciencecluster.opendatahub.io is forbidden
Root cause
Both controllers use CRDExists() to detect whether their CRDs are installed, and if so, register watchers:
c883d07 — SharedTrust controller watches cert-manager.io/Certificate8f9cfa2 — MLflow operand controller watches datasciencecluster.opendatahub.io/DataScienceCluster
The CRDs exist on the cluster (cert-manager and OpenDataHub are installed), so the controllers register. But the Helm chart's ClusterRole does not include the necessary RBAC rules. The cache informers fail with forbidden, and controller-runtime waits indefinitely for cache sync before starting any controller's workers.
Environment
- ROSA HCP (OpenShift 4.x)
- cert-manager installed
- OpenDataHub installed (RHOAI)
- Operator installed via Helm chart
Affected commits
The regression appeared between 2073d03 (PR #382, working) and b93c5cf (PR #408, broken). The operator binary, webhook, and client registration code are unchanged — only the manager startup is blocked.
Suggested fix
Either:
- Add the missing RBAC rules to the Helm chart's ClusterRole for cert-manager Certificates and DataScienceClusters
- Make the controllers resilient — use a separate non-blocking cache or start these controllers in a goroutine that doesn't block the manager's
WaitForCacheSync
- Check RBAC before registering — extend
CRDExists() to also verify the SA has list/watch permissions via a SubjectAccessReview before registering the watcher
Option 2 or 3 is preferred — optional controllers should not take down the entire operator when their RBAC is missing.
Workaround
Grant the missing permissions manually:
# cert-manager
oc create clusterrole kagenti-cert-manager-reader --verb=get,list,watch --resource=certificates.cert-manager.io
oc create clusterrolebinding kagenti-cert-manager-reader --clusterrole=kagenti-cert-manager-reader --serviceaccount=kagenti-system:controller-manager
# OpenDataHub
oc create clusterrole kagenti-datasciencecluster-reader --verb=get,list,watch --resource=datascienceclusters.datasciencecluster.opendatahub.io
oc create clusterrolebinding kagenti-datasciencecluster-reader --clusterrole=kagenti-datasciencecluster-reader --serviceaccount=kagenti-system:controller-manager
Summary
When cert-manager and/or OpenDataHub CRDs are installed on the cluster but the operator's ServiceAccount lacks RBAC to list/watch their resources, the SharedTrust and MLflow operand controllers register cache watchers that fail indefinitely. Because controller-runtime blocks all controllers until every cache is synced, this prevents the entire operator from reconciling — including unrelated controllers like
clientregistration,agentruntime, and the authbridge webhook.Symptoms
clientregistrationcontroller starts EventSources but never starts workers — no Keycloak client registration happens, nokagenti-keycloak-client-credentials-*secrets are createdContainerCreatingwaiting for credential secrets that never appearcache.UnhandledError/Failed to watchlog entries for:*v1.Certificate(cert-manager):certificates.cert-manager.io is forbidden*mlflow.DataScienceCluster(OpenDataHub):datascienceclusters.datasciencecluster.opendatahub.io is forbiddenRoot cause
Both controllers use
CRDExists()to detect whether their CRDs are installed, and if so, register watchers:c883d07— SharedTrust controller watchescert-manager.io/Certificate8f9cfa2— MLflow operand controller watchesdatasciencecluster.opendatahub.io/DataScienceClusterThe CRDs exist on the cluster (cert-manager and OpenDataHub are installed), so the controllers register. But the Helm chart's ClusterRole does not include the necessary RBAC rules. The cache informers fail with
forbidden, and controller-runtime waits indefinitely for cache sync before starting any controller's workers.Environment
Affected commits
The regression appeared between
2073d03(PR #382, working) andb93c5cf(PR #408, broken). The operator binary, webhook, and client registration code are unchanged — only the manager startup is blocked.Suggested fix
Either:
WaitForCacheSyncCRDExists()to also verify the SA has list/watch permissions via a SubjectAccessReview before registering the watcherOption 2 or 3 is preferred — optional controllers should not take down the entire operator when their RBAC is missing.
Workaround
Grant the missing permissions manually: