feat: OTel collector bootstrap as operator startup Runnable

[Bobbins228]

Summary

Adds an OtelBootstrapRunnable (internal/bootstrap/otel.go) that runs once at operator startup to handle OTel collector infrastructure setup. This replaces the Helm-based otel-ingress-ca-job.yaml Job and the kagenti.otel.collectorConfig Helm helper with operator-managed equivalents that can dynamically discover available components.

Step 1 — Ingress CA trust (replaces otel-ingress-ca-job.yaml):

• Detects OpenShift via discovery API; skips on non-OCP clusters.
• Reads the ingress CA from openshift-config-managed/default-ingress-cert and root CA from openshift-config/kube-root-ca.crt.
• Projects the combined bundle into kagenti-system/otel-ingress-ca.
• Idempotent — skips if already present and unchanged.

Step 2 — OTel collector ConfigMap assembly (replaces kagenti.otel.collectorConfig):

• Discovers the MLflow CRD; if missing, logs a warning with restart guidance and continues with remaining presets.
• When the MLflow CRD is present, discovers the MLflow CR and derives the in-cluster traces endpoint and workspace namespace from status.address.url (the CRD is cluster-scoped so cr.Namespace is always empty).
• If the MLflow service is not yet ready, retries with exponential backoff (up to 5 minutes).
• Discovers Phoenix service availability.
• Assembles the collector ConfigMap by merging component-specific presets (ported from the Helm values.yaml).
• Rollout-restarts the OTel collector Deployment only when the config hash changes.

Changes

• New: internal/bootstrap/otel.go — OtelBootstrapRunnable implementing manager.Runnable
• New: internal/bootstrap/presets.go — OTel collector YAML presets as Go constants (ported from kagenti-deps Helm chart)
• New: internal/bootstrap/otel_test.go — unit tests covering all acceptance criteria
• Modified: internal/mlflow/types.go — added Address field to MLflowStatus to capture status.address.url
• Modified: cmd/main.go — --enable-otel-bootstrap flag, registers runnable via mgr.Add()
• Modified: charts/kagenti-operator/values.yaml — otelBootstrap.enable toggle
• Modified: charts/kagenti-operator/templates/manager/manager.yaml — passes flag to container args
• Modified: charts/kagenti-operator/templates/rbac/role.yaml — adds endpoints permission

All bootstrap reads use APIReader (direct API server) rather than the manager's cached client to avoid race conditions during early startup.

Test Plan

• Unit tests pass: go test ./internal/bootstrap/...
• Deployed to OCP cluster with otelBootstrap.enable=true
• Ingress CA ConfigMap created with valid cert chain on OpenShift
• MLflow endpoint dynamically resolved from CR (https://mlflow.redhat-ods-applications.svc:8443/v1/traces)
• Collector ConfigMap assembled with correct presets and workspace namespace
• Rollout restart triggered on config change; skipped on idempotent re-run

Made with Cursor

[pdettori]

Well-structured PR — operator-native startup logic with proper feature gating (off by default), idempotent create/update, leader-election guard, and solid test coverage (16 tests).

One blocking issue: both commits use Co-authored-by: Cursor which violates the repo's attribution policy (CLAUDE.md requires Assisted-By: instead of Co-authored-by: to avoid inflating GitHub contributor stats). Please amend or interactive-rebase to fix the trailers.

Two non-blocking suggestions inline.

[pdettori]

suggestion: If parsed.Scheme is empty (e.g., a URL like //host:port/path which url.Parse accepts without error), this produces an invalid "://host:port/v1/traces". Consider guarding:

if parsed.Scheme == "" || parsed.Host == "" {
    log.Info("MLflow address URL missing scheme or host, falling back", "url", cr.Status.Address.URL)
    // fall through to Status.URL fallback
} else {
    info.tracesURL = fmt.Sprintf("%s://%s/v1/traces", parsed.Scheme, parsed.Host)
    ...
}

[pdettori]

suggestion: Missing negative test cases for mlflowInfoFromCR — an empty Status.Address.URL and a malformed URL (e.g. "://badurl") would exercise the fallback logic and protect against regressions.

[pdettori]

All review comments addressed:

1. Empty scheme/host guard — mlflowInfoFromCR now checks parsed.Scheme != "" && parsed.Host != "" before constructing the traces endpoint (commit 44016d4).
2. Negative test cases — Three tests added: empty URL, malformed URL (://badurl), and scheme-less URL (//host:port) all exercise the fallback logic.
3. Attribution trailers — All commits now use Assisted-by: instead of Co-authored-by:.

LGTM — well-structured fix commit on top of an already solid PR.