Operator exits on missing config at startup, so an ordering race costs exponential backoff

[stxkxs]

What happens

On a fresh cluster the operator crashloops before becoming Ready. Observed on a live build: 6 restarts over 6m20s, then healthy.

{"level":"error","logger":"setup","msg":"unable to load operator config from SSM",
 "clusterName":"development-platform",
 "error":"ssm GetParametersByPath /eks-agent-platform/development-platform/:
          operation error SSM: GetParametersByPath, get identity: get credentials:
          failed to refresh cached credentials, failed to retrieve credentials,
          operation error STS: AssumeRoleWithWebIdentity, https ..."}

The inner cause is not a missing SSM parameter — it is that the operator's IRSA role does not exist yet, so it cannot get credentials to read SSM at all.

Why the race exists

Two owners install these on different clocks:

	installs the operator	creates the role + SSM params
who	eks-gitops addons-agent-operator, sync-wave 21	landing-zone agent-iam
when	as soon as ArgoCD syncs the catalog	later in the substrate sequence

Nothing orders them, and nothing should have to — they are different systems. The operator is simply installed before the identity it assumes exists.

The part worth changing

Failing closed is correct and should stay. Config.Validate() treats TenantPermissionsBoundaryARN and friends as required precisely so the operator cannot mint tenant roles with no boundary. That reasoning is sound and the comment in operatorconfig/config.go states it well.

The issue is the mechanism: it fails closed by exiting, which hands the problem to CrashLoopBackOff. Backoff is exponential and caps at 5 minutes, so once the role finally appears the pod can still sit idle for minutes before its next attempt. The delay is unrelated to how long the dependency actually took.

A startup that cannot yet read its config is not the same as a startup that is misconfigured. The first is a normal state on a cold cluster and resolves on its own; the second never does.

Shape of a fix

Have the manager start with a not-ready readiness probe and poll for config, rather than exiting:

• start the manager, register the health/ready endpoints, report not ready
• poll SSM on a fixed interval, logging at a level that does not read as an error for the first N attempts
• once Validate() returns empty, wire the reconcilers and flip ready
• keep exiting for the genuinely-unrecoverable cases (malformed values, a cluster name that cannot be resolved), so "misconfigured" and "not yet configured" stay distinguishable

That converts an exponential penalty into a linear poll, and makes the pod's own readiness the honest signal for "config has arrived" — which is also what a kubectl wait --for=condition=Available upstream of it actually wants to mean.

Why it matters beyond the e2e

The landing-zone e2e absorbed this because its operator wait is generous. A real single-account install has the same ordering and no such allowance — the operator is installed by the GitOps catalog the moment the cluster is up, and the IAM substrate lands afterwards. The visible symptom there is a crashlooping operator on a cluster where nothing is wrong, which is expensive to diagnose and reads as a much worse problem than it is.

[stxkxs]

Reproducible across runs

Second consecutive live cluster build shows the same pattern, so this is the steady-state startup behaviour rather than a one-off race that happened to land badly:

run	restarts before Ready
11	6 (over 6m20s)
12	5 (over ~4m30s)

Both recovered on their own. Both cost minutes that the dependency itself did not.

The clock is set by CrashLoopBackOff, not by how long the IRSA role actually took to appear: once the role exists, the pod still waits out whatever backoff interval it had reached. That is why the two runs differ — not because the substrate was slower, but because the exponential landed differently.

Worth noting the failure is invisible in the place an operator would look. kubectl get pods shows 1/1 Running with a restart count, and the reason is only in --previous logs, which are gone once the container has been up long enough. On a cluster where everything else is healthy, a five-restart operator reads as a much worse problem than "its dependency had not been created yet."