Skip to content

Add a smoke/full addon profile so dev spokes can skip the heavy catalog #83

Description

@stxkxs

Context

#82 ("Full addon catalog overwhelms small/dev clusters") was substantially resolved by three infra fixes (Cilium prefix-delegation for ~4x node IPs, the NodePool arch fix, and a system-cluster-critical Karpenter priority). Prefix-delegation removes the IP-exhaustion root cause directly. This issue tracks the remaining, separable optimization from that ticket: a profile knob so a throwaway dev spoke doesn't install the full catalog at all.

What

A cluster-secret label addon_profile (smoke | full, default full) that gates the heavy, optional, spoke-targeting addons:

  • Heavy/optional on spokes: falco, trivy, gpu-operator, aws-neuron, the grafana/prometheus stack.
  • Always-on (critical): cert-manager, external-secrets, cilium, karpenter, the operator, kyverno policies.

ApplicationSet cluster generators gate with a safe-by-default expression — a cluster with no label still matches, so prod is unaffected:

matchExpressions:
  - key: addon_profile
    operator: NotIn
    values: [smoke]

The wrinkle

addons-security bundles kyverno (policy enforcement — should stay) with falco + trivy (heavy — should skip). Appset-level gating is all-or-nothing, so this needs the security appset split into a policy appset (always-on) and a scanning appset (profile-gated) rather than a single matchExpression.

Companion (landing-zone)

The gate is inert until something sets the label. landing-zone cluster-bootstrap should stamp addon_profile on the cluster secret from the environment (dev → smoke, staging/prod → full), or expose it as a Cluster claim field in eks-fleet.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions