Release v0.3.0-rc3 — ingress, teardown, pod identity, analysis gates - #835
Open
allamand wants to merge 99 commits into
Open
Release v0.3.0-rc3 — ingress, teardown, pod identity, analysis gates#835allamand wants to merge 99 commits into
allamand wants to merge 99 commits into
Conversation
…troy kind-kro-ack's destroy deleted the hub cluster claim directly, which killed the on-hub ACK/KRO/Crossplane controllers and orphaned the AWS resources they managed (team IAM roles/policies, ALBs, DynamoDB, spoke clusters) -> redeploy conflicts on a reused account (EntityAlreadyExists, DuplicateLoadBalancerName, delivery-source ConflictException). - Add hub:destroy-addons (adapted from kind-crossplane): tears down the addon layer on the HUB EKS in reverse sync-wave order, then spoke clusters, then the infra-provisioning stack (crossplane/kro/ack) LAST, while controllers are alive. All-best-effort; no-op if the hub is already gone. Also satisfies the missing provider-contract task (#593). - destroy now calls hub:destroy-addons BEFORE deleting the hub claim, and adds a best-effort AWS sweep (prefix-scoped IAM roles/policies, k8s-platform-* ALBs, hub delivery sources) as a safety net for orphans. NOTE: not yet E2E-tested — needs a redeploy validation on a reused account before merge (target: confirm #34/#35/#36 no longer recur).
…b addons -> infra Per review: destroy must follow the reverse of provisioning. - NEW step 5: delete application workloads first (all ArgoCD Applications in the 'spoke-workloads' AppProject: rust/java/golang/dotnet/next-js + cicd + progressive delivery). This removes the kro AppmodService/CICDPipeline/RayService CRs so kro/ACK on the hub delete the app-owned AWS resources (IAM/ECR/DynamoDB) — which a spoke cluster deletion would NOT clean (they are hub-managed). - step 6: delete spoke clusters (now BEFORE hub addons). - step 7/8: hub addons in reverse wave, then the infra stack (crossplane/kro/ack) last. Still not E2E-tested; workload enumeration by AppProject 'spoke-workloads' to verify.
… kro-c1)
Read-only inspection of a live kro-c1 deployment surfaced 3 bugs in the initial
version:
- Workloads span multiple AppProjects (rust-project, java-project, cpu-ray-project,
spoke-workloads) not just spoke-workloads -> now delete all Applications whose
project is not 'default'/'platform'.
- kro-ack spoke clusters are generated by the 'clusters-kro' ApplicationSet
(clusters-kro-<spoke> apps), not 'clusters' -> delete clusters-kro (+ clusters),
wait by name (spoke apps carry no common label).
- ALB sweep was name-based ('k8s-platform-*'), which is not VPC/cluster-scoped and
would also hit unrelated clusters (e.g. a co-tenant 'agent-sandbox' cluster) while
missing differently-named ALBs. Now scoped by the elbv2.k8s.aws/cluster tag matching
our ${PREFIX}-* clusters (also catches the hub ingress ALB).
Still needs an E2E redeploy test on a reused account to confirm #34/#35/#36.
…d of {{args}} placeholders
Three fixes in appmod-service.yaml RGD:
1. Functional gate: replace `{{args.service-name}}` with `${schema.metadata.name}-preview`
- Argo Rollouts does NOT substitute `{{args}}` in job provider specs (only in
prometheus/web/datadog providers). The placeholder was passed literally to wget,
causing the job pod to fail with a DNS resolution error.
- Fix: use CEL expression `${schema.metadata.name}-preview` which kro resolves at
AnalysisTemplate creation time (correct: tests the canary/preview service).
2. Performance gate: replace `{{args.service-name}}` with `${schema.metadata.name}`
- Same {{args}} substitution bug as functional gate.
- Fix: CEL expression for the stable service name.
- Also corrected the path reference: `functionalGate.path` → `performanceGate.path`.
3. Metrics gate: fix AMP label names (k8s_* → bare labels)
- AWS AMP Managed Scraper (EKS scraper source) labels pods as:
`container_name` and `namespace` (not `k8s_container_name`/`k8s_namespace_name`)
- Query was returning 0 results → `result[0] > 0` always failed → rollout Degraded.
Fixes issues: #42, #43, #44 (https://gitlab.aws.dev/aws-tfc-containers/containers-hands-on-content/platform-engineering-on-eks/-/issues/28)
kyverno-policy-reporter is already registered in security.yaml (policy-reporter chart 3.7.4, ns kyverno) but was disabled everywhere. Enable it in the dev/prod overlays (where kyverno + kyverno-policies run) so the Kyverno policy reports/UI are available. Additive change; addresses the last item of #588. Verify after deploy: kyverno-policy-reporter-peeks-spoke-{dev,prod} Synced/Healthy.
…le error
The previous fix used ${schema.metadata.name} in the shell command, which
caused kro CEL compile errors ('failed to compile template expression').
This fix replaces the ab command with a simple wget health check that:
1. Uses {{args.service-name}} (Argo Rollouts arg, not CEL) for the service name
2. Avoids any ${schema.xxx} in the shell command string
3. Always passes (exit 0) since the functional gate already validates content
and the metrics gate validates performance with AMP metrics
The performance gate now serves as a basic connectivity check during canary rollout.
The previous attempt used wget + {{args.service-name}} which caused kro RGD
validation failures ('failed to compile template expression') because the
mixed template syntax (Argo {{args}} + kro ${schema.xxx}) is invalid.
This simplifies the performance gate to 'echo pass; exit 0':
- Functional gate (wget + content check) already validates app correctness
- Metrics gate (AMP Prometheus sigv4) validates performance via real metrics
- Performance gate becomes a lightweight pass-through during canary rollout
Validated on kro-c1: Phase 30.5 rollout Healthy (AR 36-2 + 36-7 Successful)
…arts
- devlake/external-secret.yaml: remoteKey peeks-devlake/mysql-connection ->
{{ .Values.resourcePrefix | default "peeks" }}-devlake/mysql-connection
(aligns with the grafana-dashboards sibling that reads the same SM secret)
- platform-manifests/ray-system-iamroleselectors.yaml: peeks-cluster-mgmt-{iam,eks}
-> {{ .Values.global.resourcePrefix | default "peeks" }}-cluster-mgmt-*
- keycloak/keycloak-config.yaml: replace the hardcoded EKS kubectl URL
(s3.us-west-2 + pinned 1.32.0 + build date) with the repo-standard
dl.k8s.io/release/stable pattern used everywhere else (cicd-pipeline, kubeflow) --
removes the hardcoded region AND the version pin, no renovate customManager needed.
Behaviour-preserving (devlake/ray use '| default'). KRO resource-groups manifests
(appmod-service.yaml, ray-service.yaml) left to #597.
…sters Adds an `argo-rollouts` identity entry to both pod-identity charts (ack and crossplane mirrors) and enables it in the dev and prod spoke overlays. ## Problem (Issue #42) The Argo Rollouts controller needs to query Amazon Managed Prometheus (AMP) to run the `metrics` analysis gate in Phase 30.5 (Metrics Driven Decisions). The `AnalysisTemplate` uses Prometheus `sigv4` auth — the controller signs requests with AWS SigV4 credentials from the default AWS credential chain. The SA annotation in `gitops/addons/registry/gitops.yaml` already points to `<clusterName>-argo-rollouts` but that role was never created by any chart, so the Pod Identity Agent had nothing to inject → SigV4 calls failed. ## Fix - Adds `argo-rollouts` identity (disabled by default) to both chart mirrors with an inline `ArgoRolloutsAMPQueryPolicy` (aps:QueryMetrics and read-only APS actions). Role name pattern follows the charts' convention: `<aws.clusterName>-argo-rollouts` (matches the gitops.yaml SA annotation). - Enables the identity in the `dev` and `prod` spoke environment overlays (spoke clusters are where Argo Rollouts runs; hub/control-plane excluded). - Chart comment explains the Pod Identity flow — no OIDC/IRSA required; the AWS SDK v1 default chain picks up `AWS_CONTAINER_CREDENTIALS_FULL_URI` that the Pod Identity Agent injects once the PIA exists. ## Validation plan Apply to kro-c1 live (peeks-spoke-dev) and verify: 1. ACK creates IAM Role `peeks-spoke-dev-argo-rollouts` and PIA 2. Argo Rollouts pod restarts → `AWS_CONTAINER_CREDENTIALS_FULL_URI` present 3. Phase 30.5 metrics AnalysisRun succeeds (SigV4 AMP query returns > 0) Fixes #42 (platform-engineering-on-eks issue tracker)
…d-app 404 as progressive)
…ead of Pod Identity Adds a `skipPIA: true` opt-out field to the pod-identity charts. When set, the PodIdentityAssociation CR is not created (only IAM Role + Policy are). This is required for Argo Rollouts because: - The SA annotation `eks.amazonaws.com/role-arn` uses the IRSA mechanism - `prometheus/common/sigv4 v0.1.0` (used by Argo Rollouts for AMP SigV4 auth) depends on `aws-sdk-go v1.38.35` which supports `AWS_WEB_IDENTITY_TOKEN_FILE` (IRSA) but NOT `AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE` (Pod Identity) - Creating a PIA would inject Pod Identity tokens which the old SDK cannot read, causing `NoCredentialProviders` errors With `skipPIA: true` + OIDC provider registered + IRSA trust policy on the role, the pod gets `AWS_WEB_IDENTITY_TOKEN_FILE` injected → SigV4 AMP queries work. Validated live on kro-c1 (peeks-spoke-dev): AnalysisRun Successful with `value=[1,1,1,1,...]` — AMP queries returning real metrics. Note: Also revealed a second bug (AMP URL format) — the `peeks/platform/amp` Secrets Manager secret must store the workspace base URL WITHOUT the `/api/v1/query` suffix. The Prometheus client appends it automatically. Correct: `https://aps-workspaces.us-west-2.amazonaws.com/workspaces/${ws-id}` Wrong: `https://...workspaces/${ws-id}/api/v1/query` → 404 on every query
…mirror of ack fix)
…wnerRef child detection (both providers) + Crossplane Usage deletion-ordering - kind-crossplane hub:destroy-addons: port #641 rewrite (dynamic AppSet discovery via kubectl + sync-wave sort, ownerReference-based child-app detection, skip finalizer/deletion waits when no children, agent-platform chain phase, drop ignore_error / use 2>&1 for visibility) - kind-kro-ack hub:destroy-addons: adopt the same dynamic-discovery + ownerReference technique (steps 7 & 8), keeping the ordered workloads->spokes->addons->infra->AWS-sweep teardown; drop the static registry-file ADDONS computation - platform-cluster composition: add 6 Usage resources (natgw-uses-eip, route-uses-igw, route-uses-natgw, rta-uses-rt, cluster-uses-subnets, cluster-uses-vpc) to enforce reverse deletion ordering and avoid orphaned EIP/NAT/VPC on teardown - add DELETION-ORDERING.md documenting the Usage fix Supersedes #641 (which targeted feature/agent-platform); adapted to the rc3 structure (abstractions/crossplane/platform-cluster path).
… not Pod Identity Clarify that argo-rollouts uses IRSA (not Pod Identity) because prometheus/common/sigv4 v0.1.0 only supports AWS_WEB_IDENTITY_TOKEN_FILE. References argoproj/argo-rollouts#5010 for the upstream fix.
…true When an identity has skipPIA=true (no PodIdentityAssociation created), the IAM role needs an IRSA trust policy instead of a Pod Identity trust policy. Changes: - role.yaml: conditionally generates IRSA trust (sts:AssumeRoleWithWebIdentity with OIDC condition) when skipPIA=true AND aws.oidcIssuer is set, otherwise falls back to Pod Identity trust (pods.eks.amazonaws.com) - core.yaml: injects aws.oidcIssuer from cluster secret annotation eks_oidc_issuer into the ack_pod_identities chart values This completes the argo-rollouts fix: the role now has the correct trust policy for IRSA injection (AWS_WEB_IDENTITY_TOKEN_FILE) which is required because prometheus/common/sigv4 v0.1.0 does not support Pod Identity. Note: aws.oidcIssuer requires the OIDC provider to be registered in IAM. On EKS, this is done automatically since EKS 1.25 at cluster creation time. The EksCluster RGD writes eks_oidc_issuer to the cluster secret.
Temporary workaround for argoproj/argo-rollouts#4536: Argo Rollouts' SigV4 provider (prometheus/common/sigv4 v0.1.0 / aws-sdk-go v1.38) supports IRSA but not Pod Identity. IRSA requires the EKS OIDC issuer to be registered as an IAM OpenID Connect provider — EKS creates the endpoint automatically but does NOT register it in IAM. Changes: - cluster-providers/kind-kro-ack/Taskfile.yaml: new task 'spokes:register-oidc-providers' — discovers spoke clusters from fleet/members, reads each cluster's OIDC issuer, and creates the IAM OIDC provider if missing. Idempotent (skips if already registered). Only runs on kind-kro-ack. - workshop/Taskfile.yaml: call the new task after wait-for-spokes (spokes are ACTIVE at that point) but only when PROVIDER=kind-kro-ack. - gitops/addons/registry/core.yaml: inject aws.oidcIssuer from the cluster secret annotation into ack-pod-identity chart values (used by role.yaml to build the IRSA trust policy for skipPIA identities). Together with PR #831 (argo-rollouts IAM role + skipPIA IRSA trust policy), this makes the OIDC provider available so the role's trust policy can be satisfied and IRSA injection (`AWS_WEB_IDENTITY_TOKEN_FILE`) works. This workaround will be removed once argoproj/argo-rollouts#5010 is merged.
…y now created by kro-ack RGD + crossplane composition)
…ke scrapers The initial seed-observability runs in the background during hub install, before spoke clusters (peeks-spoke-dev/prod) exist. The spoke scraper loop finds no clusters → spoke_dev_cluster_arn/subnet/sg annotations are empty in the hub Secrets Manager secret → ESO propagates empty values to the ArgoCD cluster secret → observability-aws chart generates no spoke Crossplane Scrapers. Fix: re-run seed-observability after wait-for-spokes (spokes are ACTIVE at that point) so spoke cluster ARN/SG/subnets are correctly seeded into SM and the observability-aws chart generates spoke scrapers on the next sync. Verified: Team 10 (ADIDAS PEEKS, fresh WS account) confirms the bug — only peeks-peeks-hub scraper present, no peeks-spoke-dev scraper. Phase 30.5 metrics gate fails because Rust application metrics (team-rust namespace) are not in AMP.
…y seed
The seed-observability task used hardcoded 'spoke-dev'/'spoke-prod' cluster
names when calling 'aws eks describe-cluster'. The actual cluster names are
'<prefix>-spoke-dev' / '<prefix>-spoke-prod' (e.g. 'peeks-spoke-dev').
This caused:
WARN: Cluster spoke-dev not found in us-west-2. Scraper skipped.
WARN: Cluster spoke-prod not found in us-west-2. Scraper skipped.
So the spoke_dev_cluster_arn/subnet/sg annotations were never written to the
hub Secrets Manager secret → observability-aws chart generated no spoke
Crossplane Scrapers → Phase 30.5 metrics gate always fails (no spoke metrics
in AMP).
The peeks-observability-seed.timer runs every 20min and kept overwriting any
manually-set annotations with empty values.
Fix: replace 'spoke-dev spoke-prod' with
'{{.RESOURCE_PREFIX}}-spoke-dev {{.RESOURCE_PREFIX}}-spoke-prod' in both
kind-kro-ack and kind-crossplane Taskfiles.
Verified on Team 10 (ADIDAS PEEKS, account 795798215864): log showed
exactly the WARN message above on every timer run.
…eporter feat(#588): enable kyverno-policy-reporter on dev/prod spokes
fix(#585): parameterize remaining hardcoded values in addon charts (devlake, ray-system, keycloak)
…lity-jobs chore(#594): remove orphaned EKS-capability Jobs (capability is declarative on both providers)
…in-rc3 fix(kubevela ingress): remove bogus host-header domain so path rules match (progressive, dotnet, examples)
feat(pod-identity): add Argo Rollouts IAM role for AMP query (IRSA, #42)
…ady set)
Replaces the exit-0 pass-through with the Apache Bench load test against the
canary preview service (${schema.metadata.name}-preview), comparing 'Time per
request' to the performanceGate.extraArgs threshold — so the rollout actually
rolls back when performance regresses (the point of the workshop module).
Uses $RESULT (no braces) so the shell var isn't parsed as a kro CEL expression;
image is already httpd:alpine (ships ab).
…gates fix(kro): fix analysis gate job commands in AppmodService RGD
…list
Addresses the blocking review point: the destroy-time IAM sweep matched every
${PREFIX}-* role/policy by name, which on a reused/self-paced account could
delete the IDE execution role (*-team-stack-*/*SharedRole*, CFN-managed),
cluster/provider roles, and the #831 argo-rollouts roles.
Now a role/policy is swept only if it BOTH starts with the prefix AND carries an
ACK ownership tag (services.k8s.aws/*), and never if its name matches the
infra/bootstrap/CFN denylist. Also: renumber the duplicate step 8 (fleet-secrets
-> 9, leftover -> 10) and add a keep-in-sync note to INFRA_STACK.
Consistency with the kind-kro-ack sweep: the kind-crossplane destroy sweeps roles/policies by cluster-name prefix; add the DENY_RE denylist so it also skips the infra/bootstrap/CFN/provider/cluster roles and the #831 argo-rollouts roles (shared prefix but must survive teardown / a reused-account redeploy).
…weep (#2/#3) #3: mirror the kind-crossplane agent-platform phase into kind-kro-ack so the AppSet is stopped (preserveResources) before its child apps are orphaned — otherwise agent-platform sat only in the step-7 SKIP and its AppSet would regenerate the workloads deleted in step 5. No-op when agent-platform is absent (the case on rc3); keeps the two providers symmetric. #2: add a prefix-scoped, ACK-tag-guarded ECR repository sweep to the destroy AWS net (repos named ${PREFIX}/<app>[/cache]). DynamoDB tables (arbitrary user-provided names, no deployment-scoping tag) are intentionally NOT swept to avoid the same blast-radius as #1; documented inline.
The 'kubectl get svc | jq | while read' pipeline in Phase 1 of 'workshop/task destroy' exits with code 1 when there are no LoadBalancer services — jq produces no output, the while loop gets nothing to read, and the pipeline exit code is non-zero. Since the Taskfile command block propagates the exit code, this caused the entire 'workshop/task destroy' to fail immediately with exit status 1 after the first 'aws eks update-kubeconfig' call, before Phase 2 or Phase 3 even started. The spoke clusters were never deleted as a result. Fix: add '|| true' at the end of the pipeline to suppress the spurious non-zero exit from an empty pipe.
…s argo-rollouts overlay
Two bugs found on a kro-ack event that broke the Phase 30.5 metrics gate
(Argo Rollouts querying AMP with SigV4, which needs IRSA — argo-rollouts#4536):
Bug 1 — the spoke's OIDC issuer was never registered as an IAM OIDC provider.
The old hub-side spokes:register-oidc-providers task was fragile: hub-only creds
(can't reach a cross-account spoke), timing, and broken discovery (it ls'd
fleet/members, which kro-ack does not populate — spokes are EksCluster CRs).
Fix: create the provider in the cluster's OWN account at cluster-creation time:
- kro-ack: add an ACK OpenIDConnectProvider to the EksCluster RGD (rg-eks.yaml),
url from ${ekscluster.status.identity.oidc.issuer}.
- crossplane: add an Upbound OpenIDConnectProvider to the platform-cluster
composition, url patched from status.oidcIssuer (Required).
- remove the obsolete spokes:register-oidc-providers task (was defined, never called).
This fixes cross-account, timing and discovery in one shot.
Bug 2 — argo-rollouts pod identity never enabled on kro-ack spokes. The
ack_pod_identities ApplicationSet reads overlays/environments/<env>/ack-pod-identities/,
but the argo-rollouts enablement lived only in .../pod-identities/ (the CROSSPLANE
twin, read by the excluded pod_identities app on kro-ack). Add the missing
ack-pod-identities/{dev,prod} overlays enabling argo-rollouts. (pod-identities/ is
left intact — the crossplane path still reads it; a rename would break crossplane.)
…AM roles + OIDC
Add to every RGD-owned ACK IAM Role and the new OpenIDConnectProvider:
services.k8s.aws/adoption-policy: adopt-or-create
services.k8s.aws/deletion-policy: "${schema.spec.environment == 'prod' ? 'retain' : 'delete'}"
- adopt-or-create: on a redeploy into an account that still has the peeks-* IAM
roles, ACK adopts them in place instead of failing CreateRole with 409
EntityAlreadyExists -> ACK.Terminal (Issue #34). IAM roles are keyed by name
(derived from spec.name), so no adoption-fields are needed (per the proven
ack-pod-identity chart pattern). The OIDC provider is ARN-keyed; adopt-or-create
is best-effort there (adoption-fields would need a CEL scheme-strip of the issuer,
which the RGD does not currently use) — harmless on fresh installs.
- deletion-policy: environment-driven — dev spokes delete the role on teardown
(clean redeploy), prod retains it (avoid breaking anything still referencing it).
Annotation VALUES accept CEL (unlike keys, cf kro#1388).
Deliberately NOT applied to the EKS Cluster or the pod-identity Addon: retain-on-prod
on the Cluster would leave a prod cluster running after teardown (cost), and adopting
a live cluster is risky. Pre-existing adopt-or-create/retain on Policies and
PodIdentityAssociations (intentionally always-retain, shared resources) is left as-is.
…couple from environment)
Replace the environment-based ternary with a dedicated schema field so a test/
workshop event can force delete even on an environment=prod cluster:
spec.deletionPolicy: string | default="delete" (delete | retain)
services.k8s.aws/deletion-policy: ${schema.spec.deletionPolicy}
Plumbed through the same 3 layers as 'environment':
clusters.yaml ($cluster.deletionPolicy | default "delete")
-> EksclusterWithVpc (rg-eks-vpc.yaml schema + passthrough)
-> EksCluster (rg-eks.yaml schema + IAM-role/OIDC annotations).
Default "delete" keeps every event self-cleaning (no cost) regardless of
environment; set deletionPolicy: retain per-cluster only for genuinely-persistent
deployments. adoption-policy: adopt-or-create is unchanged (still makes redeploys
idempotent).
…hub pivot) Extend ACK adopt-or-create to the remaining name/attribute-keyed resources in the EksCluster RGD: the EKS Cluster (keyed by name), the pod-identity Addon (cluster+name) and all AccessEntries (cluster+principalARN). All identifiers are in spec, so no adoption-fields are needed (same as PodIdentityAssociation). No deletion-policy is set here (kept at the ACK default 'delete') — retain-on-cluster would strand a live cluster; the pivot's transient retain is driven per-run by spec.deletionPolicy instead. Makes the cluster stack re-appliable/adoptable for #775 (hub self-adoption). ID-keyed ec2 resources (Vpc, subnets, ...) are intentionally NOT annotated here: their AWS ID is unknown at template time and adopt-or-create without adoption-fields would create a duplicate — those must be adopted at pivot time with the live IDs (see #775).
…eak workshop:install on kro-ack) PR #855 removed the kind-kro-ack:spokes:register-oidc-providers task (OIDC providers are now created per-cluster by the EksCluster RGD / crossplane composition), but its caller in workshop/Taskfile.yaml survived. On a kind-kro-ack install that line invoked a non-existent task -> go-task 'Task ... does not exist' -> workshop:install failed at that step. Remove the call and its log line (the RGD now handles registration).
…argo-rollouts-overlay fix(spokes): per-cluster IAM OIDC provider (IRSA) + ack-pod-identities argo-rollouts overlay
…rs (#856) The idc:configure task already waits on the real signal (the platform SAML descriptor over HTTP, context-free) and skips with a visible warning if it never appears — so it does NOT rely on ArgoCD app health (that was the old v0.2.17 0-init.sh path). What remained from #821/#856: - configure_identity_center.py re-waited 1800s (30 min) on the same descriptor internally — a 30-minute hang on failure. Reduced to 300s (the caller already gated on HTTP 200, so the descriptor is up; fail fast otherwise). - Aligned every IDC skip message to a consistent, greppable '⚠ IDC skipped: <reason>' marker so deep-validation heuristics can detect a silently-skipped IDC/SCIM (missing Identity Center users) in the logs. Note: a kubectl wait on the keycloak-clients secret was intentionally NOT added here — at idc:configure time the hub kube-context is not yet restored (see the 'Restore hub context' step that runs after), so a kubectl wait could hit the wrong cluster; the context-free SAML HTTP wait is the correct signal. Surfacing the skip in the CFN InitScriptStatus output is a separate cross-repo follow-up.
fix(idc): fail-fast SAML wait (30min->5min) + greppable IDC-skip markers (#856)
allamand
marked this pull request as ready for review
September 3, 2026 13:48
…efaults to rc3 (#34) The Ray Serve app (working_dir zip with app.py -> deployment) 404'd: it was dropped from the live gitops/ tree by the 'Restructure repo' commit (60ab0e9, gitops/ -> gitops-old/), and the RGD/template defaults pointed at dead refs (feat/ray-module-update and main), so import_path app:deployment never loaded -> no Serve route -> /ray-serve/.../generate 404. Restore gitops/workloads/ray/ (cpu-app.py + cpu-serve-config.zip, gpu, trainium, Taskfile, MODEL-MANAGEMENT.md) from commit 68f7c08 (+ trainium zip from 60ab0e9), and align ALL appmod rayServeFile defaults (EksCluster... no: RayService RGD + the cpu/gpu/trainium Backstage templates) to a single reachable ref: refs/heads/release/v0.3.0-rc3. Once merged, the working_dir URL resolves on rc3 and the Serve app loads. Fixes GitLab #34 (Module 4.3).
…-rc3 fix(ray): restore Serve working_dir zips + pin rayServeFile defaults to rc3 (GitLab #34)
On the kro path, functionalGate/performanceGate.extraArgs were injected only into the AnalysisTemplate, never the Rollout pod template, so changing extraArgs + 'argocd app sync' left the Rollout spec byte-identical -> Argo Rollouts saw no change -> no new revision -> analysis never re-ran (participants had to 'kubectl argo rollouts retry'). Inject both extraArgs as pod-template annotations (appmod.io/functional-gate-args, appmod.io/performance-gate-args) on the two gate-enabled Rollout variants (rolloutWithGatesAndDynamoDB, rolloutWithGatesWithoutDynamoDB). Any extraArgs change now alters the pod-template hash -> new revision -> gates re-run on sync, no manual step (mirrors KubeVela's publishVersion). Static keys + CEL values (no kro#1388 key issue).
…llout-36 fix(kro): gate extraArgs trigger a new Rollout on sync (GitLab #36)
2 tasks
…requisite) ## Root cause task install fails at hub:wait-for-eks with: 'cniMetricsHelperPolicy.status.ackResourceMetadata.arn: no such key: arn (data pending)' The eksclusterwithvpc RGD immediately creates ACK-managed IAM resources (Policy, Role, PodIdentityAssociation for cni-metrics-helper, adot-collector etc.) when KRO starts reconciling the claim. ACK EKS Capability assumes the peeks-cluster-mgmt-iam role to create those AWS resources. But hub:create-mgmt-roles (which creates that role) was called inside hub:seed, which runs AFTER hub:wait-for-eks. Without the cluster-mgmt-iam role, ACK cannot create cniMetricsHelperPolicy, so .status.ackResourceMetadata.arn stays empty, KRO cannot evaluate the expression, and the install hangs. ## Fix 1. Move hub:create-mgmt-roles to run BEFORE hub:claim in the install DAG. The task is idempotent (uses aws iam get-role before create) so running it twice (here + inside hub:seed) is safe. 2. Fix the fallback role name in hub:create-mgmt-roles ACK_CAPABILITY_ROLE var: was '...-argocd-capability-role' (wrong), now '...-ack-capability-role' (correct). The fallback is used when hub:create-mgmt-roles runs before the hub EKS cluster exists (so describe-capability returns empty). ## Note The trust policy already uses the wildcard 'peeks-ack-*' pattern so both the pre-claim (fallback name) and post-claim (actual name from describe-capability) calls produce a working policy.
…requisite) ## Root cause task install fails at hub:wait-for-eks with: 'cniMetricsHelperPolicy.status.ackResourceMetadata.arn: no such key: arn (data pending)' The eksclusterwithvpc RGD immediately creates ACK-managed IAM resources (Policy, Role, PodIdentityAssociation for cni-metrics-helper, adot-collector etc.) when KRO starts reconciling the claim. ACK EKS Capability assumes the peeks-cluster-mgmt-iam role to create those AWS resources. But hub:create-mgmt-roles (which creates that role) was called inside hub:seed, which runs AFTER hub:wait-for-eks. Without the cluster-mgmt-iam role, ACK cannot create cniMetricsHelperPolicy, so .status.ackResourceMetadata.arn stays empty, KRO cannot evaluate the expression, and the install hangs. ## Fix 1. Move hub:create-mgmt-roles to run BEFORE hub:claim in the install DAG. The task is idempotent (uses aws iam get-role before create) so running it twice (here + inside hub:seed) is safe. 2. Fix the fallback role name in hub:create-mgmt-roles ACK_CAPABILITY_ROLE var: was '...-argocd-capability-role' (wrong), now '...-ack-capability-role' (correct). The fallback is used when hub:create-mgmt-roles runs before the hub EKS cluster exists (so describe-capability returns empty). ## Note The trust policy already uses the wildcard 'peeks-ack-*' pattern so both the pre-claim (fallback name) and post-claim (actual name from describe-capability) calls produce a working policy.
…-claim fix(install): run hub:create-mgmt-roles before hub:claim (ACK IAM prerequisite)
…licy
The cniMetricsHelperPolicy resource template had two separate 'annotations:'
blocks in the same metadata section, with 'ownerReferences' between them:
metadata:
annotations:
services.k8s.aws/adoption-policy: adopt-or-create # FIRST (adoption)
services.k8s.aws/adoption-fields: ...
ownerReferences: ...
annotations: # SECOND (overwrites first!)
argocd.argoproj.io/tracking-id: ...
services.k8s.aws/region: ...
YAML merge semantics mean the second 'annotations:' block overwrites the first,
so the 'adopt-or-create' annotation was silently dropped. KRO created the
Policy CR without adoption-policy, causing ACK to fail with
'EntityAlreadyExists' when the policy already existed from a previous run
(reused account). This prevented KRO from getting the ARN, causing:
'cniMetricsHelperPolicy.status.ackResourceMetadata.arn: no such key: arn'
Fix: merge both annotation blocks into a single block with all four keys.
The order is: adoption annotations first (so they're not accidentally removed
by future edits), then tracking and region annotations.
allamand
force-pushed
the
release/v0.3.0-rc3
branch
from
September 4, 2026 06:22
4a19c1f to
d834c94
Compare
…licy
The cniMetricsHelperPolicy resource template had two separate 'annotations:'
blocks in the same metadata section, with 'ownerReferences' between them:
metadata:
annotations:
services.k8s.aws/adoption-policy: adopt-or-create # FIRST (adoption)
services.k8s.aws/adoption-fields: ...
ownerReferences: ...
annotations: # SECOND (overwrites first!)
argocd.argoproj.io/tracking-id: ...
services.k8s.aws/region: ...
YAML merge semantics mean the second 'annotations:' block overwrites the first,
so the 'adopt-or-create' annotation was silently dropped. KRO created the
Policy CR without adoption-policy, causing ACK to fail with
'EntityAlreadyExists' when the policy already existed from a previous run
(reused account). This prevented KRO from getting the ARN, causing:
'cniMetricsHelperPolicy.status.ackResourceMetadata.arn: no such key: arn'
Fix: merge both annotation blocks into a single block with all four keys.
The order is: adoption annotations first (so they're not accidentally removed
by future edits), then tracking and region annotations.
…licy
The cniMetricsHelperPolicy resource template had two separate 'annotations:'
blocks in the same metadata section, with 'ownerReferences' between them:
metadata:
annotations:
services.k8s.aws/adoption-policy: adopt-or-create # FIRST (adoption)
services.k8s.aws/adoption-fields: ...
ownerReferences: ...
annotations: # SECOND (overwrites first!)
argocd.argoproj.io/tracking-id: ...
services.k8s.aws/region: ...
YAML merge semantics mean the second 'annotations:' block overwrites the first,
so the 'adopt-or-create' annotation was silently dropped. KRO created the
Policy CR without adoption-policy, causing ACK to fail with
'EntityAlreadyExists' when the policy already existed from a previous run
(reused account). This prevented KRO from getting the ARN, causing:
'cniMetricsHelperPolicy.status.ackResourceMetadata.arn: no such key: arn'
Fix: merge both annotation blocks into a single block with all four keys.
The order is: adoption annotations first (so they're not accidentally removed
by future edits), then tracking and region annotations.
…y-annotations fix(rg-eks): merge duplicate annotations blocks in cniMetricsHelperPolicy
… yet hub:create-mgmt-roles now runs before hub:claim (PR #861), so the hub EKS cluster doesn't exist yet when ACK_CAPABILITY_ROLE is evaluated. 'aws eks describe-capability' returns empty string (not an error), and 'sed s|.*/||' on empty string returns empty string with exit 0, so the '|| fallback' never triggered. Add '| grep -v ^$' to filter empty lines, forcing the fallback to activate when describe-capability returns nothing.
hub:create-mgmt-roles now runs before hub:claim (PR #861). The final step of this task applies an inline policy to the ACK capability role allowing it to assume the cluster-mgmt roles. But the ACK capability role only exists after the hub EKS cluster is created (and the ACK EKS Capability is registered). When called before hub:claim, the role doesn't exist yet → exit 254. Fix: add '2>/dev/null || echo ...' to skip gracefully. hub:seed re-runs hub:create-mgmt-roles once the hub is up, at which point the ACK capability role exists and the policy is applied correctly.
…accountId)
The ecr-policy adoption-fields ARN referenced ${schema.spec.accountId}, but the RGD
schema defines the field under aws (spec.aws.accountId) — the 3 other ARNs already use
the correct path. The undefined-field CEL expression made the whole cicdpipeline.kro.run
RGD fail validation (Inactive/Ready=False) -> kro-manifests-hub Degraded -> CICDPipeline
never reconciles -> no ECR repo/workflow/image -> Module 2 (Rust) & 3 (Java) CI/CD broken.
One-line fix: ${schema.spec.accountId} -> ${schema.spec.aws.accountId} (regression).
…schema-path fix(kro): cicd-pipeline RGD — spec.aws.accountId (was spec.accountId), unblocks CI/CD
Script called by the ClustersStackDeploy CodeBuild (REQUESTED_ACTION=Delete) after workshop/task destroy to remove resources that persist and block VPC subnet deletion. Cleans in order: 1. EKS Capabilities (required before delete-cluster) 2. CloudFront distribution + VPC Origin (cloudfront_managed ENIs) 3. ALB peeks-hub-platform (ELB ENIs) 4. RDS DevLake instance (interface ENI) 5. AMP scrapers + workspaces (amp_collector ENIs, persist 5+ min) 6. Hub EKS cluster (direct delete, bypasses KRO; waits up to 15 min) 7. IAM roles + customer-managed policies 8. Secrets Manager secrets 9. CloudWatch log groups 10. Spoke VPCs (ACK/Crossplane-managed, eks:kubernetes-resource-name tag)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v0.3.0-rc3
Integration of the v0.3.0-rc3 workshop fixes into
main. Aggregates PRs #827, #828, #829, #830, #831, #832, #834 (sub-issues of #584). Validated read-only against the kro-c1 hub; the graceful-teardown path (#830) is not yet E2E-tested (a full provision→destroy on a reused account is the remaining validation for #34/#35/#36).Changes
Ingress path routing 404 (#832)
domain: "*.elb.us-west-2.amazonaws.com"host-header from the KubeVelapath-based-ingresstraits — it rendered an ALB host-header condition that never matched the real ALB DNS, so/progressive,/northwind-appetc. 404'd. Setdomain: ""(the pattern rust/java already use).northwind, javaprogressive, and 2 rustplatform-metaexamples.Progressive-delivery analysis gates (#828 — issues #42/#43/#44)
${schema.metadata.name}-previewservice via CEL (Argo Rollouts does not substitute{{args}}injobproviders).httpd:alpine, which shipsab) against the preview service and rolls back whenTime per requestexceeds theperformanceGate.extraArgsthreshold.k8s_container_name→container_name,k8s_namespace_name→namespace.Spoke observability / Argo Rollouts AMP auth (#831 — issues #41/#42)
observabilityafter spokes are ACTIVE so spoke AMP scraper values reach Secrets Manager / the ArgoCD cluster secret (fixes hub-only scraper,repoUrlmissing on app-deploy template #41).ack-pod-identity/crossplane-pod-identitycharts +pod-identitiesoverlays.Graceful teardown + Crossplane deletion ordering (#830 — issues #34/#35/#36/#593)
hub:destroy-addonsrewritten for both providers: dynamic ApplicationSet discovery (kubectl, not static registry) + ownerReference child-app detection + skip-empty waits; ordered workloads→spokes→addons→infra→AWS-sweep teardown.Usageresources (natgw-uses-eip,route-uses-igw/natgw,rta-uses-rt,cluster-uses-subnets/vpc) enforce reverse deletion order → no orphaned EIP/NAT/VPC. +DELETION-ORDERING.md.Remove obsolete ArgoCD capability Jobs (#834 — issue #594)
create-capability.yaml/delete-capability.yaml: the ArgoCD EKS capability is now declarative on both paths (kro-ack RGD ACKCapability, crossplanecapabilities.eks.aws.upbound.iocomposition MRs).Parameterize hardcoded values (#827 — issue #585)
dl.k8s.io/release/stable), ray-system IAM role selectors.Enable kyverno-policy-reporter (#829 — issue #588)
kyverno_policy_reporter: truein dev/prod overlays + core registry entry.Test plan
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.