fix(install): run hub:create-mgmt-roles before hub:claim (ACK IAM prerequisite) - #861
Merged
Merged
Conversation
…requisite) ## Root cause task install fails at hub:wait-for-eks with: 'cniMetricsHelperPolicy.status.ackResourceMetadata.arn: no such key: arn (data pending)' The eksclusterwithvpc RGD immediately creates ACK-managed IAM resources (Policy, Role, PodIdentityAssociation for cni-metrics-helper, adot-collector etc.) when KRO starts reconciling the claim. ACK EKS Capability assumes the peeks-cluster-mgmt-iam role to create those AWS resources. But hub:create-mgmt-roles (which creates that role) was called inside hub:seed, which runs AFTER hub:wait-for-eks. Without the cluster-mgmt-iam role, ACK cannot create cniMetricsHelperPolicy, so .status.ackResourceMetadata.arn stays empty, KRO cannot evaluate the expression, and the install hangs. ## Fix 1. Move hub:create-mgmt-roles to run BEFORE hub:claim in the install DAG. The task is idempotent (uses aws iam get-role before create) so running it twice (here + inside hub:seed) is safe. 2. Fix the fallback role name in hub:create-mgmt-roles ACK_CAPABILITY_ROLE var: was '...-argocd-capability-role' (wrong), now '...-ack-capability-role' (correct). The fallback is used when hub:create-mgmt-roles runs before the hub EKS cluster exists (so describe-capability returns empty). ## Note The trust policy already uses the wildcard 'peeks-ack-*' pattern so both the pre-claim (fallback name) and post-claim (actual name from describe-capability) calls produce a working policy.
allamand
requested review from
elamaran11,
hmuthusamy,
punkwalker and
shapirov103
as code owners
September 3, 2026 18:16
allamand
added a commit
that referenced
this pull request
Sep 4, 2026
… yet hub:create-mgmt-roles now runs before hub:claim (PR #861), so the hub EKS cluster doesn't exist yet when ACK_CAPABILITY_ROLE is evaluated. 'aws eks describe-capability' returns empty string (not an error), and 'sed s|.*/||' on empty string returns empty string with exit 0, so the '|| fallback' never triggered. Add '| grep -v ^$' to filter empty lines, forcing the fallback to activate when describe-capability returns nothing.
allamand
added a commit
that referenced
this pull request
Sep 4, 2026
hub:create-mgmt-roles now runs before hub:claim (PR #861). The final step of this task applies an inline policy to the ACK capability role allowing it to assume the cluster-mgmt roles. But the ACK capability role only exists after the hub EKS cluster is created (and the ACK EKS Capability is registered). When called before hub:claim, the role doesn't exist yet → exit 254. Fix: add '2>/dev/null || echo ...' to skip gracefully. hub:seed re-runs hub:create-mgmt-roles once the hub is up, at which point the ACK capability role exists and the policy is applied correctly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
task installfails athub:wait-for-eksafter ~40 min with:Root cause: The
eksclusterwithvpcRGD immediately creates ACK-managed IAM resources (Policy, Role, PodIdentityAssociation for cni-metrics-helper, adot-collector etc.) when KRO starts reconciling the claim. ACK EKS Capability assumes thepeeks-cluster-mgmt-iamrole (created byhub:create-mgmt-roles) to create those AWS resources. Buthub:create-mgmt-roleswas called insidehub:seed, which runs afterhub:wait-for-eks. Without the cluster-mgmt-iam role, ACK cannot createcniMetricsHelperPolicy, so.status.ackResourceMetadata.arnstays empty, KRO cannot evaluate the expression, and the install hangs indefinitely.Note: ACK in
kind-kro-ackis an EKS Capability (runs AWS-side), not pods in the cluster.Fix
Move
hub:create-mgmt-rolesbeforehub:claimin the install DAG. The task is idempotent so running it twice (here + insidehub:seed) is safe.Fix the fallback role name in
hub:create-mgmt-roles: was...-argocd-capability-role(wrong), now...-ack-capability-role(correct). This fallback is used when the task runs before the hub EKS exists (aws eks describe-capabilityreturns empty).Related
PR #835 (v0.3.0-rc3 teardown validation) — install failure observed on fresh account