fix: pin hypershift and clusters-service images to avoid e2e breakage (AROSLSRE-919, AROSLSRE-944) by raelga · Pull Request #5371 · Azure/ARO-HCP

raelga · 2026-05-23T11:09:13Z

https://redhat.atlassian.net/browse/AROSLSRE-919
https://redhat.atlassian.net/browse/AROSLSRE-944

What

Applies all image digest bumps from PR #5170 except the HyperShift operator and Clusters Service images, which remain pinned to their current working versions.

Also pins both images in tooling/image-updater/config.yaml to prevent the automated bumper from picking up the broken versions:

HyperShift operator: pinned tag from latest to cf2b91fbc02ebe5d3d66515cb2ea3e097290ac13
Clusters Service: pinned tag from latest to dbb022a3dd3f0533ae1c8eebd4e6929ba1ca1ede

The hypershift-shared-ingress image is not pinned — it uses tagPattern and is a separate component unaffected by either issue.

Why

HyperShift (AROSLSRE-919): PR #5170 has been failing e2e for 2+ weeks. Root cause: the new HyperShift operator image (commit d24af10) includes be263214 which adds a webhook validation that rejects Azure HostedClusters when service hostnames shadow the cluster base domain.

Kusto logs show the klusterlet-agent repeatedly failing with:

HostedCluster.hypershift.openshift.io is invalid: spec: Invalid value:
Azure service hostname domain must not overlap with the cluster base domain

This blocks all cluster creation (43/74 tests fail with timeout). Meanwhile, PR #5368 (code-only, same base SHA) passes e2e — confirming the environment is healthy and the issue is the image bump.

Clusters Service (AROSLSRE-944): The CS bump PR #5348 (commit 84b200b) includes swift-nic annotation changes (ARO-27209) that require HyperShift CPO overrides (openshift/hypershift#8552) not yet backported and deployed. Without the CPO fix, Kubernetes rejects router pods because limits are required for non-overcommittable resources (aro.openshift.io/swift-nic).

Testing

make verify-yamlfmt — passes
make -C config detect-change — passes (no drift)
make -C acm helm-charts — ACM charts regenerated from new bundle digests
Hypershift operator digest verified unchanged from main
Clusters Service digest verified unchanged from main

Special notes for your reviewer

The hypershift operator image is pinned — unpin (back to tag: "latest") once the domain shadowing validation is fixed. Tracked in AROSLSRE-921.
The clusters-service image is pinned — unpin (back to tag: "latest") once HyperShift CPO swift-nic overrides are backported to 4.20+ and deployed to CSPR. Tracked in AROSLSRE-946.
This PR supersedes Automated - Update component image digests #5170 for all non-pinned image bumps.

Copilot

Pull request overview

This PR updates ARO-HCP component image digests while preventing the automated image bumper from advancing the HyperShift operator image to a known-bad “latest” build (domain shadowing validation regression), by pinning the HyperShift source tag in the image-updater configuration.

Changes:

Pin tooling/image-updater/config.yaml HyperShift source.tag to a specific commit SHA to stop auto-bumps from latest.
Bump multiple component image digests in config/config.yaml and regenerate the rendered dev WestUS3 configs.
Refresh ACM/MCE helm chart artifacts/fixtures to align with updated ACM bundle digests and operand image digests.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tooling/image-updater/config.yaml	Pins HyperShift image-updater tag to avoid the broken `latest` image selection.
config/config.yaml	Updates default component image digests (non-HyperShift) used across deployments.
config/rendered/dev/prow/westus3.yaml	Regenerated rendered config with updated digests for prow dev env.
config/rendered/dev/pers/westus3.yaml	Regenerated rendered config with updated digests for pers dev env.
config/rendered/dev/perf/westus3.yaml	Regenerated rendered config with updated digests for perf dev env.
config/rendered/dev/dev/westus3.yaml	Regenerated rendered config with updated digests for dev env.
config/rendered/dev/cspr/westus3.yaml	Regenerated rendered config with updated digests for cspr dev env.
config/rendered/dev/ci01/westus3.yaml	Regenerated rendered config with updated digests for ci01 dev env.
acm/zz_fixture_TestHelmTemplate_dev_westus3_mgmt_1_mce.yaml	Updated generated helm template fixture output.
acm/zz_fixture_TestHelmTemplate_dev_westus3_mgmt_1_mce_crds.yaml	Updated generated CRD fixture output (primarily formatting/line wrapping).
acm/zz_fixture_TestHelmTemplate_dev_westus3_mgmt_1_mce_config.yaml	Updated generated policy/CRD fixture output (formatting + content alignment).
acm/deploy/helm/multicluster-engine/templates/multicluster-engine-operator.deployment.yaml	Updates operand image digests in the MCE operator deployment template.
acm/deploy/helm/multicluster-engine/Chart.yaml	Updates chart `sources` digest reference for the MCE bundle.
acm/deploy/helm/multicluster-engine-crds/templates/multiclusterengines.multicluster.openshift.io.customresourcedefinition.yaml	Updates CRD manifest formatting/content from refreshed bundle.
acm/deploy/helm/multicluster-engine-crds/Chart.yaml	Updates chart `sources` digest reference for the MCE bundle (CRDs chart).
acm/deploy/helm/multicluster-engine-config/charts/policy/values.yaml	Updates policy chart image override digests.
acm/deploy/helm/multicluster-engine-config/charts/policy/crds/policy.open-cluster-management.io_policysets.yaml	Updates CRD content/formatting (document start + wrapped descriptions).
acm/deploy/helm/multicluster-engine-config/charts/policy/crds/policy.open-cluster-management.io_policyautomations.yaml	Updates CRD content/formatting (document start + wrapped descriptions).
acm/deploy/helm/multicluster-engine-config/charts/policy/crds/policy.open-cluster-management.io_policies.yaml	Updates CRD content/formatting (document start + wrapped descriptions).
acm/deploy/helm/multicluster-engine-config/charts/policy/crds/policy.open-cluster-management.io_placementbindings.yaml	Updates CRD content/formatting (document start + wrapped descriptions).
acm/deploy/helm/multicluster-engine-config/charts/policy/crds/apps.open-cluster-management.io_placementrules_crd_v1.yaml	Updates CRD content/formatting (wrapped descriptions).
acm/deploy/helm/multicluster-engine-config/charts/policy/crds/agent.open-cluster-management.io_klusterletaddonconfigs_crd.yaml	Adds YAML doc start and adjusts wrapped description formatting.
acm/deploy/helm/multicluster-engine-config/charts/policy/charts/grc/templates/grc-policy-addon-role.yaml	Minor formatting-only change (added blank line).
acm/deploy/helm/multicluster-engine-config/charts/policy/charts/grc/templates/grc-policy-addon-clusterrole.yaml	Minor formatting-only change (added blank line).
acm/deploy/helm/multicluster-engine-config/charts/policy/charts/cluster-lifecycle/templates/klusterlet-addon-role.yaml	Formatting changes to RBAC manifest (including trailing whitespace that needs fixing).
acm/deploy/helm/multicluster-engine-config/charts/policy/charts/cluster-lifecycle/templates/klusterlet-addon-role_binding.yaml	Minor formatting-only change (added blank line).

raelga · 2026-05-23T12:47:44Z

/retest

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

raelga · 2026-05-23T12:55:46Z

/lgtm
/label lgtm

This PR contains only image digest bumps copied from the automated bumper PR #5170 (by aro-hcp-robot), which is auto-approved. No code changes, no new features — just updated container image SHAs and regenerated ACM helm charts + config manifests.

The only non-digest change is pinning the HyperShift operator tag in tooling/image-updater/config.yaml to avoid the broken latest image (AROSLSRE-919).

openshift-ci · 2026-05-23T12:55:48Z

@raelga: you cannot LGTM your own PR.

Details

In response to this:

/lgtm

This PR contains only image digest bumps copied from the automated bumper PR #5170 (by aro-hcp-robot), which is auto-approved. No code changes, no new features — just updated container image SHAs and regenerated ACM helm charts + config manifests.

The only non-digest change is pinning the HyperShift operator tag in tooling/image-updater/config.yaml to avoid the broken latest image (AROSLSRE-919).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

raelga · 2026-05-23T13:12:43Z

/test ci/prow/e2e-parallel

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

raelga · 2026-05-23T13:18:54Z

/label lgtm

This PR contains only image digest bumps copied from the automated bumper PR #5170 (by aro-hcp-robot), which is auto-approved. No code changes, no new features — just updated container image SHAs and regenerated ACM helm charts + config manifests.

The only non-digest change is pinning the HyperShift operator tag in tooling/image-updater/config.yaml to avoid the broken latest image (AROSLSRE-919).

openshift-ci · 2026-05-23T13:18:57Z

@raelga: The label(s) lgtm cannot be applied or removed, because you are not in one of the allowed teams and are not an allowed user. Consider assigning one of the following members: deads2k,stevekuznetsov

Details

In response to this:

/label lgtm

This PR contains only image digest bumps copied from the automated bumper PR #5170 (by aro-hcp-robot), which is auto-approved. No code changes, no new features — just updated container image SHAs and regenerated ACM helm charts + config manifests.

The only non-digest change is pinning the HyperShift operator tag in tooling/image-updater/config.yaml to avoid the broken latest image (AROSLSRE-919).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

inbharajmani · 2026-05-26T07:35:51Z

/lgtm

openshift-merge-bot · 2026-05-26T08:58:33Z

/retest-required

Remaining retests: 0 against base HEAD 7fb96f7 and 2 for PR HEAD 36b278b in total

openshift-merge-bot · 2026-05-26T12:58:07Z

/retest-required

Remaining retests: 0 against base HEAD 3fa57ce and 1 for PR HEAD 36b278b in total

openshift-merge-bot · 2026-05-26T17:58:21Z

/retest-required

Remaining retests: 0 against base HEAD bb7c0d4 and 0 for PR HEAD 36b278b in total

…on (AROSLSRE-919) The automated image bumper PR #5170 has been failing e2e for 2+ weeks. The new HyperShift image (d24af10) includes commit be263214 which adds a webhook that rejects Azure HostedClusters when service hostnames shadow the cluster base domain, breaking all cluster creation. This PR applies all image bumps from PR #5170 except the HyperShift operator and shared-ingress images, which remain pinned to the last known working versions. The image-updater config is also updated to pin the hypershift tag to prevent future auto-bumps past the broken version.

…-944) The latest CS image (84b200b) includes swift-nic annotation changes that require HyperShift CPO overrides not yet deployed, breaking cluster creation for 4.23+ and 5.0. Pin CS tag to dbb022a (last known working version) in the image-updater config to prevent auto-bumps past the broken version.

The CS digest in config.yaml was bumped to b8a87db (from PR #5170 image bumps) while the image-updater tag was pinned to dbb022a. This mismatch would cause the image-updater to revert the digest on next run. Align both to dbb022a (the last known working CS version).

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Update all non-pinned image digests to latest from the automated bumper PR #5170. HyperShift operator and Clusters Service remain pinned to the versions currently on main to avoid the domain shadowing rejection (AROSLSRE-919) and swift-nic breakage (AROSLSRE-944) respectively.

sclarkso · 2026-05-27T02:03:08Z

/lgtm

openshift-ci · 2026-05-27T02:03:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inbharajmani, raelga, sclarkso

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~acm/OWNERS~~ [raelga,sclarkso]
~~config/OWNERS~~ [raelga,sclarkso]
~~tooling/image-updater/OWNERS~~ [raelga]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot AI review requested due to automatic review settings May 23, 2026 11:09

openshift-ci Bot requested review from janboll and tony-schndr May 23, 2026 11:09

openshift-ci Bot added the approved label May 23, 2026

Copilot started reviewing on behalf of raelga May 23, 2026 11:09 View session

Copilot AI reviewed May 23, 2026

View reviewed changes

Comment thread ...er-engine-config/charts/policy/charts/cluster-lifecycle/templates/klusterlet-addon-role.yaml Outdated

Comment thread acm/deploy/helm/multicluster-engine/templates/multicluster-engine-operator.deployment.yaml Outdated

raelga force-pushed the fix/pin-hypershift-bump-images branch from 9a1aa99 to 81abe21 Compare May 23, 2026 12:20

Copilot started work on behalf of raelga May 23, 2026 12:24 View session

Copilot finished work on behalf of raelga May 23, 2026 12:25

Copilot AI review requested due to automatic review settings May 23, 2026 12:50

raelga force-pushed the fix/pin-hypershift-bump-images branch from 81abe21 to 73d75e7 Compare May 23, 2026 12:50

Copilot started reviewing on behalf of raelga May 23, 2026 12:50 View session

Copilot AI reviewed May 23, 2026

View reviewed changes

Comment thread tooling/image-updater/config.yaml

Copilot started work on behalf of raelga May 23, 2026 12:54 View session

raelga added the lgtm label May 23, 2026

Copilot finished work on behalf of raelga May 23, 2026 12:56

raelga requested a review from Copilot May 23, 2026 13:12

Copilot started reviewing on behalf of raelga May 23, 2026 13:13 View session

Copilot AI reviewed May 23, 2026

View reviewed changes

Comment thread tooling/image-updater/config.yaml Outdated

raelga force-pushed the fix/pin-hypershift-bump-images branch from 73d75e7 to 4384867 Compare May 23, 2026 13:18

openshift-ci Bot removed the lgtm label May 23, 2026

Copilot started work on behalf of raelga May 23, 2026 13:18 View session

openshift-ci Bot assigned inbharajmani May 26, 2026

openshift-ci Bot added the lgtm label May 26, 2026

openshift-ci Bot added the needs-rebase label May 26, 2026

raelga added 3 commits May 26, 2026 22:09

fix: apply yamlfmt to ACM helm charts

68a3ed1

raelga force-pushed the fix/pin-hypershift-bump-images branch from 36b278b to 677342f Compare May 26, 2026 20:12

openshift-ci Bot removed lgtm needs-rebase labels May 26, 2026

Copilot AI review requested due to automatic review settings May 26, 2026 22:16

Copilot started reviewing on behalf of raelga May 26, 2026 22:16 View session

Copilot AI reviewed May 26, 2026

View reviewed changes

Comment thread config/config.yaml

Comment thread config/config.yaml

Comment thread config/config.yaml

raelga force-pushed the fix/pin-hypershift-bump-images branch from 6c2d315 to 259491c Compare May 26, 2026 22:53

Copilot AI review requested due to automatic review settings May 26, 2026 23:03

raelga force-pushed the fix/pin-hypershift-bump-images branch from 259491c to a267a86 Compare May 26, 2026 23:03

Copilot started reviewing on behalf of raelga May 26, 2026 23:04 View session

Copilot AI reviewed May 26, 2026

View reviewed changes

Comment thread config/config.yaml

Comment thread config/config.yaml

raelga force-pushed the fix/pin-hypershift-bump-images branch from a267a86 to 9bfee61 Compare May 26, 2026 23:15

openshift-ci Bot added the lgtm label May 27, 2026

openshift-merge-bot Bot merged commit f6e1daa into main May 27, 2026
15 checks passed

openshift-merge-bot Bot deleted the fix/pin-hypershift-bump-images branch May 27, 2026 05:21

hbhushan3 mentioned this pull request May 27, 2026

fix(image-updater): use short-SHA tag for clusters-service pin (AROSLSRE-954) #5418

Closed

Conversation

raelga commented May 23, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Testing

Special notes for your reviewer

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

raelga commented May 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

raelga commented May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented May 23, 2026

Uh oh!

raelga commented May 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

raelga commented May 23, 2026

Uh oh!

openshift-ci Bot commented May 23, 2026

Uh oh!

inbharajmani commented May 26, 2026

Uh oh!

openshift-merge-bot Bot commented May 26, 2026

Uh oh!

openshift-merge-bot Bot commented May 26, 2026

Uh oh!

openshift-merge-bot Bot commented May 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

sclarkso commented May 27, 2026

Uh oh!

openshift-ci Bot commented May 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

raelga commented May 23, 2026 •

edited by openshift-ci Bot

Loading

raelga commented May 23, 2026 •

edited

Loading