Skip to content

UI proxy: migrate ring 2 clusters and remove base-ring2 (production ring 2) - #1

Closed
mshaposhnik wants to merge 83 commits into
ui-proxy-token-refactor-production-ring1from
ui-proxy-token-refactor-production-ring2
Closed

UI proxy: migrate ring 2 clusters and remove base-ring2 (production ring 2)#1
mshaposhnik wants to merge 83 commits into
ui-proxy-token-refactor-production-ring1from
ui-proxy-token-refactor-production-ring2

Conversation

@mshaposhnik

Copy link
Copy Markdown
Owner

Summary

Ring 2 of 2 — migrates the remaining 3 production clusters to the new short-lived token hot-reload mechanism and cleans up the temporary base-ring2.

Depends on: redhat-appstudio#11800 (ring 1) — merge ring 1 first, validate, then merge this into main.

Ring 2 clusters (this PR)

stone-prd-rh01, stone-prod-p02, kflux-ocp-p01

Changes

  • Switch ring 2 cluster overlays from base-ring2 back to base
  • Fix container index in overlay patches for ring 2 clusters
    • oauth2-proxy-args-patch.yaml: containers/2 → containers/3
    • remove-run-as-user-proxy-patch.yaml: add containers/4
  • Delete the temporary base-ring2 directory

Test plan

  • Verify ring 1 clusters are healthy after ring 1 merge
  • Rebase this branch onto main after ring 1 merges
  • Create PR against redhat-appstudio/infra-deployments main
  • Verify ring 2 clusters pick up the change and rotate tokens successfully

Made with Cursor

@github-actions

Copy link
Copy Markdown

🤖 Gemini AI Assistant Available

Hi @mshaposhnik! I'm here to help with your pull request. You can interact with me using the following commands:

Available Commands

  • @gemini-cli /review - Request a comprehensive code review

    • Example: @gemini-cli /review Please focus on security and performance
  • @gemini-cli <your question> - Ask me anything about the codebase

    • Example: @gemini-cli How can I improve this function?
    • Example: @gemini-cli What are the best practices for error handling here?

How to Use

  1. Simply type one of the commands above in a comment on this PR
  2. I'll analyze your code and provide detailed feedback
  3. You can track my progress in the workflow logs

Permissions

Only OWNER, MEMBER, or COLLABORATOR users can trigger my responses. This ensures secure and appropriate usage.


This message was automatically added to help you get started with the Gemini AI assistant. Feel free to delete this comment if you don't need assistance.

@github-actions

Copy link
Copy Markdown

🤖 Hi @mshaposhnik, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

rh-tap-build-team Bot and others added 28 commits May 14, 2026 10:32
…ppstudio#11798)

Co-authored-by: rh-tap-build-team[bot] <127938674+rh-tap-build-team[bot]@users.noreply.github.com>
It's been baking in staging for a few weeks, and no issues have been
identified.  Synchronize the production policy with the staging policy.

One small change to the policy is required, however: production has not
upgraded to the latest version of kueue, so the v1beta1 version of
kueue's resources is required by this policy.

Fixes: KFLUXINFRA-3646

Signed-off-by: Andy Sadler <ansadler@redhat.com>
Some further maintenance items were identified following last week's
maintenance, and some follow-up maintenance is required.

All production clusters except for the fedora cluster will be impacted
by this maintenance.

Inform users that this maintenance will be occurring and that their
pipelineruns may be held for a short time.

Signed-off-by: Andy Sadler <ansadler@redhat.com>
…pstudio#11776)

* update opentelemetry-collector for KubeArchive stage config

KONFLUX-12789

KONFLUX-12711

KONFLUX-12789

KONFLUX-12711

Signed-off-by: obetsun <obetsun@redhat.com>

* fix linter errors

Signed-off-by: obetsun <obetsun@redhat.com>

* refactor otel-collector patch

Signed-off-by: obetsun <obetsun@redhat.com>

* fix kustomization files

Signed-off-by: obetsun <obetsun@redhat.com>

* fix linter errors

Signed-off-by: obetsun <obetsun@redhat.com>

---------

Signed-off-by: obetsun <obetsun@redhat.com>
…t-appstudio#11802)

Co-authored-by: rh-tap-build-team[bot] <127938674+rh-tap-build-team[bot]@users.noreply.github.com>
Add a new component to deploy the k8s groups located in the
internal-infra-deployments repository to the tenant Konflux clusters.

KFLUXINFRA-3596
In order to fix this, we need to apply the patch from
k-components/deploy-to-all-clusters. This is already done in
argo-cd-apps/base/all-clusters/kustomization.yaml so fix this by moving
k8s-groups within that folder. Since this is first component deploy from
another repo than infra-deployments, create all the basic patches for
this new repo.

KFLUXINFRA-3596

Signed-off-by: Hugo Ares <hares@redhat.com>
Signed-off-by: Robert Cerven <rcerven@redhat.com>
* add tokens in tekton-kueue

This proposal adds the new ephemeral resource `konflux-ci.dev/token` in
tekton-kueue. This is to weight PipelineRuns with respect to known
patterns in tekton-kueue and to better balance them.

As an example, we noticed that Build PipelineRuns are usually heavier
than Non-Build PipelineRuns, so we can identify Build PipelineRuns and
mark them to require 2 tokens. The other PipelineRuns will only take 1
instead.
Assuming both `tekton.dev/pipelineruns` and `konflux-ci.dev/token` are
600, we could have in execution:
* 300 Build PipelineRuns
* 200 Build PipelineRuns + 200 Non-Build PipelineRuns
* 100 Build PipelineRuns + 400 Non-Build PipelineRuns
* 600 Non-Build PipelineRuns

Signed-off-by: Francesco Ilario <filario@redhat.com>

* fix linters

Signed-off-by: Francesco Ilario <filario@redhat.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* update tests

Signed-off-by: Francesco Ilario <filario@redhat.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix tests

Signed-off-by: Francesco Ilario <filario@redhat.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix tests

Signed-off-by: Francesco Ilario <filario@redhat.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* remove duplicate test

Signed-off-by: Francesco Ilario <filario@redhat.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

---------

Signed-off-by: Francesco Ilario <filario@redhat.com>
Fedora should not use rover groups, use FAS ones.

Signed-off-by: Hugo Ares <hares@redhat.com>
redhat-appstudio#11803)

* KONFLUX-13356: add ring 1 tekton-kueue base pinned to cec0b3c8

Copy of production base tekton-kueue with resources ref and image tag
updated to cec0b3c8. This temporary directory will be removed when
ring 2 upgrades and the shared base is updated.

Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: Gal Levi <glevi@redhat.com>

* KONFLUX-13356: upgrade kueue operator and tekton-kueue for prod ring 1

Upgrade kueue operator from stable-v1.2 to stable-v1.3 and tekton-kueue
to cec0b3c8 for ring 1 production clusters: stone-prod-p01,
kflux-ocp-p01, kflux-prd-rh02.

- Ring 1 clusters reference base components individually, using
  base-ring1-tekton-kueue instead of the shared base tekton-kueue.
- Per-cluster kustomize patch overrides the Subscription channel from
  stable-v1.2 to stable-v1.3.
- ClusterQueue specs updated: whenCanBorrow Borrow to MayStopSearch,
  removed stopPolicy None to prevent ArgoCD drift from the v1beta2
  conversion webhook.

Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: Gal Levi <glevi@redhat.com>

---------

Signed-off-by: Gal Levi <glevi@redhat.com>
Tekton-Kueue sanitizes and normalizes the resource names as it
needs to use them in an annotation's key.
The normalized name needs to be used in the ClusterQueue.

Signed-off-by: Francesco Ilario <filario@redhat.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED
…aging) (redhat-appstudio#11782)

* KFLUXINFRA-3786: revoke Role write permissions from tenant admins (staging)

Split the combined roles+rolebindings RBAC rule in konflux-admin-user-actions
so that roles is read-only (get/list) while rolebindings retains full CRUD.

Co-authored-by: Cursor <cursoragent@cursor.com>

* KFLUXINFRA-3786: revoke Role write permissions from tenant admins (production)

Split the combined roles+rolebindings RBAC rule in konflux-admin-user-actions
so that roles is read-only (get/list) while rolebindings retains full CRUD.

Co-authored-by: Cursor <cursoragent@cursor.com>

* KFLUXINFRA-3786: add watch verb for roles read-only rule

Co-authored-by: Cursor <cursoragent@cursor.com>

* KFLUXINFRA-3786: revert production change, staging only for now

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
…d perfscale-3 tenants in staging (redhat-appstudio#11875)

Add konflux-perfscale-2-tenant and konflux-perfscale-3-tenant namespaces
to the appsre-stonesoup-vault ClusterSecretStore in staging only.
Production will follow in a separate PR.

Generated-by: Claude
…ppstudio#11842)

Migrate all kueue custom resources from v1beta1 to v1beta2 for ring 1
production clusters: stone-prod-p01, kflux-ocp-p01, kflux-prd-rh02.

- ClusterQueue, ResourceFlavor apiVersion v1beta1 to v1beta2
- Add base-ring1-queue-config with v1beta2 WorkloadPriorityClasses,
  ring 1 clusters reference this instead of base/queue-config
- OCP-specific WorkloadPriorityClasses v1beta1 to v1beta2
- nominalQuota 1000 to 1k (canonical resource.Quantity form)

Assisted-by: Claude Code <noreply@anthropic.com>

Signed-off-by: Gal Levi <glevi@redhat.com>
…d perfscale-3 tenants in prod (redhat-appstudio#11883)

Add konflux-perfscale-2-tenant and konflux-perfscale-3-tenant namespaces
to the appsre-stonesoup-vault ClusterSecretStore in production via
overlay patch, matching the existing staging configuration.

Generated-by: Claude
…hat-appstudio#11846)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…n.yaml (redhat-appstudio#11893)

Co-authored-by: rh-tap-build-team[bot] <127938674+rh-tap-build-team[bot]@users.noreply.github.com>
…hat-appstudio#11887)

Adding a placeholder entrypoint that will used by OpenShift CI to
trigger e2e tests the Konflux Operator.

Assisted-by: Cursor

Signed-off-by: Yftach Herzog <yherzog@redhat.com>
…ton-kueue (redhat-appstudio#11901)

Required by the TektonKueueRolloutStuck alert to detect stuck
rollouts where updated replicas are fewer than desired.

Co-authored-by: Asaf Aviv <asaviv@asaviv-thinkpadp1gen7.raanaii.csb>
…ppstudio#11908)

Co-authored-by: rh-tap-build-team[bot] <127938674+rh-tap-build-team[bot]@users.noreply.github.com>
jhutar and others added 28 commits May 26, 2026 11:22
…hat-appstudio#11984)

Backport tenants-rbac from staging to production, granting konflux-bot-0
permission to read events in konflux-perfscale-{1,2,3}-tenant namespaces.

Remove the namespace directive from production kustomization since all
resources already specify their namespaces explicitly, and it would
conflict with the per-tenant namespaces in tenants-rbac.


Generated-by: Claude
Assisted-by: Claude

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…etheus-reader (redhat-appstudio#11983)

Add Role and RoleBinding granting the konflux-performance group permission
to create tokens for perf-team-prometheus-reader-cluster-sa and
perf-team-prometheus-reader-oomcrash-sa service accounts. Added to base/core
which is used by development and staging overlays only (production has its
own standalone resources).


Generated-by: Claude
Assisted-by: Claude

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…o 100GB in staging (redhat-appstudio#11940)

With redirect caching enabled, nginx now caches actual artifact content
from S3 rather than passing through 302 redirects. Increase the cache volume
from 50GB to 100GB to accommodate the larger working set. Production will be
increased to 1TB in a separate PR.

Note: This change requires manual intervention after ArgoCD sync fails.
See KFLUXVNGD-994 for the resize procedure.


Jira-Url: https://redhat.atlassian.net/browse/KFLUXVNGD-994

Signed-off-by: Homaja Marisetty <hmariset@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…oduction. (redhat-appstudio#11974)

Signed-off-by: Gabriel Soares <gasoares@redhat.com>
We are done with testing in us-south, go back to us-east to be able to
clean up IBMC resources in us-south.

Also clean up ssh key ExternalSecrets, since we no longer have 2 for ppc
we can use standard name for the one left.

KFLUXINFRA-3315

Signed-off-by: Hugo Ares <hares@redhat.com>
Signed-off-by: Marta Anon <manon@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…3 and consolidate base (redhat-appstudio#12013)

* KONFLUX-13356: remove temporary base-ring1-tekton-kueue directory

Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: Gal Levi <glevi@redhat.com>

* KONFLUX-13356: revert rings 1/2 to shared base and fix ring 3 ClusterQueue drift

- Revert rings 1/2 kustomizations to ../base, remove per-cluster
  Subscription patches and base-ring1-tekton-kueue references
- Fix ring 3 ClusterQueue drift: whenCanBorrow Borrow → MayStopSearch,
  remove stopPolicy

Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: Gal Levi <glevi@redhat.com>

* KONFLUX-13356: update shared kueue base to stable-v1.3 and tekton-kueue cec0b3c8

- Subscription channel stable-v1.2 → stable-v1.3
- tekton-kueue ref and image 964790ee → cec0b3c8

Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: Gal Levi <glevi@redhat.com>

---------

Signed-off-by: Gal Levi <glevi@redhat.com>
…dhat-appstudio#12012)

Backport sa-token-rbac from staging (cdcaa5f) to production, granting
the konflux-performance group permission to create tokens for
perf-team-prometheus-reader-cluster-sa and
perf-team-prometheus-reader-oomcrash-sa service accounts.


Generated-by: Claude
Assisted-by: Claude

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: release-service <release-service@redhat.com>
…hat-appstudio#11975)

Replace appstudio-utils image reference in Konflux UI resources by
task-runner image in prod environment.

Jira KONFLUX-13284
…12016)

* Add docs for perf-team-prometheus-reader component

Rewrite README.md with directory layout, promotion workflow (soak in
staging then copy-paste to production), expanded SA descriptions with
RBAC grants, production PR template, and link to repo-level conventions.

Add AGENTS.md with kustomize build commands and production isolation
warning.

Add operational hints as YAML comments to SA files (token generation
commands and vault path for storing secrets).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Generated-by: Claude
Assisted-by: Claude

* docs: Adding links, small manual cleanup, serviceaccount-oomcrash token is not stored in Vault

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…at-appstudio#12009)

Update the OpenShift Pipelines OLM catalog index image in dev and staging environments.

v5.0.5-819 → v5.0.5-830

Key changes:
- PAC: v0.46.0 → v0.47.0 (label fixes, CEL extensions, Bitbucket/GitLab fixes)
- Pipeline: release-v1.12.x HEAD (+18 commits, includes generateName fix)
- Chains: CVE-2026-33814 fix (golang.org/x/net HTTP/2)
- CLI: v0.44.1 → v0.45.0
- Pruner: v0.3.5 → v0.4.0

Image: quay.io/openshift-pipeline/pipelines-index-4.18@sha256:a4aa211a70b0a92c98c885cca8fed2cce490e3fec656ad1ea68bf7e7c3c18304
…edhat-appstudio#12023)

* KONFLUX-13356: delete temporary base-ring1-queue-config directory

Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: Gal Levi <glevi@redhat.com>

* KONFLUX-13356: revert ring 1/2 queue-config references to shared base

Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: Gal Levi <glevi@redhat.com>

* KONFLUX-13356: migrate CRs to v1beta2 (ring 3 + shared base)

Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: Gal Levi <glevi@redhat.com>

---------

Signed-off-by: Gal Levi <glevi@redhat.com>
Co-authored-by: release-service <release-service@redhat.com>
Finalize the development overlay CR for the Konflux Operator.

Also drop the appstudio-pipelines-scc from the pipeline-service for
the development-operator overlay, as it is deployed by the operator.

This allows deploying Konflux using the operator in dev environments.

Assisted-by: Cursor

Signed-off-by: Yftach Herzog <yherzog@redhat.com>
* update components/mintmaker/development/kustomization.yaml

* update components/mintmaker/staging/base/kustomization.yaml

---------

Co-authored-by: rh-tap-build-team[bot] <127938674+rh-tap-build-team[bot]@users.noreply.github.com>
…roduction ring 1) (redhat-appstudio#11800)

Ring 1 clusters: kflux-fedora-01, kflux-osp-p01, kflux-prd-rh02,
kflux-prd-rh03, kflux-rhel-p01, stone-prod-p01

Ring 2 clusters (stone-prd-rh01, stone-prod-p02, kflux-ocp-p01)
temporarily point to base-ring2 which preserves the old layout,
to be migrated in a follow-up PR.

Changes:
- Add generate-nginx-configs-loop sidecar with liveness probe
- Replace direct nginx command with bash wrapper for hot-reload
- Replace long-lived SA token secret with projected token (600s TTL)
- Add script ConfigMaps for proxy-nginx-run and proxy-nginx-generate-loop
- Port bug fixes from konflux-ci/konflux-ci#6780
- Fix container index in overlay patches for ring 1 clusters
- Update hack/new-cluster templates

Signed-off-by: Max Shaposhnyk <mshaposh@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
)

Split MIGRATION_VERSION into two configmaps following the staging/p02
pattern: kubearchive-schema-version (for the migration Job) and
kubearchive-deployment-schema-version (for deployments). This ensures
deployments pick up the new schema version only after the migration
Job completes via ArgoCD sync waves.

Signed-off-by: Marta Anon <manon@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…pstudio#12018)

Add per-cluster overlays for ring 1 production clusters: kflux-ocp-p01,
kflux-prd-rh02, stone-prod-p01. Each overlay is self-contained with its
own helm generator and credentials. Existing flat production files are
kept for unmigrated clusters. Rings 2 and 3 will follow in subsequent PRs.

Assisted-by: Claude Code
…pstudio#12050)

Add per-cluster overlays for ring 2 production clusters: kflux-osp-p01,
kflux-prd-rh03, stone-prod-p02. Flat production files kept for remaining
3 unmigrated clusters.

Assisted-by: Claude Code
…appstudio#12051)

Add per-cluster overlays for kflux-fedora-01, kflux-rhel-p01, and
stone-prd-rh01. Add empty-base fallback directory and squid-specific
production patch. Remove flat production files.

Assisted-by: Claude Code
Co-authored-by: release-service <release-service@redhat.com>
…n clusters (redhat-appstudio#12052)

With redirect caching enabled (KFLUXVNGD-997), nginx caches actual
artifact content from S3 rather than passing through 302 redirects.
This increases cache disk usage significantly. Bump nginx.cache.size
from 51200 (50GB) to 1048576 (1TB) and update chart version to
0.1.1521+3e9d8b9 which includes the int rendering fix for large values.

Clusters: kflux-ocp-p01, kflux-prd-rh02, stone-prod-p01

Assisted-by: Claude Code
If the PR has the "skip-ring-deployment/hotfix" label applied, the
"enforce-ring-deployments" Github Action will automatically pass.
This is to avoid blocking critical hotfixes that need to reach both
staging and production environments as quickly as possible.

Assisted-by: Cursor
…ing 2)

Switch stone-prd-rh01, stone-prod-p02, kflux-ocp-p01 from base-ring2
back to the updated base with short-lived token hot-reload.

- Fix container index in overlay patches for ring 2 clusters
- Delete the temporary base-ring2 directory

Signed-off-by: Max Shaposhnyk <mshaposh@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@mshaposhnik
mshaposhnik force-pushed the ui-proxy-token-refactor-production-ring2 branch from 66d6650 to 501093c Compare May 29, 2026 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.