Skip to content

fix(): Missing cluster silently skips gateway creation, leaving slice with no inter-cluster connectivity#390

Open
Ady0333 wants to merge 2 commits into
kubeslice:masterfrom
Ady0333:fix/gateway-missing-cluster-error
Open

fix(): Missing cluster silently skips gateway creation, leaving slice with no inter-cluster connectivity#390
Ady0333 wants to merge 2 commits into
kubeslice:masterfrom
Ady0333:fix/gateway-missing-cluster-error

Conversation

@Ady0333

@Ady0333 Ady0333 commented May 18, 2026

Copy link
Copy Markdown

Description

createMinimumGatewaysIfNotExists() queries each cluster referenced in SliceConfig.Spec.Clusters
to build the gateway mesh. When a Cluster CR is absent, util.GetResourceIfExist() returns
(false, nil). The original code checked if !found || err != nil { return ctrl.Result{}, err }
when !found==true and err==nil, this returns nil to the work queue.

Before (broken):

found, err := util.GetResourceIfExist(ctx, client.ObjectKey{...}, &cluster)
if !found || err != nil {
    return ctrl.Result{}, err  // returns nil when cluster missing
}

The SliceConfig reconcile sees nil, treats it as success, and drops from the work queue. Meanwhile,
the earlier CreateMinimalWorkerSliceConfig step already created WorkerSliceConfig objects for all
clusters (it doesn't validate cluster existence). The slice is now in a ghost state: worker configs
exist, but the gateway mesh is never built zero cross-cluster connectivity.

  After (fixed):
  if err != nil {
      return ctrl.Result{}, err
  }
  if !found {
      return ctrl.Result{}, fmt.Errorf("cluster %s not found in namespace %s", clusterName, namespace)
  }                                                                                                     
  Now a missing cluster explicitly fails the reconcile, causing the work queue to requeue with
  exponential backoff until the Cluster CR is registered.

How Has This Been Tested?

  • Wrote a unit test (Test_ClusterNotFound_GatewayCreationSilentlySkipped) that mocks a missing
    Cluster CR during gateway creation
  • Before fix: function returns nil error ghost slice confirmed
  • After fix: function returns error "cluster ... not found" -- SliceConfig will be requeued

Checklist:

  • The title of the PR states what changed and the related issues number (used for the release note).
  • Does this PR requires documentation updates?
  • I've updated documentation as required by this PR.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have tested it for all user roles.
  • I have added all the required unit test cases.

Does this PR introduce a breaking change for other components like worker-operator?

No breaking changes. The fix makes error handling stricter only affects behavior when a referenced Cluster is missing (previously silent, now properly requeued).
-->


Ady0333 added 2 commits May 18, 2026 07:49
Make jobCreationInProgress per-slice instead of process-global to prevent
one slice's certificate rotation from blocking another's. Replace atomic.Bool
with sync.Map keyed by namespace/sliceName.

Also fixes pre-existing broken test type hints (util.Client -> client.Client).

Signed-off-by: Ady0333 <adityashinde1525@gmail.com>
Return an error when a referenced Cluster CR is not found, instead of
silently returning nil. This ensures the SliceConfig is requeued by the
work queue until the cluster is registered, preventing ghost slices with
no inter-cluster connectivity.

Signed-off-by: Ady0333 <adityashinde1525@gmail.com>
@Ady0333 Ady0333 requested a review from that-backend-guy as a code owner May 18, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant