Skip to content

fix(vpn): retry cert rotation when job fails instead of dropping it#392

Open
Ady0333 wants to merge 1 commit into
kubeslice:masterfrom
Ady0333:fix/vpn-cert-rotation-job-retry
Open

fix(vpn): retry cert rotation when job fails instead of dropping it#392
Ady0333 wants to merge 1 commit into
kubeslice:masterfrom
Ady0333:fix/vpn-cert-rotation-job-retry

Conversation

@Ady0333

@Ady0333 Ady0333 commented May 18, 2026

Copy link
Copy Markdown

Description

When a VPN certificate-rotation Job enters a Failed or Suspended state,
reconcileVpnKeyRotationConfig returned ctrl.Result{} with a nil error. Back in
ReconcileVpnKeyRotation, this produced return ctrl.Result{}, nil controller-runtime treats that as
success and removes the VpnKeyRotation object from the work queue permanently.

Because the VpnKeyRotation controller registers only For(&VpnKeyRotation{}) (no
Owns(&batchv1.Job{})), Job status changes generate no reconcile events. The object would only ever
be reconciled again on a spec change or a controller restart. Meanwhile the existing certificates
march toward CertificateExpiryTime with no replacement once they expire, the slice's gateway-pair
TLS handshakes fail and all cross-cluster traffic on that slice blackouts, with no self-healing.

The jobCreationInProgress flag was also never reset on the failure path, so even a forced reconcile
would skip triggerJobsForCertCreation and never recreate the failed Job.

This is realistic: Kubernetes Jobs fail routinely (image pull errors, transient API failures,
ResourceQuota rejection, node pressure, backoffLimit exhaustion). It is a silent failure no error, no requeue, no stuck-loop to alert on.

Fix: In both JobStatusError || JobStatusSuspended branches of reconcileVpnKeyRotationConfig, reset
jobCreationInProgress and return ctrl.Result{RequeueAfter: 30 * time.Second} (matching the cadence
already used for JobStatusRunning/JobNotCreated). The reconciler now retries rotation and recreates
the failed Job, self-healing once the underlying cause clears.


How Has This Been Tested?

Added Test_VpnCertRotationJobFailure_NoRequeue in service/vpn_cert_rotation_job_failure_bug_test.go. The test was first written to confirm the bug (asserting RequeueAfter == 0 and
jobCreationInProgress stuck true), then updated to verify the fix.

go test ./service/ -run Test_VpnCertRotationJobFailure_NoRequeue -v
--- PASS: Test_VpnCertRotationJobFailure_NoRequeue
ok  github.com/kubeslice/kubeslice-controller/service

Checklist:

  • The title of the PR states what changed and the related issues number (used for the release note).
  • Does this PR requires documentation updates?
  • I've updated documentation as required by this PR.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have tested it for all user roles.
  • I have added all the required unit test cases.

Does this PR introduce a breaking change for other components like worker-operator?

No. The change only affects the failure-handling path of the VpnKeyRotation reconciler successful
rotation behavior is unchanged. No API, CRD, or worker-facing contract changes.


A failed or suspended cert-rotation Job caused the reconciler to return
ctrl.Result{} with no error, so the work queue forgot the VpnKeyRotation
permanently and certs expired with no recovery. Requeue after 30s and
reset jobCreationInProgress so the job is recreated.

Signed-off-by: Ady0333 <adityashinde1525@gmail.com>
@Ady0333 Ady0333 requested a review from that-backend-guy as a code owner May 18, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant