Skip to content

feat(): POC Active/Standby HA foundation for the controller#391

Draft
Ady0333 wants to merge 1 commit into
kubeslice:masterfrom
Ady0333:feat/ha-controller-active-standby
Draft

feat(): POC Active/Standby HA foundation for the controller#391
Ady0333 wants to merge 1 commit into
kubeslice:masterfrom
Ady0333:feat/ha-controller-active-standby

Conversation

@Ady0333

@Ady0333 Ady0333 commented May 18, 2026

Copy link
Copy Markdown

Description

The KubeSlice controller currently runs as a single replica, if that pod crashes, is evicted, or
its node fails, the entire multi-cluster control plane stalls (no SliceConfig reconciliation, no
gateway provisioning, no VPN key rotation) until the pod is rescheduled and its cache warms up cold.

This PR introduces the foundational layer for an Active/Standby topology, where multiple controller
replicas run concurrently: exactly one is Active and reconciles, while the others stay Standby
warm, observing state, and ready to take over within one lease period.

New package pkg/ha:

  • LeaderElection :- coordination/v1 Lease-based election via client-go; releases the lease on context cancellation for fast failover; callback-driven transitions.
  • StateSynchronizer :- periodically collects KubeSlice CRDs into a revisioned Snapshot and computes
    the add/remove/update SnapshotDiff vs. the prior revision, keeping a Standby warm.
  • HealthChecker :- monitors the Active controller's lease renewal; flags it unhealthy only after a
    configurable threshold of consecutive failures (debounces transient API errors); panic-safe for
    leases missing optional fields.
  • HAManager :- orchestrates the three components, owns ControllerState, drives Active↔Standby
    transitions, triggers an immediate resync on promotion, and exports Prometheus metrics.
  • ActiveGuard :- a reconcile.Reconciler wrapper that drops requests on a Standby without mutating
    cluster state; controller-runtime's resync replays the work once the instance is promoted.

ResourceCollector / LeaseReader / ILeaderElection are narrow interfaces so the logic is fully
unit-testable without a live API server.

This is a self-contained, additive POC no existing controller wiring is modified.


How Has This Been Tested?

34 unit tests run under the race detector and go vet, exercising real synchronizer/health/failover
logic via in-memory ResourceCollector and LeaseReader implementations (no stubs).

  go test -race ./pkg/ha/...
  ok    github.com/kubeslice/kubeslice-controller/pkg/ha        1.201s   (34/34 pass)

  go vet ./pkg/ha/...  

Checklist:

  • The title of the PR states what changed and the related issues number (used for the release note).
  • Does this PR requires documentation updates?
  • I've updated documentation as required by this PR.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have tested it for all user roles.
  • I have added all the required unit test cases.

Does this PR introduce a breaking change for other components like worker-operator?

No. The change is purely additive a new pkg/ha package and documentation, with no modifications to existing controllers, CRDs, or APIs.


Add pkg/ha: lease-based leader election, a versioned state
synchronizer with snapshot diffing, lease health monitoring,
an HA manager orchestrating Active/Standby transitions, and an
ActiveGuard reconcile wrapper that gates writes to the Active
instance. Includes Prometheus metrics and 34 race-clean tests.

Signed-off-by: Ady0333 <adityashinde1525@gmail.com>
@Ady0333 Ady0333 requested a review from that-backend-guy as a code owner May 18, 2026 11:15
@Ady0333 Ady0333 marked this pull request as draft May 18, 2026 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant