feat(): POC Active/Standby HA foundation for the controller#391
Draft
Ady0333 wants to merge 1 commit into
Draft
Conversation
Add pkg/ha: lease-based leader election, a versioned state synchronizer with snapshot diffing, lease health monitoring, an HA manager orchestrating Active/Standby transitions, and an ActiveGuard reconcile wrapper that gates writes to the Active instance. Includes Prometheus metrics and 34 race-clean tests. Signed-off-by: Ady0333 <adityashinde1525@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The KubeSlice controller currently runs as a single replica, if that pod crashes, is evicted, or
its node fails, the entire multi-cluster control plane stalls (no SliceConfig reconciliation, no
gateway provisioning, no VPN key rotation) until the pod is rescheduled and its cache warms up cold.
This PR introduces the foundational layer for an Active/Standby topology, where multiple controller
replicas run concurrently: exactly one is Active and reconciles, while the others stay Standby
warm, observing state, and ready to take over within one lease period.
New package pkg/ha:
the add/remove/update SnapshotDiff vs. the prior revision, keeping a Standby warm.
configurable threshold of consecutive failures (debounces transient API errors); panic-safe for
leases missing optional fields.
transitions, triggers an immediate resync on promotion, and exports Prometheus metrics.
cluster state; controller-runtime's resync replays the work once the instance is promoted.
ResourceCollector / LeaseReader / ILeaderElection are narrow interfaces so the logic is fully
unit-testable without a live API server.
This is a self-contained, additive POC no existing controller wiring is modified.
How Has This Been Tested?
34 unit tests run under the race detector and go vet, exercising real synchronizer/health/failover
logic via in-memory ResourceCollector and LeaseReader implementations (no stubs).
Checklist:
Does this PR introduce a breaking change for other components like worker-operator?
No. The change is purely additive a new pkg/ha package and documentation, with no modifications to existing controllers, CRDs, or APIs.