feat(util): add LeaderGate runtime fence for Active/Standby HA mode

[Priyanshubhartistm]

Summary

First scaffolding PR for the Active/Standby HA project (#305, #294). No behaviour change in the default mode-that's the point.

what i did

Added a LeaderGate interface in util/leader_gate.go. Every mutating helper in util/reconciliation_utility.go and util/cleanup_utility.go now calls requireLeader(ctx) before touching the API. Covers CreateResource, UpdateResource, UpdateStatus, DeleteResource, AddFinalizer, RemoveFinalizer, plus the three cleanup variants.

Two implementations

• NoOpLeaderGate - always permits. This is the default.
• managerElectedGate (via NewManagerLeaderGate(mgr)) - gates on controller-runtime's mgr.Elected().

new flag

--ha-mode={disabled,active,standby}

default: disabled

• disabled: gate is NoOp. Identical to existing behaviour. Existing deployments don't have to do anything.
• active / standby: gate is manager-backed. Non-leader mutations return util.ErrNotLeader (wrapped).

If --ha-mode != disabled without --leader-elect=true, the process exits at startup. Felt safer than silently allowing a split-brain.

why funnel-level fencing instead of per-call-site

I chose to gate inside the util helpers, not at the call sites. Two reasons:

1. Any new reconciler added later gets fenced for free - no risk of someone forgetting.
2. Any goroutine that escapes controller-runtime's leader gate (e.g. the metrics goroutine at main.go:280) still goes through util for actual writes, so it's fenced too.

This is the defence-in-depth idea @Elvand-Lie raised in the #305 thread - manager-level fence AND service-level fence.

tests

util/leader_gate_test.go

Run:

go build ./...
go test -race -count=1 ./util/...

Covers:

• noop always permits
• manager-elected gate with open and closed channels
• nil-manager defensive fallback
• SetDefaultLeaderGate(nil) resets to NoOp
• concurrent set/get is race-free
• per-request context gate overrides the package default
• nil context handled
• errors.Is(err, ErrNotLeader) works through the fmt.Errorf wrap chain

edge cases I handled

• standalone cleanup binary: doesn't call SetDefaultLeaderGate, so its mutations stay ungated. That's correct - it's a one-shot tool, not a long-running reconciler.
• nil manager passed to NewManagerLeaderGate: returns NoOp defensively instead of panicking. A misconfigured wire fails loudly at manager construction, not at every mutation.
• atomic.Pointer for the package default - concurrent reads on the hot path and the single startup write don't need locking.

what i deliberately did NOT do here

• No new CRD. The cross-cluster lease (ControllerLease) is the next PR.
• No state mirror or promotion logic. Those follow.
• No worker-operator changes - tracked in Feature: support controller endpoint/identity updates on failover worker-operator#467 / #468 / #469.
• No leader-election enabled by default. Operators flip it on themselves.

what's next

Follow-up PRs will land the ControllerLease CRD plus a resourcelock.Interface adapter (so two controller clusters can compete for the lease via a witness worker), the state mirror controller for the six controller CRDs, and the promotion controller with pre-promotion sanity checks. This PR is the foundation those sit on.