Skip to content

fix(controller): canonicalise peer order in the agent state - #16

Merged
mxhob1 merged 1 commit into
noden/mainfrom
fix/state-churn
Jul 30, 2026
Merged

fix(controller): canonicalise peer order in the agent state#16
mxhob1 merged 1 commit into
noden/mainfrom
fix/state-churn

Conversation

@mxhob1

@mxhob1 mxhob1 commented Jul 30, 2026

Copy link
Copy Markdown
Member

Follow-up to #15. Manager-side counterpart: stops the pointless state pushes at the source.

Why

The reconciler rewrites the Wireguard Secret whenever the marshalled state differs from what's stored, and every rewrite patches wgConfigLastUpdated on every tenant's WG pod. filteredPeers is built by iterating peers.Items from a controller-runtime cached r.List, which returns informer-map iteration order — so it reshuffles between reconciles even when no peer changed, and the reshuffle alone makes the bytes differ.

Measured on managed-superopti: 340 Updating secret with new config in 37h (~9/h) against 40 peers whose set never changed. Per-tenant counts track peer count (pivit 90, internal 82, optimised 80, wavesquared 51, indigo 47, legion 12, edgecentres 4).

Scope, narrowed by evidence

I originally intended to also strip volatile peer metadata (resourceVersion, generation, managedFields) from the marshalled state. Re-checking 8 captured state.json snapshots says that isn't warranted:

normalisation distinct hashes across 8 pushes
none 8
sort peers by name only 1
sort + strip metadata 1

Sorting alone accounts for all of it — resourceVersion/generation/managedFields were identical across every push. Narrowing the state format the agent consumes would be risk for no measured benefit, so this change is the sort only.

Change

Extract stateForAgent() (pure refactor of two identical construction sites) and sort a copy of the peers by name — unique within a namespace and always set, so a total order. Copying keeps the caller's slice untouched.

The manager only needs a stable order; the agent re-sorts on (Address, AddressV6, Name) for rendering regardless.

Tests

  • TestStateForAgentIsByteIdenticalRegardlessOfPeerOrder — asserts the invariant that actually decides whether the Secret is rewritten. Watched failing: marshalled state depends on peer list order.
  • TestStateForAgentDoesNotReorderCallerPeers — guards against sorting in place.

make test green, including the internal/controller envtest suite (85s, 75.8% coverage).

Not urgent

#15 already stopped the harm — the agent skips the restore when the ruleset is unchanged, so these pushes cost nothing but wasted work. This is cleanup, and is deliberately not being deployed yet: containers stays pinned at 0ce1117.

🤖 Generated with Claude Code

The reconciler rewrites the Wireguard Secret whenever the marshalled state differs from what
is stored, and every rewrite patches wgConfigLastUpdated on EVERY tenant's WG pod.
filteredPeers is built by iterating peers.Items from a controller-runtime CACHED List, which
returns informer-map iteration order, so it reshuffles between reconciles even when no peer
changed -- and that reshuffle alone makes the marshalled bytes differ.

Measured on managed-superopti: 340 "Updating secret with new config" in 37h (~9/h) against 40
peers whose set never changed, and 8 consecutive pushes carried ONE peer set in EIGHT distinct
orders. Normalising only the order collapsed all 8 to a single hash -- no other field, and
specifically not resourceVersion/generation/managedFields, differed between them. The churn is
entirely ordering, so nothing else is touched here.

Extract stateForAgent() and sort a copy of the peers by name: names are unique within a
namespace and always set, so name alone is a total order. Copy rather than sort in place --
the caller built the slice and may still be using it.

#15 already made the agent skip the iptables-restore when the rendered ruleset is unchanged,
so these pushes are no longer harmful. This removes the pointless secret rewrites and the
fleet-wide pod pokes behind them.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mxhob1
mxhob1 merged commit f2d2422 into noden/main Jul 30, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant