GridNetwork::reconcile() and GridSite::reconcile() have no mocked-kube::Client integration tests (74-75% coverage vs InferenceProvider's 90.6%)

[jordigilh]

Summary

GridNetwork::reconcile() (operator/src/controller/grid_network.rs:234) and
GridSite::reconcile() (operator/src/controller/grid_site.rs:71), along with
their major async sub-functions (e.g. reconcile_routing_overlay_inner,
reconcile_site_cert_pem, reconcile_discovered_sites in grid_network.rs), have
zero tests at the middle tier of the project's three-tier testing pyramid: unit
tests exist, and live-Kind E2E coverage exists, but there is no mocked-kube::Client
integration-test tier driving the actual reconcile() entry points.

This is the one gap in an otherwise-established pattern: operator/src/resources/test_doubles.rs
already provides a mocked-kube::Client builder, and it's already used by
operator/src/resources/secret.rs and operator/src/resources/endpoint_tls.rs. It
is simply never applied to the two largest and most complex reconcile loops in the
operator.

Evidence

• Line coverage (measured via cargo llvm-cov against fresh upstream/main,
  commit 2fdaa26):
  • operator/src/controller/grid_network.rs: 75.2% (2381/3165 lines)
  • operator/src/controller/grid_site.rs: 74.0% (662/894 lines)
  • operator/src/controller/inference_provider.rs: 90.6% (1333/1472 lines) —
    the one controller that has this middle pyramid tier
• operator/src/resources/test_doubles.rs exists and is consumed by
  operator/src/resources/secret.rs and operator/src/resources/endpoint_tls.rs
  (confirmed via repo-wide search) — not by controller/grid_network.rs or
  controller/grid_site.rs.
• grid_network.rs and grid_site.rs each have a mod tests block
  (grid_network.rs:2393, grid_site.rs:587) containing unit tests only — no
  mocked-Client test drives reconcile() itself in either file.

Risk

GridNetwork and GridSite are the two controllers responsible for cross-site
routing-overlay rendering, SWIM membership reconciliation, and site trust/cert
promotion — arguably the highest-blast-radius reconcile loops in the operator. The
missing middle tier means regressions in reconcile control flow (error handling,
requeue behavior, status transitions, finalizer logic) driven through a realistic
mocked API server are only caught by full live-Kind E2E runs (slow, harder to
isolate root cause) or not at all until production. This directly violates this
project's own three-tier testing-pyramid invariant (unit + mocked-kube::Client
integration + live-cluster E2E) for these two controllers.

Suggested fix

Extend operator/src/resources/test_doubles.rs's mocked-Client builder pattern to
cover GridNetwork::reconcile() and GridSite::reconcile(), adding integration
tests for the primary reconcile paths (happy path, missing/stale dependent
resources, status/finalizer transitions) the same way secret.rs/endpoint_tls.rs
already do for their narrower resource-management functions.

Severity: Medium