feat(security): verify GridSite gateway identity during health evaluation

[nerdalert]

Summary

Make GridSite gateway health evidence identity-aware. A reachable TCP port is
not sufficient evidence that a remote endpoint is the expected Grid gateway.

Grid should perform a bounded TLS handshake, validate the presented identity,
and report precise failure and rotation state before an endpoint becomes
eligible for routing.

Motivation

Reachability, authentication, and authorization are different properties:

• TCP reachability proves that something accepted a connection.
• A TLS handshake proves possession of a private key.
• Certificate validation proves that the peer chains to an accepted trust
  anchor and presents the expected identity.
• Grid policy determines whether that authenticated identity is eligible.

Conflating these checks can publish an endpoint that is reachable but belongs
to the wrong site, uses an expired certificate, or presents an untrusted
identity.

Scope

Implement identity-verified GridSite gateway probing with:

• TLS handshake and bounded connect/handshake timeouts;
• configured SNI and hostname or URI identity verification;
• certificate-chain and validity-period validation;
• proof of private-key possession through the handshake;
• canonical DER-based certificate or SPKI fingerprints where pinning is used;
• a defined trust-source and trust-anchor lifecycle;
• explicit probe results for unreachable, timeout, TLS failure, untrusted
  issuer, identity mismatch, expired/not-yet-valid certificate, pin mismatch,
  and success;
• fail-closed routing eligibility;
• bounded retry/backoff behavior;
• certificate-rotation overlap and recovery semantics; and
• status, Events, logs, and metrics that do not disclose certificate private
  material or create unbounded cardinality.

Public certificates and bounded trust metadata may be distributed through Grid
state when the trust model permits it. Private keys, API credentials, bearer
tokens, and complete Kubernetes Secrets must never be distributed through
SWIM or a routing overlay.

Rotation Requirements

• Define how old and new certificate identities overlap during rotation.
• Do not require all sites to switch at one instant.
• Reject an unexpected third identity even during a rotation window.
• Define behavior when a rotation is partially deployed or rolled back.
• Ensure cached probe state expires and cannot outlive the accepted trust
  window.
• Preserve routing through a still-valid identity when policy permits it.

Acceptance Criteria

• A valid gateway certificate, chain, SNI, and expected identity produce a
  healthy result.
• TCP-only listeners cannot become healthy.
• Missing, wrong-CA, wrong-SNI, wrong-identity, expired, not-yet-valid, and
  pin-mismatch certificates fail closed with distinct bounded reasons.
• Fingerprints are derived from canonical certificate material, not PEM
  formatting.
• Successful probing proves private-key possession.
• Rotation overlap, partial rollout, rollback, expiry, and recovery are
  runtime-tested.
• Probe timeouts, concurrency, retries, memory, and retained state are
  bounded.
• Multiple operator replicas and restarts converge on the same eligibility
  result.
• No secret or private-key material appears in SWIM state, ConfigMaps,
  status, logs, metrics, or test evidence.
• GridSite conditions clearly distinguish reachability from verified
  identity.

Non-Goals

• Distributing private keys or complete Secrets between sites.
• Treating a certificate fingerprint as the complete authorization policy.
• Disabling normal X.509 validation in favor of an insecure probe.
• Replacing provider-gateway request authorization or downstream mTLS policy.
• Creating a Grid-specific certificate authority service.

Open-Source Requirements

Use standard TLS and X.509 behavior with portable Kubernetes configuration.
The design must work with common certificate issuers and secret-management
systems without requiring a proprietary PKI or distribution-specific
controller.

[nerdalert]

Implemented on main in 2292dd5.

Static validation is green: make lint, make test, and git diff --check all pass. The implementation includes identity-aware TLS probing, canonical certificate pins, fail-closed routing eligibility, bounded probe execution, rotation handling, Events, metrics, and the standalone rotation/convergence verifiers.

Leaving this issue open for the two runtime acceptance gates required by the issue itself:

• cargo xtask env verify-gridsite-rotation
• cargo xtask env verify-gridsite-convergence

Once both pass from the committed source, the remaining runtime checkboxes can be marked complete and the issue can be closed.

[nerdalert]

Implementation and validation checklist

Implemented on main in 2292dd5.

Done: implementation and static validation

• Perform a bounded TLS connection and handshake instead of treating TCP reachability as gateway health.
• Validate the configured SNI, certificate chain, validity period, and expected gateway identity.
• Prove private-key possession through the successful TLS handshake.
• Reject TCP-only listeners and missing, wrong-CA, wrong-SNI, wrong-identity, expired, not-yet-valid, and pin-mismatch certificates with distinct bounded phase/reason values.
• Derive certificate pins from canonical DER certificate material rather than PEM formatting.
• Fail closed by removing an unverified site from routing eligibility.
• Bound probe timeouts, retries, concurrency, status notes, metric labels, and retained probe state.
• Implement two-pin certificate-rotation overlap and reject an unexpected third pin.
• Implement partial-rollout, rollback, expiry, and recovery transitions in the rotation verifier.
• Emit bounded Kubernetes Events only for phase/reason transitions.
• Expose bounded probe counters, duration metrics, and phase-transition counters through the operator metrics endpoint.
• Keep certificate private keys, credentials, and complete Secrets out of SWIM state, routing overlays, status, Events, logs, metrics, and evidence contracts.
• Distinguish reachability, TLS verification, identity verification, and routing eligibility through GridSite phase/reason status.
• Implement the standalone verify-gridsite-rotation runtime verifier.
• Implement the standalone verify-gridsite-convergence multi-replica runtime verifier, including replica loss, restart, reconvergence, Event checks, and metrics scraping.
• Pass committed-source static gates: formatting, check, clippy with warnings denied, workspace tests, rustdoc with warnings denied, and git diff --check.

Not done: required runtime acceptance evidence

• Run cargo xtask env verify-gridsite-rotation from committed main and retain sanitized evidence proving valid identity, mismatch, dual-pin overlap, single-pin cutover, partial rollout, rollback, expiry, and recovery.
• Run cargo xtask env verify-gridsite-convergence from committed main and retain sanitized evidence proving two operator replicas converge, remain stable, recover from replica loss/restart, avoid duplicate transition Events, and expose the expected bounded metrics.
• Confirm both runtime verifiers clean up all clusters, networks, processes, and temporary credential material.
• Link the two evidence reports here and reconcile each acceptance-criteria checkbox in the issue body.
• Close Issue feat(security): verify GridSite gateway identity during health evaluation #11 only after both standalone runtime verifiers pass without manual live-cluster patches or weakened assertions.

The implementation is complete, but the issue remains open because static and unit coverage do not replace the two runtime acceptance gates above.