Spike: SPIFFE and the alternatives for per-peer mesh identity

[hexfusion]

What would you like to be added

A spike evaluating SPIFFE as the per-peer identity model for the mesh, against the alternatives.

Why is this needed

Deciding to replace the shared key leaves the credential model open. Three candidates:

• SPIFFE/SPIRE. Per-cluster trust domain, workload identity, trust bundles exchanged between peers. This is what Consul's service mesh CA uses for cluster peering, which is the model closest to Grid's topology.
• X.509 by CSR. The site sends a CSR, a CA signs it, revocation is by non-renewal, trust is an offline-verifiable chain. Kubelet and the ACM klusterlet enroll this way.
• Registered key and token. The site registers a public key with a control plane under a bootstrap token, which records key, role, and labels and revokes by removing the entry. Per-message signing, no CSR or X.509.

What the spike should answer: which model bounds a lost or compromised site to that site, what has to be run to operate it, and what the disaster recovery surface looks like for each. A shared key means one site can force a grid-wide re-key and a partition strands the rotation. Per-peer certificates contain it, and the CA becomes the DR-critical piece. A registration control plane contains it too, with a larger DR surface across bootstrap, relay, and registration.

Grid already issues per-site certificates for gateway mTLS, so the CSR path may be closer than it looks.

Related: #75, and the identity issue this spike serves.

From the Grid review, Identity and Disaster recovery.

[hexfusion]

The shared key as an enrollment credential

Grid already issues per-site certificates. site_secret_ref holds one, site_cert_pem rides on the membership broadcast as a trailing extension, GridSite carries a cert_fingerprint to verify it against, and reconcile_site_cert_pem maintains it. That identity is never applied to the mesh, where a single shared AES-256-GCM key is both admission control and the only credential.

The idea is to demote the key rather than replace it. It becomes the credential a site presents once to enroll, and the certificate it receives becomes the credential it uses from then on.

This is the shape Kubernetes uses for a bootstrap token into a kubelet CSR, the ACM klusterlet uses for a bootstrap kubeconfig into a per-cluster certificate, and SPIRE uses for a join token into an SVID. A shared secret with narrow scope buys one per-identity credential and is not used again.

What it changes. A key holder can impersonate any site today, indefinitely, and revoking one site means re-keying every site. As an enrollment credential, a leaked key admits a new identity rather than assuming an existing one. That is bounded, visible wherever enrollment is approved, and revocable per site afterwards.

Why the migration is tractable. Every site already holds the key, so there is no new material to distribute and no ordering problem. Sites enroll on upgrade and the key demotes to bootstrap only. Both can run for a release: accept a signed envelope or a key-encrypted one, and drop the second once every site has enrolled.

Two open questions.

Who signs. Grid spans clusters, so certificates.k8s.io cannot do it, since that authority is per cluster. A grid-level CA has to exist, be operated, and be backed up. That is the topology authority the review already names as a prerequisite for any per-peer identity.

Whether enrollment is gated. A shared key alone still admits anyone holding it, so unattended enrollment moves the problem rather than solving it. Kubernetes pairs bootstrap tokens with CSR approval for this reason. Grid needs an equivalent: approval, one-time-use tokens, or a bounded enrollment window.

Applying the certificate. Signing the StateBroadcast payload works over the existing UDP transport and binds origin_site to a verified signature, which is the hole in #75. It costs roughly 64 bytes for Ed25519 against a 1400-byte datagram carrying 104-byte records. It leaves foca's own probe and suspect messages unsigned. Moving gossip to TCP with mTLS covers every message instead of just the payload, and that transport change is already proposed in #90.