fix(cluster): cover every control node in the self-signed cert SAN

[SekiXu]

Closes #1421.

The problem

The cluster's self-signed certificate named only the VIP, while several clients address each
control node directly. nginx listens on <node-ip>:9999 and keystone's httpd on
<node-ip>:5443 — node IPs, not the VIP — and those clients do verify the hostname. AMQP is
in the same shape: RabbitMqServers() builds the HA transport_url by walking
cubesys.control.addrs, one entry per node. The SAN and the set of connection targets were
disjoint.

Corrected since this PR was opened. The paragraph above originally said that py-amqp
always passes a server hostname, so check_hostname is on, and therefore that enabling
RabbitMQ TLS (#1167) over this certificate would fail the handshake on every node.
Implementing #1427 measured that directly and it is false. The
cert_reqs = CERT_REQUIRED half is right, but check_hostname is False on this path:
oslo_messaging's _fetch_ssl_params() never puts a server_hostname key in the dict it
hands kombu, kombu fills that key in only when it already exists and is None, and py-amqp
therefore gets the default None — check_hostname = (ssl.HAS_SNI and server_hostname is not None) is False. Verification is chain-only, and the appliance's self-signed
server.cert passes as its own issuer. Measured by calling py-amqp's own
_wrap_socket_sni with the two dicts _fetch_ssl_params() actually produces, on a Caracal
venv. A VIP-only SAN does not block AMQP TLS, so this PR is not a prerequisite for
#1427 — the two are independent. The HTTPS evidence below is what carries this change, and
it was always the measured half.

Confirmed on a live HA cluster before the change:

cubesys.control.addrs = 10.32.10.141,10.32.10.142,10.32.10.143

X509v3 Subject Alternative Name:
    DNS:localhost, DNS:cube-controller, IP Address:127.0.0.1, IP Address:10.32.10.140

This was not only a future problem. nginx listens on <node-ip>:9999 and keystone's
httpd on <node-ip>:5443 — node IPs, not the VIP. Probing all three nodes on :443,
:9999 and :5443 returned hostname mismatch on all nine. Nobody had noticed because
nothing addresses those endpoints by node IP today.

What this does

buildCertSANs() assembles the SAN from the control group: every node IP and hostname,
plus the controller name (which was missing too), alongside the existing
localhost / cube-controller / 127.0.0.1 / VIP entries. Empty values are dropped
rather than emitted as a bare IP: — cubesys.control.addrs is absent altogether from
settings.txt on a non-HA node and strings.Split("", ",") yields [""], which OpenSSL
rejects with invalid null value. Without that guard a fresh non-HA install would fail to
sign a certificate at all.

cubectl config cluster gencerts gives an existing cluster a way to pick this up.
genSelfSignCerts() only runs while creating a new cluster and only when the certificate
is absent, and CONFIG_MIGRATE(cluster, "/var/www/certs") carries the old one across
upgrades untouched — so without this, the wider SAN would reach new installs only.
Deleting the certificate and committing does not work either: the commit path skips
generation once the node has joined, and health_httpd_repair rsyncs the old certificate
back from a peer.

The re-signed certificate now reaches every node, not just the control group.
distributeCerts() pushed /var/www/certs with --role=control, but a compute node holds
that directory too — it is rsynced there at join time regardless of role — so a re-sign left
every compute node trusting a certificate the cluster had stopped presenting, with nothing in
the command's output to say so. Today that mostly goes unnoticed, because nova copies
server.pem but only the control group's novnc proxy reads it back. It stops going unnoticed
with #1427, where the AMQP listener runs TLS and every nova and neutron agent on a compute node
verifies the broker against this directory; the failure there is agents dropping off their
queues, which points nowhere near a certificate command run some time earlier. Dropping
--role from the rsync and from the stale-copy removal reaches all nodes (cubectl node rsync/exec default to ROLE_ALL). The restart stays on the control group, because
certConsumerUnits are control-only services and nothing on a compute node needs one: py-amqp
builds its SSL context per connection and reads ca_certs off disk each time, so a
reconnecting client picks the new file up by itself.

The command signs into a staging directory, reads the result back against
buildCertSANs() itself to confirm the control group landed in the SAN, backs the live
certificate up outside /var/www/certs (health_httpd_repair treats any file count
other than three as damage), and only then swaps it in, syncs to every control node,
drops nova's copy, restarts the consumers and reconfigures ceph.

Three behaviours worth calling out:

• --dry-run touches nothing — verified on a live cluster: mtimes unchanged, all three
  files byte-identical, no staging left behind. It exists so the command can be inspected
  on a shared cluster first.
• A certificate this cluster did not issue is left alone unless --force is given. The
  FQDN procedure installs the customer's own chain in this exact location.
• nova's copy is deleted, not overwritten — config_nova.cpp only refreshes
  /var/lib/nova/certs/server.pem when its own copy is missing, so a regenerated
  certificate would otherwise never reach it.

Ceph is reconfigured with the two dashboard and two config-key commands rather than
through ceph_dashboard_init(), which also creates a radosgw user, wires up SAML2 SSO and
runs terraform. Rancher and k3s carry their own platform PKI and are left to the runbook.

The config_vip.go commit is a prerequisite, not a drive-by: its // +build no-compile
has been an invalid build constraint since Go 1.25, which made go vet and go test fail
to build the whole config package — none of its tests could run, on the build node or in
CI. The fix is gofmt's own output.

Verification

Measured on the sky 3cc HA lab, then fully rolled back.

SAN after gencerts	covers all three node IPs, all three hostnames, the VIP and the controller name
all three nodes	serve the identical certificate (rsync)
hostname verification	9 endpoints × by-IP and by-hostname = 18 checks, zero mismatch (only 18 self-signed, as expected)
consumers	15 units back to active; ceph HEALTH_OK; dashboard, mgr restful and on-disk certificate all the same
nova copy	dropped on all three nodes
file count	still three; backups land in /var/lib/cube-certs-backup/
runtime	17 seconds end to end

Before/after with one offline probe on the old (backed-up) and new certificates:

target	old	new
10.32.10.140 (VIP)	match	match
10.32.10.141/142/143	NOT match	match
sky, sky141	NOT match	match
cube-controller	match	match

The VIP and cube-controller still match, so the consumers that reach the cluster through
the VIP are unaffected.

Unit tests: TestBuildCertSANs covers HA, non-HA with the keys absent, empty members,
de-duplication, and a non-control node; TestGenSelfSignCerts signs end to end against the
real openssl. Every commit builds, vets and tests clean on its own.

Known gaps

• If rsync or the restarts fail, the certificate has already been swapped in on the master
  — the nodes are then inconsistent. There is a backup and its path is logged, but there is
  no automatic rollback. Deliberate: an automatic rollback can fail too, and would make the
  resulting state harder to read.
• The private key stays 0644, as it was before. Tightening it needs an audit of which uid
  each of the consumers runs as; worth a separate ticket.
• Non-HA was covered by tests rather than a real 1cc run, since the SAN content there is
  equivalent to before the change and the risk is concentrated in whether it signs at all.

🤖 Generated with Claude Code