Generate the Ceph fsid once per install instead of compiling it in

[traviswu-bigstack]

What type of PR is this?

/kind bug

What this PR does / why we need it

Every CubeCOS cluster reports the same Ceph fsid. It was a compile-time constant that ceph.fsid merely defaulted to, so two installs are indistinguishable to anything that asks "is this the same cluster?".

The fsid is now minted once, by the bootstrap node, and recorded at /etc/cube/cos/ceph/fsid — inside a directory the ceph module already declares to CONFIG_MIGRATE, so it survives the partition switch an upgrade performs with no new migrate line. Every later commit reads the record rather than recomputing, which matters because the fsid reaches monmaptool: a value that drifted between commits would build a monmap the mons reject.

ceph_fsid_resolve owns the decision and config_ceph.cpp only asks. Resolution order:

1. ceph.fsid, when an operator has pinned one. The tunable's default changes from the constant to "", so empty now means "not pinned" rather than "use the shared one".
2. Whatever already exists, on a node that predates the record — the fsid in ceph.conf, else this node's own mon store via ceph_mon_map_create, which extracts with ceph down. That second source is the one that matters: an upgrade commit runs before ceph is necessarily up, and /var/lib/ceph is CONFIG_MIGRATE'd.
3. Mint a UUID — bootstrap node only.

Generated, not derived. Deriving from cubesys.seed would have matched what rbd_secret_uuid already does (uuidgen --sha1 --namespace @dns), and it was the first thing I tried to justify. Two reasons not to: the seed also generates the cluster's SSH key (config_cubesys.cpp:368), and the fsid is published in every ceph.conf — a v5 UUID there hands anyone an offline oracle against a low-entropy secret. Independently, the seed is not unique today (#1500).

An existing cluster never moves. Step 2 outranks step 3, so an upgraded cluster adopts and records the fsid its mons and OSDs already carry, the old constant included. LEGACY_FSID is kept only to name it.

A joining node takes the bootstrap node's fsid over ssh, inside the retry loop that already waits for the master's mon bootstrap — the same wait for the same reason, since neither answer exists until the master has made it. Only the bootstrap node may mint, so a cluster gets exactly one.

Nothing falls back to a constant. An unresolved fsid refuses the commit, and sync_ceph_config keeps the existing ceph.conf rather than rewriting it with a worse one. Every value is shape-checked as a UUID before use, on both sides, because HexUtilPOpen discards exit status and the shape is the only signal a C++ caller gets — same lesson as #1486. Validate() now also rejects a malformed operator-pinned ceph.fsid, which ValidateRegex did not.

Which issue(s) this PR fixes

Fixes #1490

Special notes for your reviewer

Note

Not exercised on a cluster. config_ceph.o compiles in the jail (which caught one real bug — newValue() returns std::string, not const char*), and core/sdk_sh/tests/test_ceph_fsid.sh is 16/16, mutation-tested two ways: letting ceph_fsid_record overwrite gives 1 failure, dropping the UUID shape check gives 10. That is the whole of the verification. The two runs I would want before this merges:

• an upgrade of a cluster carrying c6e64c49-09cf-463b-9d1c-b6645b4b3b85, confirming it adopts and records rather than mints;
• a fresh multi-node install, confirming all three control nodes converge on one fsid. The joining-node ssh path is argued, not observed.

Two things I would like a second opinion on:

1. The retry loop now breaks only when both the bootstrap mon ip and the fsid are known. That is deliberate — a node with one and not the other cannot configure itself — but it does mean an fsid problem now presents as the existing "waiting for master's mon" warning rather than its own.
2. A compute node has neither a ceph.conf that survives the partition switch nor a mon store, so on upgrade it resolves by ssh to the master. That is fine if control nodes roll first, and the loop waits, but it is a cross-node dependency where there was none — the old constant needed nobody.

An earlier revision of this PR was larger. Dropped on review: a live ceph fsid probe (redundant — the CLI needs ceph.conf to find the mons, so it can never answer where reading ceph.conf could not), and sync_ceph_config asking a peer for the fsid (it resyncs an already-bootstrapped node, so the record exists; and GetMaster on a single-node cluster would have ssh'd to self). I also considered doing the adoption in the module's migrate hook, using prevRootDir — but no hook in this repo uses that argument as a path, and I did not want the design resting on an assumption about it.

Related, not fixed here: #1500 — FTS accepts an empty cubesys.seed, which makes SSH key material identical across installs. Same shape of defect (a cluster identity that is constant across installs), and the more serious of the two.

Additional documentation

kb/cubecos/known-issues/ceph-fsid-identical-across-installs.md — the defect, its blast radius, and why the preflight guard built on it was withdrawn
kb/cubecos/architecture/live-volume-move-between-ceph-backends.md — why retype is unaffected either way