feat(monitoring): replicate the metrics pipeline, and stop health repairs causing outages

[Eandalf-Bigstack]

What type of PR is this?

• bug
• feature

What this PR does / why we need it

The next slice of the telemetry replacement: make the metrics pipeline survive a control node going away, and stop the health machinery from turning transient faults into outages.

• Kafka topics replicate across the control nodes. Every topic was created at RF 1, so any broker loss took its partitions offline and the monitoring pipeline with them. TargetRF() is min(control nodes, 3) — not a flat 3, which would fail topic creation on a 2-control HA cluster — and it now backs RecreateTopic, UpdateCfg, and the three broker defaults (default.replication.factor, offsets.topic.replication.factor, transaction.state.log.replication.factor). kafka_topic_rf_reconcile raises existing topics, which is what carries v3.1.10-and-older clusters across the upgrade: it prunes, places by least-loaded broker, is idempotent, and defers entirely while any broker is offline
• Thanos, so the Prometheus replicas stop being islands. Each control node scraped and stored on its own, so a node that was down answered its own gap forever and a query hit whichever replica haproxy chose. Bare-metal tarball install (v0.42.4, verified against sha256sums.txt), systemd units running as prometheus, installed disabled and turned on by hex_config like everything else. config_prometheus.cpp writes external_labels: replica: <hostname>, the sidecar/querier defaults and /etc/thanos/endpoints.yml; the querier deduplicates on replica. Thanos is HA-only — thanosEnabled = enabled && s_ha — and haproxy routes /prometheus at the three queriers on HA
• Stop health_mysql_repair causing the outage it was sent to fix. Four defects, one shape. It fired on a galera cluster that was merely still forming, restarted mariadb on every control node to repair one, copied the whole datadir once per retry, and never pruned the copies. On accept-3cc that filled / twice — once taking every ceph mon down with ENOSPC for 8h, once blowing OpenSearch's 85% watermark and stalling log ingestion behind it
• InfluxDB starts again past ~25G. influxd-systemd-start.sh runs influx_inspect buildtsi -compact-series-file before every start — not once, every boot and every restart — and inherits DefaultTimeoutStartSec=300s. Past the ceiling the unit never starts again: Type=forking waits, KillMode=control-group kills the compaction at the deadline, Restart=on-failure restarts it from zero, and health_influxdb_repair kills it again. A drop-in raises the ceiling to 1800s (~120G)
• Prometheus and Thanos get health coverage, under a new MetricsDb service. Neither had any: sdk_health.sh contained no mention of either, so a dead Prometheus or a querier that had lost its peers was invisible to cluster check and to auto-repair
• cube_remote_cluster_check compares a real hostname. There is no hostname() in the sdk, so remote_run $N "$HEX_SDK hostname" returned 1081 lines of usage into the substitution and the comparison could never match — the moderator/edge remote-check path was dead. Reported by a peer session on 3.1.10

Which issue(s) this PR fixes

• Refs [Story] Keep CubeCOS telemetry working by replacing Monasca with Prometheus and exporters #672

Special notes for your reviewer

This PR stacks on jim.lin/feat/replace-monasca (#1428). The base here is develop, so until that merges this PR's commit list also shows #1428's five commits. Only the thirteen below belong to this PR:

• fix(kafka): replicate topics across the control nodes, capped at three
• feat(kafka): raise existing topics to the target replication factor
• feat(thanos): package the thanos sidecar and querier
• feat(prometheus): configure and run thanos on the control nodes
• fix(haproxy): route /prometheus at the thanos queriers on HA
• fix(health): bound the mysql datadir copy the repair takes
• fix(health): only repair galera when a node is truly down, not merely absent
• fix(health): restart only the mariadb nodes that need it, not the whole tier
• fix(health): take the mysql datadir safety copy once per repair, not per attempt
• fix(cluster): compare the real hostname in cube_remote_cluster_check
• fix(influxdb): raise TimeoutStartSec past the per-start series rebuild
• feat(health): check and repair prometheus and thanos
• feat(cluster): add the MetricsDb service for prometheus and thanos

Review it after #1428, or diff against jim.lin/feat/replace-monasca.

Why the four galera fixes are one story. health_mysql_check is strict on purpose — wsrep_cluster_size equal to the control-node count, every node Synced, every unit active — and that strictness is what makes it the right gate for a roll to advance on (handbook rolling-upgrade-no-galera-health-gate.md, #651). So the check is deliberately not weakened here. The fix is in the repair: "the cluster is short a member" and "the cluster is broken" are different statements, and only the second is repairable. _health_mysql_node_state classifies each control node reachability first — asking galera whether a node is in the cluster means nothing on its own, because a node the network cannot reach was evicted correctly, and bootstrapping cannot bring it back, only destroy the cluster the survivors still have.

Why this class of bug keeps recurring. Four of the fixes here are the same failure mode: a repair fires at a system that is recovering normally, and the repair is what breaks it. Galera still forming; Synced members restarted to fix a peer; a datadir copied per attempt; a TSI rebuild killed mid-flight. The house precedent for the right answer already existed in 741f4093 (pacemaker_settled(), "the fix is to wait for quorum, not repair"). The three questions worth asking of any health_*_repair: can the check tell mid-recovery from failed, does the repair touch only the faulty member, and is its work bounded per invocation.

Why the operator is never gated. An earlier revision of this branch deferred essential-tier repair during a roll. That was backwards — cluster_check_repair_essential has no in-tree caller, so its only caller is a human typing the command, and the gate would only ever have blocked the person who asked for it. It was dropped. The roll is protected by health_mysql_repair itself declining to act on a joining node, which covers every caller.

Thanos ports and prefixes are not uniform, and the checks had to measure rather than assume. Prometheus derives --web.route-prefix from --web.external-url, so :9091/-/healthy is a 404 on a healthy server and :9091/prometheus/-/healthy is the live endpoint. Thanos keeps /-/healthy and /-/ready at the root even though the querier runs --web.route-prefix=/prometheus — only its API moves under the prefix, which is why haproxy's option httpchk GET /-/ready works and :10904/api/v1/stores is a 404 while :10904/prometheus/api/v1/stores is not.

Two auto-repairs deliberately decline. prometheus ERR 3 (up, scraping nothing) is a config or service-discovery problem a restart cannot fix, so repairing it would be a restart loop against a server answering fine. thanos ERR 4 (a peer's sidecar unreachable) is usually a node that is down; restarting the local querier does not bring a peer back, and during a roll it would restart a querier on every pass.

Companion PRs, same metricsDb service: bigstack-oss/cube-cos-openapi#111 (base), bigstack-oss/cube-cos-api#650, bigstack-oss/cube-cos-ui#867.

Additional documentation

Verified on accept-3cc (cc1/cc2/cc3, control-converged, HA) unless stated otherwise.

For the requirement "Kafka survives a control node going away",

[cc1 ~]# hex_sdk kafka_topic_rf_reconcile cc1:9095
[cc1 ~]# /opt/kafka/bin/kafka-topics.sh --bootstrap-server cc1:9095 --describe | grep -c "ReplicationFactor: 3"
111
[cc1 ~]# # one broker down, then two
[cc1 ~]# /opt/kafka/bin/kafka-topics.sh --bootstrap-server cc1:9095 --describe --unavailable-partitions | wc -l
0
[cc1 ~]# # partition balance across the three brokers
111 111 111

At RF 1 the same probe showed 2 leaderless partitions per topic with a single broker down. An auto-created topic lands at RF 3 with ISR [2,1,0], and a rolling restart of all three brokers keeps unavailable partitions at 0 throughout.

For the requirement "queries reach more than one replica",

[cc3 ~]# systemctl stop prometheus          # 4 minutes
[cc3 ~]# # samples for the same series over the gap:
cc3 prometheus (raw)            9
cc3 thanos-query                16
via cc3 haproxy /prometheus     16

The querier fills its own node's hole from its peers, which is the whole point — before Thanos, cc3 answered 9 and there was no way to get 16 out of it.

For the requirement "a galera repair only fires when a node is truly down",

[cc1 ~]# # classifier across all four states
A. healthy            NO-OP   (cc1=synced cc2=synced cc3=synced)
B. mariadb stopped    REPAIR  -> cc3 truly down (cc1=synced cc2=synced cc3=down)
C. mid-rejoin         NO-OP   (cc1=synced cc2=synced cc3=joining)
D. unknown host       -> unreachable

B is the control: a genuinely dead mariadbd on a reachable node still repairs, so #155's stale-mariadbd case is preserved. C is the fix — during the rejoin the unit sits in activating and answers no query at all for the whole state transfer, which is the longest phase of joining and the only window the old code could not see.

For the requirement "a repair never takes the healthy members with it",

[cc1 ~]# # which nodes the repair would restart
all Synced            would restart: (none)   [cc3=synced cc2=synced cc1=synced]
cc3 mariadb stopped   would restart: cc3      [cc3=down   cc2=synced cc1=synced]
after rejoin          would restart: (none)   [cc3=synced cc2=synced cc1=synced]
[cc1 ~]# # cluster intact afterwards
cc1 size=3  cc2 size=3  cc3 size=3

And the datadir safety copy is now once per repair rather than once per attempt — 1 copy whether the repair succeeds on attempt 1 or 3, re-armed on the next call, and still skipped entirely on the live-primary rejoin path (#1328).

For the requirement "prometheus and thanos are checked and repaired",

[cc1 ~]# hex_sdk health_prometheus_check ; echo rc=$?
rc=0
[cc1 ~]# hex_sdk health_thanos_check ; echo rc=$?
rc=0
[cc1 ~]# hex_sdk health_errcode_lookup prometheus 3 ; hex_sdk health_errcode_lookup thanos 4
no scrape target up
querier missing a replica
[cc3 ~]# systemctl stop thanos-sidecar
[cc1 ~]# hex_sdk -v health_thanos_check ; echo rc=$?
thanos-sidecar on cc3 is not running
rc=1
[cc1 ~]# hex_sdk health_thanos_repair ; hex_sdk health_thanos_check ; echo rc=$?
rc=0
[cc3 ~]# systemctl stop prometheus
[cc1 ~]# hex_sdk -v health_prometheus_check ; echo rc=$?
prometheus on cc3 is not running
rc=1
[cc1 ~]# hex_sdk health_prometheus_repair ; hex_sdk health_prometheus_check ; echo rc=$?
rc=0

The endpoint behaviour the checks are built on, measured rather than assumed:

[cc1 ~]# for u in 9091/-/healthy 9091/prometheus/-/healthy 10902/-/ready 10904/-/ready ; do
>   printf "%-28s %s\n" $u "$(curl -s -o /dev/null -w '%{http_code}' http://cc1:$u)" ; done
9091/-/healthy               404
9091/prometheus/-/healthy    200
10902/-/ready                200
10904/-/ready                200
[cc1 ~]# curl -s http://cc1:10904/prometheus/api/v1/stores | jq -r '.data.sidecar[] | .name'
10.1.0.1:10901
10.1.0.2:10901
10.1.0.3:10901

For the requirement "the MetricsDb service is reported",

$ g++ -fsyntax-only -Wall -Werror -std=c++17 ... core/modules/cli_cluster.cpp   # rc=0
$ ./xm    # standalone X-macro expansion after the edit
TOP=29  S[Metrics]=Metrics  S[MetricsDb]=MetricsDb  S[Node]=Node

For the requirement "influxdb starts past the 300s ceiling" — verified on cube4510, not accept-3cc, because that is where the large datadirs are:

[cube451 ~]# systemctl show influxdb -p TimeoutStartUSec
TimeoutStartUSec=30min
[cube451 ~]# systemd-analyze verify influxdb.service     # clean
[cube451 ~]# # rebuild time by datadir size
21G  ->  3m26s   (cube451)
25G  ->  6m12s   (cube452)

cube452 had been restart-looping indefinitely after its 3.1.0 → 3.1.10 upgrade, taking Notifications and Metrics down with it; with the drop-in it completed in 6m12s and the bootstrap ran through to 83/83 modules committed.

For the requirement "the remote cluster check compares a real hostname",

[cc1 ~]# # old form vs fixed form, per node
cc1  old=[Usage: hex_sdk <option> <fun]  match=no   |  new=[cc1]  match=YES
cc2  old=[Usage: hex_sdk <option> <fun]  match=no   |  new=[cc2]  match=YES
cc3  old=[Usage: hex_sdk <option> <fun]  match=no   |  new=[cc3]  match=YES
[cc1 ~]# hex_sdk remote_run cc2 "hex_sdk hostname" | wc -lc
1081   23652

Audited every $HEX_SDK <word> invocation in the tree against the 1079 defined sdk function names; this was the only genuine miss. Note the usage listing goes to stdout, so inside $( ) it is captured rather than printed — a control node returns at the is_control_node guard above it and shows nothing, which is why cluster check there stays at 28 clean lines.

[SekiXu]

LGTM.

Checked the things that would only surface at build or run time, since CI here is CodeQL/DCO with no build gate:

• config_prometheus.cpp uses hex_string_util::split without including <hex/string_util.h>. It resolves transitively through hex/config_tuning.h, the same way config_apache2.cpp and config_cinder.cpp already do, so this compiles.
• TargetRF(): GetClusterSize() returns 1 when !ha (core/cube_sdk_library/src/cluster.cpp:95), so non-HA keeps RF 1 and matches the old ha ? 2 : 1; HA gets min(ctrlAddrs, 3).
• kafka_target_rf()s cubectl node list -r control | wc -l matches the four existing uses of that idiom in sdk_health.sh / sdk_ceph.sh, so no off-by-one from a header row.
• The Grafana datasource move to http://localhost/prometheus works: both cube_cos_http and cube_cos_https already route path_beg /prometheus at prometheus_backend.
• option httpchk GET /-/ready is the same form ceph_prometheus_ep already uses in the srvlist table.

The _health_mysql_repairable gate is the part I most wanted to see here. Deciding reachable/joining/synced/down before anything gets restarted, and testing reachability first so a correctly evicted node cannot trigger a bootstrap, is the right way round.