[Story] Keep CubeCOS telemetry working by replacing Monasca with Prometheus and exporters

[jdjgya]

Description

As a CubeCOS operator, I want metrics collection, the Watcher data source, Grafana dashboards and the service-health checks to keep working after Monasca is removed, so that observability is preserved without CubeCOS carrying a retired upstream project as a private fork.

Monasca was retired upstream on 2025-08-15 and has no maintained fork. Spike #646 investigated both options on live clusters and concluded that forward-porting Monasca is the more expensive path, not the cheaper one — it buys a private fork whose CVE triage, security reporting and regression testing we own forever, plus a second Python runtime on every node. The decision is recorded in handbook ADR 0004; the design and evidence are in the Prometheus telemetry replacement blueprint.

This story is now the replacement: Prometheus plus packaged exporters, phased so Prometheus runs alongside Monasca until every consumer is repointed, with Monasca removed last.

Most of the work is packaging rather than engineering — the prometheus repo CubeCOS already configures carries 12 of the 15 monasca-agent plugins' equivalents as el9 RPMs, and the host half was proven end to end during the spike (Watcher's own host_cpu_usage expression returned 23.46 % against Monasca's 23 % and Telegraf's 22.64 % at the same moment). The one real gap is per-instance metrics, which is the long pole.

Acceptance Criteria

Functional behavior and constraints, not code-level implementation.

Phase 1 — collection (runs in parallel with Monasca, no consumer changes)

• node_exporter, blackbox_exporter, memcached_exporter, apache_exporter and ipmi_exporter are installed and enabled per role, on a port that does not collide with CubeCOS haproxy (node_exporter's default :9100 is taken) — done in #1441. haproxy_exporter is not among them and is not needed: haproxy 2.8 is built +PROMEX and reports Available services : prometheus-exporter, so one http-request use-service line on the stats listener it already binds is the whole of it — and it reports haproxy's own counters rather than re-parsing the CSV stats page. Both listeners get it, because there are two haproxies and they are not interchangeable: haproxy.cfg per control node with stats on :9100, and haproxy-ha.cfg, the Pacemaker singleton owning the VIP, with stats on :9000.
  • Sourcing is split, and it is not the packagecloud repo. That repo was deleted from the build in #1428: it is stale for every exporter (node_exporter 1.9.0 vs 1.12.1, blackbox 0.26.0 vs 0.28.0, memcached 0.15.0 vs 0.17.0, apache 1.0.10 vs 1.1.1, snmp 0.28.0 vs 0.30.1, statsd 0.28.0 vs 0.31.0). EPEL carries node-exporter at 1.12.1 with a full packaging set and none of the others; no upstream exporter publishes an rpm at all — every one ships only a linux-amd64.tar.gz. So this is one ROOTFS_DNF line plus a tarball-install pattern, not a repo enable.
• config_prometheus.cpp generates the scrape jobs and per-node target lists, each carrying the fqdn label whose value matches the compute node's hostname — done in #1441. The target lists are file_sd files written by a cron generator rather than static config, because compute and storage are not in cubesys.control.addrs and a joining node must not need a hex_config commit before it is scraped. Verified against Watcher's own expression: host_ram_usage reads 27.8 GiB on cc1 where free -g reports 27 used.
• Prometheus scrapes itself successfully — done in #1428. --web.external-url ends in /prometheus/, so Prometheus serves its own metrics at /prometheus/metrics and 404s a bare /metrics; the job now carries a matching metrics_path. Fixed by moving the scrape target, not by dropping the prefix — haproxy forwards the path unchanged, so --web.route-prefix=/ would break the UI and Grafana. up{job="prometheus"} is now 1 and ~700 prometheus_* series exist that never did before.
• Works on both single-node and 3-node HA, with shared exporters reached through the VIP (the existing ceph job at :9285 is the pattern) — done in #1441. 16 of 16 targets up on jim-1cc and 33 of 33 across twelve jobs on accept-3cc.
• Queries reach more than one replica — done in #1430. On HA, haproxy's prometheus_backend now lists all three thanos-query instances at :10904 with option httpchk GET /-/ready, and Grafana's Dashboard1 datasource points at http://localhost/prometheus rather than localhost:9091. Was: haproxy's prometheus_backend had exactly one server — 127.0.0.1:9091, the local Prometheus — and Grafana's Dashboard1 datasource points at localhost:9091 too. So an answer depends entirely on which node holds the VIP, with no load-balancing across replicas and no deduplication. Verified on accept-3cc.
• Add Thanos so replicas stop being islands — done in #1430. Sidecar + querier installed from the release tarball (v0.42.4, sha256sums.txt verified), run under systemd as the prometheus user, installed disabled and enabled by config_prometheus.cpp, which also writes external_labels: replica: <hostname>; the querier deduplicates on that label. HA-only (thanosEnabled = enabled && s_ha). Proven on accept-3cc: with cc3's Prometheus stopped for 4 minutes, cc3's raw TSDB held 9 samples for a series while its own querier returned 16. Original finding: prometheus.yml has no remote_write, remote_read, federation or rule_files (zero matches) — the three instances never exchange data. While a node is down that is fine and Prometheus degrades visibly (up=0), but a node that comes back keeps a permanent hole for the whole outage, and if the VIP later lands on it, queries silently serve that hole — the same silent-degradation shape as the InfluxDB path, reached by a different mechanism. Thanos, not Mimir: we run this on bare metal via systemd, not as a k8s deployment.
• Fix the Kafka replication factor — done in #1430. TargetRF() is min(control nodes, 3) — not a flat 3, which would fail topic creation on a 2-control HA cluster — and backs RecreateTopic, UpdateCfg and all three broker defaults. kafka_topic_rf_reconcile raises existing topics, which is what carries v3.1.10-and-older clusters across the upgrade (broker defaults only govern new topics); it prunes, places on the least-loaded broker, is idempotent, and defers while any broker is offline. Verified on accept-3cc: 0 unavailable partitions with one and with two brokers down, balance 111/111/111, auto-created topics landing at RF 3. Original finding: needed so the existing pipeline survives a node loss while both stacks run in parallel. On a 3-node cluster only alarms and events are RF 2; metrics, logs, audit-logs, telegraf-*, notifications.info, transformed-logs and __consumer_offsets are RF 1, so losing one broker leaves 2 of 6 partitions with Leader: none. Three causes, all fixable in the generated config: config_kafka.cpp creates topics with --if-not-exists (a silent no-op once a producer has auto-created them at RF 1, and the following --alter changes partitions only, never RF); nothing sets auto.create.topics.enable or default.replication.factor, so Kafka's defaults (true / 1) win the race; and offsets.topic.replication.factor = 1 is inherited unmodified from upstream Kafka's sample server.properties. Fix = set both replication-factor keys in the generated config, and either disable auto-create or have UpdateTopics reconcile RF on existing topics (a partition reassignment — --alter cannot do it).

Prerequisite work already landed (#1428)

• Prometheus is on a supported release — 3.13.1 from EPEL, the current LTS line (to 2027-07-31). It was on 2.55.1, which was never an LTS and sits behind three EOL 2.x LTS lines. The source build and the packagecloud repo are both retired.
• Grafana is on a supported release — 12.4.10, the last minor of major 12 and so its LTS equivalent (to 2027-05-24). 12.3 went EOL on 2026-08-19.
• Grafana's twelve Angular pie charts migrated to the core piechart panel, and the two dead grafana-cli plugin installs removed.

Also landed in #1430 (not in the original criteria — surfaced by this work)

• Prometheus and Thanos are health-checked, repaired and auto-repaired. Neither had any coverage — sdk_health.sh contained no mention of either — so a dead Prometheus, or a querier that had lost its peers, was invisible to cluster check and to auto-repair. Both auto-repairs deliberately decline the fault a restart cannot fix: prometheus ERR 3 (up but scraping nothing) and thanos ERR 4 (a peer's sidecar unreachable).
• A MetricsDb service reports them, rather than folding them into Metrics (collection and visualisation). influxdb/kapacitor stay under Notifications for now, since Kapacitor is a write proxy and alerting engine here, not only storage. Landed across all four repos: cubecos#1430, cube-cos-openapi#111, cube-cos-api#650, cube-cos-ui#867.
• Health repairs stopped causing outages. Four defects, one shape: a repair firing at a system that was recovering normally. health_mysql_repair bootstrapped over a still-forming Galera cluster, restarted Synced members to fix a peer, and copied the whole datadir once per retry until / filled (twice: once killing every ceph mon with ENOSPC, once turning OpenSearch red and stalling log ingestion); health_influxdb_repair killed a TSI rebuild mid-flight. Fixed by classifying nodes reachability-first and repairing only a node that is reachable but neither in the cluster nor joining. health_mysql_check was deliberately not weakened — its strictness is what makes it the right gate for a roll to advance on.
• cube_remote_cluster_check compares a real hostname. There is no hostname() in the sdk, so the comparison captured 1081 lines of usage text and could never match — the moderator/edge remote-check path was dead. Reported by a peer session on 3.1.10.

Also landed in #1439 (retention items 1, 2, 5 and 6 of the comment above, plus the credential collision they surfaced)

• Both time-series stores are bounded, and the bounds are tunable. InfluxDB's def policy was 364d + 35d — those two numbers add, because InfluxDB 1.x drops whole shard groups and a group expires only once its end time plus the duration has passed, so the worst case on disk was 399 days. Now influxdb.def.rp.duration/.shard = 14/7 and influxdb.hc.rp.* = 7/2, with a clamp for a shard duration larger than the retention duration. Applied live: jim-1cc 125M → 58M; accept-3cc 279 → 122M, 316 → 147M, 265 → 124M.
• Prometheus has a size cap as well as a time cap — prometheus.rp.duration 30d and prometheus.rp.size 5 GiB, written into prometheus.yml's storage.tsdb.retention block rather than the flags. 3.13 marks both flags [DEPRECATED], and a flag would take precedence over the block and silently pin the value. This is 3.x-only: 2.55 refuses to parse the field rather than ignoring it, which took all three accept-3cc nodes to activating before that cluster was upgraded to 3.13.1.
• The capped history is no longer thrown away — the Thanos sidecar ships finished blocks to a Ceph RGW bucket, a store gateway serves them back to the querier, and a compactor downsamples and enforces prometheus.thanos.rp.duration (90d) in the bucket. Retention is Thanos's rather than an RGW lifecycle rule: expiry Thanos does not know about strands block metadata. The sidecar also required Prometheus's own compaction off (max-block-duration == min-block-duration == 2h) — not a set-and-forget constant, since Prometheus derives max as 10% of retention and the 30d cut moved it to 3d, which would have silently stopped every upload.
• Exactly one compactor runs cluster-wide, placed by Pacemaker like cinder-volume — two against one bucket corrupt it. The bootstrap window where Pacemaker is up but the compactor has no config is closed by a mask/unmask handshake (config_pacemaker masks, config_prometheus unmasks, SetupCluster creates the resource from CommitLast). The openvswitch mask moved out of bootstrap_cube_config into the same place; not a behaviour change, since Pacemaker and corosync are disabled in systemd and only the commit pass starts them.
• CONFIG_REQUIRES(prometheus, ceph) — without it Prometheus commits at L8, three levels ahead of Ceph at L11, so the bootstrap would create a bucket in an object store that does not exist yet. Now L12.
• Health checks cover the two new components — thanos ERR 5/6 (store gateway down / not responding, per node) and ERR 7 (compactor not exactly one instance, checked cluster-wide). The compactor's repair asks Pacemaker to replace the instance rather than starting the unit, which would risk a second compactor against the bucket — the exact corruption the singleton prevents.
• Every listener was audited off 0.0.0.0 — each port binds the management address only where something on another node calls it, and loopback otherwise. Prometheus is the only one bound twice (loopback for the sidecar/self-scrape, mgmt for the cross-node health check); --web.listen-address is documented "Can be repeated".
• cube-cos-api no longer shares admin's EC2 credential — the residue of [Bug]: s3 credential secret key is empty, make Health page detail log Access Denied #703. Its accessKey setting is a keystone user name that defaulted to admin, so it minted a credential whose access key is literally "admin", while sdk_health and cube_cluster_start_cluster list admin's credentials, cache the first in /run/ec2.key and delete by that access key. It now gets its own keystone user, reconciled by api_s3_user_setup. No cube-cos-api change was needed. Access to the shared log bucket needs no policy: RGW runs with rgw keystone implicit tenants = false, so the S3 owner is the keystone project and log is owned by the admin project id.

Also landed in #1441 (phase 1, plus what surveying the exporters turned up)

• Four services needed no exporter at all, checked on a live node rather than assumed: haproxy 2.8 is built +PROMEX, rabbitmq 3.11 ships rabbitmq_prometheus (shipped disabled, so enabling it is not a no-op), influxdb 1.12 already serves /metrics, and zookeeper 3.8 has a PrometheusMetricsProvider built in. That is why the exporter list above is five, not nine.
• Provenance was checked against the GitHub API, not the prometheus.io catalogue, which still lists two of these at their old homes. blackbox_exporter and memcached_exporter are prometheus/, ipmi_exporter is prometheus-community/ (moved off soundcloud/), node_exporter is prometheus/ via EPEL. apache_exporter is the one exception — Lusitaniae/, not org-backed; it is what the official catalogue lists for Apache, has no org-backed alternative, and feeds Grafana middleware panels only, so it is the one entry droppable without breaking a consumer. danielqsj/kafka_exporter was rejected on the same test (single maintainer, five months stale).
• Three defaults were wrong and a live cluster is what said so — all three would have shipped silently broken. apache_exporter pointed at port 80, which is the front-end proxy: it answers /server-status with a 302 to https and the exporter reports that as apache_up 0 rather than as an error; httpd's own vhost is 8080. The haproxy job assumed one stats listener when there are two. And the blackbox module pinned valid_http_versions, which made Octavia — whose API answers HTTP/1.0 — report probe_http_status_code 200 alongside probe_success 0, a false failure that would have left health_octavia_check permanently NG once phase 3 moves it onto this series.
• ZooKeeper was attempted and backed out, recorded so it is not tried again. 3.8 does carry a PrometheusMetricsProvider, but the class lives in zookeeper-prometheus-metrics.jar and Kafka's bundled distribution ships only zookeeper.jar and zookeeper-jute.jar — setting metricsProvider.className took ZooKeeper down with status=2/INVALIDARGUMENT on jim-1cc. It needs prometheus/jmx_exporter, as does Kafka, which has no native endpoint at all; one artifact covers both, and it attaches to the JVM rather than being a scrape target, so it is deferred as a separate change.
• jmx_exporter and mysqld_exporter were both scoped and deliberately deferred. jmx was dropped for its memory cost against no downstream consumer that needs it; mysqld_exporter was checked against the consumers and is not required by any of them.
• influxdb is scraped for what it will report, not what it reports today. 1.12's /metrics is 116 lines of go, process and promhttp series with not one influx-specific metric — no shards, series counts, writes or queries, because 1.x keeps those in the _internal database. 2.x exposes them properly and [Story] [Optional] Upgrade Kapacitor from 1.5.7 to 1.8.6 #648 moves us there, so wiring the job now means that lands with no further change.
• A missing EnvironmentFile is not a benign default. The units carry EnvironmentFile=/etc/default/<name> and nothing in the packages creates those files (rpm -qf reports the four tarball exporters' as owned by no package). A missing one does not mean "start with no arguments": systemd refuses the unit with Result: resources and says nothing about the cause. config_prometheus writes each file before enabling its unit, so the ordering is right; the build only guarantees the directory.

Also landed in #1442 (the ceph half of phase 3, brought forward because it was a live defect)

• ceph-mgr no longer reports to InfluxDB, because that route was never HA. The influx mgr module wrote to shared_id:8086 — straight at InfluxDB, bypassing Kapacitor :9092, which is where the peer relay lives — so its points were never replicated. Measured on accept-3cc: the ceph database held 5 measurements on cc2 and zero on cc1 and cc3, while Grafana reads localhost. The mgr's own prometheus module is a pull and is complete on all three. This makes it a defect fix rather than parity work.
• Upgraded clusters need an explicit migration; the A/B switch does not clear it. The module enablement lives in the mon quorum's mgr map and mgr/influx/* in the mon config store, neither on the rootfs, so a rolling upgrade carries both forward. migrate_ceph_mgr_influx disables the module and strips the keys, gated from config_ceph.cpp's Commit() behind a marker the migrate hook drops and clears only on rc 0 — a one-shot per-node marker cannot retry, and this needs a serving mgr a freshly booted node often lacks.
• The mgr prometheus endpoint binds per daemon. mgr/prometheus/server_addr is one value shared by every mgr, and a process can only bind an address that exists on its host — setting a cluster-wide 10.1.0.1 took the endpoint down on all three accept-3cc nodes because the active mgr was cc3. Scoped as mgr.<hostname>, so failover lands on a daemon that can bind; the inherited cluster-wide 0.0.0.0 is removed. exclude_perf_counters=false exposes the per-OSD series the dashboards need — not new collection, since the influx module always shipped exactly those counters, at a measured ~4 ms a scrape.
• The consumers moved with it. Grafana's storage dashboard is on PromQL (21 targets and 2 variables converted, six deprecated table-old panels rebuilt, a latency unit corrected from ns to s), the influxdb-ceph datasource is listed under deleteDatasources — dropping an entry from the provisioning file leaves the row in Grafana's own database on an already-provisioned cluster — and the dead stats_storage_* functions plus join_ceph.tick are deleted. health_ceph_mgr's influx error codes are gone, including one whose repair would have re-enabled the module, and one that was already dead because it read a variable assigned nowhere in the tree.
• InfluxDB is off the wildcard, a year after the first attempt was reverted. #125 bound it to loopback behind haproxy and was reverted the next day by #143 — because of the ceph bypass above. Two corrections to how that revert was recorded: the mgr module is a plain HTTP client, not "performing file operations on /var/lib/influxdb/ceph/"; and the bind change did not create the split, since shared_id was the target before and after feat(security): set up authentication for mongodb and only expose mongodb and influxdb on mgmt ip and vip #125 — it moved which node held the data away from the one serving VIP reads, because that listener balances on source rather than roundrobin. Issue #141 is not the trigger despite the timing: the revert commit predates it by a day. With that writer gone the design is re-landed: InfluxDB binds 127.0.0.1, a per-node haproxy frontend serves the management address, and the VIP listener serves reads but denies /write and /api/v2/write so nothing can rediscover the trap. Reads stay because 42 call sites reach InfluxDB as influx -host $(shared_id), six of them health checks reading monasca http_status — making those local-only would turn one node's dead InfluxDB into six services reporting down.

Phase 2 — per-instance metrics — done in #1448, and it was not the long pole after all

• Per-VM CPU and memory utilisation are published to Prometheus, labelled with the instance UUID (Watcher's instance_uuid_label, default resource). Exactly two series were needed, not a general per-instance capability: ceilometer_cpu and ceilometer_memory_usage. Watcher's _build_prometheus_query implements only four meters and raises Cannot process prometheus meter for the rest, so instance_ram_allocated and instance_root_disk_size are mapped but unsupported and nothing has to produce them.
• Delivered by extending lachesis — delivered as a node_exporter textfile written by hex_sdk watcher_instance_metrics instead. lachesis is scoped to network telemetry and maintained by another team for another project, so putting a libvirt collector in it was the wrong home. The textfile route needs no Go build, no lachesis release, no .mk bump and no new service, port or scrape job — the node's existing node_exporter target already carries the fqdn label. Ceilometer + sg-core was still declined: two services for two metrics.
• No nova lookup is needed for the label. nova sets the libvirt domain UUID to the instance UUID, so virsh list --uuid is already the value instance_uuid_label names, and virsh domstats supplies both figures in one call per domain.
• The units are a fixed contract, not a choice. ceilometer_cpu is a counter of cumulative nanoseconds (Watcher divides by 10e+8); ceilometer_memory_usage is a gauge in MiB of guest-used memory, available - unused, which is what Ceilometer's inspect_memory_usage and Monasca's libvirt check both compute. balloon.rss is a different quantity — 133 MiB against a guest-used 28 MiB on a 128 MiB guest — so using it would move every memory decision by several multiples.

Why this shrank. The original sizing assumed per-instance metrics were both missing and load-bearing. They were missing, but only two of them matter, and the strategies that need them are one: workload_balance. The other verified strategy, allocation_balance (#1128), reads no datasource at all. See #673.

Phase 3 — repoint consumers

• The six health_*_check functions in sdk_health.sh that read Monasca's http_status are served by blackbox_exporter probes instead — done in #1457. Each probe target now carries a service label so the lookup is a name rather than a port number. One behaviour change on purpose: an absent series is no longer a failure. Under Monasca a missing point compared as null != 0 and read as "endpoint unreachable", so a metric pipeline that was itself down turned all six checks NG and pointed the operator at six innocent services.

• Grafana's influxdb-monasca and influxdb-monasca-flux panels are rebuilt against Prometheus — done in #1457. Instance and Top Instances, 23 panels and three template variables, against the pinned Dashboard1 uid. Both datasources are listed for deletion rather than just dropped, because removing an entry from datasource.yaml leaves the row in Grafana's own database on an already-provisioned cluster. Two deliberate changes: rates are computed by rate() rather than stored pre-divided, and the Top Instances tables sum across devices instead of averaging them, which is what a "top bandwidth" table is actually asking.

• The two per-project Kapacitor templates (tpl_alert_vm_cpu, tpl_alert_vm_mem) are re-sourced off monasca.def — done in #1457, now telegraf.def. Kapacitor cannot read Prometheus, so the same libvirt read also posts influx line protocol to the Kapacitor write proxy under Monasca's exact measurement, tag and field names; the templates changed by one line and sdk_alert.sh's alert_vm_event_list, which scrapes the thresholds back out with awk, keeps working untouched.

  This also corrected a handbook entry that would have sent the next author the wrong way. kapacitor-stream-tasks-inert-without-subscriptions concluded that disabled influx subscriptions make stream tasks inert, and that any new per-VM alert must therefore be a batch task. A stream task is fed by writes arriving at that node's Kapacitor, and haproxy fronts :9092 on the VIP, so a VIP-addressed write reaches exactly one of them. Measured on accept-3cc with subscriptions disabled throughout: cc1's task went processed 0 → 1 on a POST to its own /kapacitor/v1/write, → 2 on its own /write, and stayed at 2 for a write to the VIP. The endpoints are identical; the node is the discriminator, and processed="0" on a node haproxy did not choose proves nothing.

• Watcher reads Prometheus — done in #1448, tracked in [Story] Move Watcher Data Source from Monasca to Prometheus #673. All three verified strategies exercised on both datasources with matching outcomes. Note the failure mode this exposed: a Watcher audit reports SUCCEEDED and produces an action plan whether or not it read a single metric, so audit state is not evidence — the per-host workload numbers are.

Phase 4 — removal

• monasca-api, monasca-agent, monasca-persister and monasca-statsd are removed from the build — done in #1457. core/monasca, config_monasca.cpp, the HEAVY_COMPONENTS entry, the haproxy listener, the InfluxDB database, the Kapacitor relay tasks, the filebeat input, the two Kafka topics, the cli_cluster service entry and the whole hex_sdk surface.

  The venv half of this item is out of scope here — it belongs to [Task] Language - Python upgrade from 3.10 to 3.11 #625 (Python 3.10 → 3.11). Monasca is not the last occupant of /opt/openstack-antelope: ospurge 2.0.1.dev64 is still in it, driven by hex_sdk os_purge_project (core/appfw). project.mk already recorded that and the wording of this item had not caught up; project.mk is updated to name the remaining occupants rather than imply the venv can go. Retiring the venv is [Task] Language - Python upgrade from 3.10 to 3.11 #625's to finish once ospurge moves.

• ib_network and rdt_l3 have been scoped against real hardware before removal — moot, and out of scope. Neither check was ever supported on a shipped CubeCOS configuration, so there was no behaviour to preserve and nothing to scope: removing them takes away something that was never in service. This item was written on the assumption that they were live.

  Recorded because it is useful either way: ib_network's function is already covered should it ever be wanted — node_exporter's infiniband collector is enabled and reports node_scrape_collector_success{collector="infiniband"} 1, with zero series on a VM lab, which is the expected reading with no IB hardware. rdt_l3 has no exporter equivalent and no consumer: its only reader was Watcher's noisy_neighbor through Monasca's L3 getter, and the Prometheus datasource does not implement instance_l3_cache_usage at all.

Review findings on #1457, all resolved

@SekiXu's review raised two blocking items and four secondary ones. Both blocking items were the shape this issue already had a name for — a reference that does not contain the string monasca, or contains it where grep was not pointed — and following that lens found two further leftovers neither of us had listed.

• check_service monasca survived and took the whole system.services batch with it. Dropping it from the array was the prescribed one-word fix, but the document still did not parse afterwards on either lab: the monasca banner was one instance of a general fragility, not the fragility. check_service passes any check's stdout into the JSON telegraf parses, so health_heat_check leaking the openstack client's discovery failure breaks it just as thoroughly. Fixed at both layers — check_service now discards the check's output, and heat's misplaced 2>/dev/null moved onto the command that actually writes to stderr.

• cube-cos-api read the monasca bucket in 12 places — the bucket is named explicitly, so it does not follow the measurement/tag/field preservation that made sdk_stats.sh one word per query. Three companion PRs, merge openapi first:

  repo	PR
  cube-cos-openapi	#112
  cube-cos-api	#656
  cube-cos-ui	#870

  The upgrade case is worse than the blank panel predicted: migrate_monasca_retire deliberately keeps the InfluxDB database so operators can read history, so the queries succeed and silently return data frozen at the upgrade. Verified end to end — binary staged on jim-1cc and all three accept-3cc controllers, cpuUsage/rank/vms and memoryUsage/rank/vms both 200 with live data, and the memory figure (17.47 %) matching Monasca's usable-based definition rather than ceilometer's.

• systemctl stop logstash monasca-persister in cube_cluster_recreate — the unit is no longer built, so a stop naming it exits 5 and prints a failure on the line that also stops logstash.

• The image still shipped the monasca account — and, following the same lens, mon-agent too: monasca-agent's unprivileged account, one line above monasca in all four account files, sharing no substring with it. No name-based sweep would have found the second.

• Two // allow Monasca agents to access comments — rewritten to name apache_exporter's scrape, and then the code under them removed: the per-node Require ip entries are unreachable ACL, since 3a829848 moved httpd to loopback eight months after b0fd95ae added them. Measured on all four nodes; a peer's :8080 does not connect at all.

• A line in stats_inactive_vm_drop explaining why the database-wide drop series where resource_id is still bounded — written as the invariant a future writer needs, backed by SHOW SERIES on both labs showing resource_id on exactly the seven vm.* measurements and nothing else.

hex_cli -v -c cluster check is 28/28 OK on both jim-1cc and accept-3cc after all of it — which required staging the rebuilt hex_cli, because the service-group list is compiled in from cli_cluster.cpp and replacing the sdk shell modules alone leaves a stale binary reporting monasca(1 1 undefined).

Throughout

• Telegraf, the log/event pipeline and the Kapacitor alert loop are unaffected — verified with Monasca stopped, disabled and masked on all four lab nodes. telegraf, logstash, filebeat, kafka, influxdb, kapacitor, grafana, prometheus, thanos, lachesis, watcher and httpd all report ok on both clusters. The Kapacitor loop is more than unaffected: the two per-VM alert templates it carries now fire for the first time.

  The alert loop did touch Monasca, contrary to this line — those two templates streamed from monasca.def, which is why phase 3 has an item for them.

• No regression in what operators can see — every family a consumer reads is still served, cross-checked against the live Monasca series on the same guest seconds apart:

  	Monasca	replacement
  vm.mem.total_gb	0.10746	cube_instance_memory_visible_mb/1024 0.10746
  vm.cpu.total_cores	1	ceilometer_vcpus 1
  100 - vm.mem.free_perc	17.38 %	1 - usable/visible 17.35 %
  vm.net.in_bytes_sec	23.46	rate(ceilometer_network_incoming_bytes) 27.3

  One thing operators see differently, on purpose. Monasca's vm.mem.free_perc and vm.mem.used_gb are computed from balloon.usable (the guest's MemAvailable), not balloon.unused — read from the shipped libvirt check, and it contradicts what feat(watcher): move the decision engine onto prometheus and supply the instance metrics it needs #1448's commit message claimed. Ceilometer's memory.usage is available - unused: 26.2 % against Monasca's 17.4 % for the same instant. Both are published rather than one disguised as the other — ceilometer_memory_usage keeps Ceilometer's definition because it is Watcher's meter and has to survive the datasource patch being dropped on Epoxy, while cube_instance_memory_visible_mb and cube_instance_memory_usable_mb carry the raw guest figures the dashboard gauge and the Kapacitor threshold are defined against.

  Also found by running it rather than reading it: health_httpd_check probed port 8070, which was monasca-api's httpd vhost, in a list of bare port numbers that named Monasca nowhere. Removing Monasca turned every control node's httpd check NG on a healthy cluster.

Verified: the current pipeline is not HA

Measured on accept-3cc on 2026-09-03 with cc3 down (cc1 + cc2 online, quorum held, VIP on cc1). This is the baseline the replacement has to beat, and it is lower than assumed:

Observation	Detail
1 of 3 Kafka partitions has no leader	metrics, logs, audit-logs, telegraf-*, notifications.info, transformed-logs are RF=1 — kafka-topics --describe --unavailable-partitions reports Leader: none for partitions 1 and 4 of every one of them. Only alarms / events (RF=2) keep a leader.
Kafka's own tooling breaks	kafka-consumer-groups --describe --group telegraf_persister times out after 5,393 retries — the leaderless partitions make offset/lag unobservable.
Only 1 of the 2 surviving nodes delivers metrics — WITHDRAWN, this was a measurement error	Re-checked 2026-09-05: all three nodes deliver normally (cc1=10, cc2=11, cc3=10 points in the last 10 min, queried per host). The original reading came from select … from cpu group by host parsed as results[0].series — but the InfluxDB CLI returns one results element per group, so that only ever read the first group. Every group by figure taken that way showed cc1 alone. Query each host explicitly, or iterate every results element.
Reads succeed and return the wrong answer, silently	hex_sdk stats_cluster_cpu_usage returned total=29.27 with a full 60-point chart — a confident cluster-wide figure computed from one node out of three, with nothing marking it degraded.
Log ingestion stopped entirely	Newest OpenSearch index is logs-20260902-*; 0 documents in the last 15 minutes from any host.
The Monasca route is dead	Last monasca point is ~23 h old, while telegraf's is 53 s old.
Kapacitor logs a failed replication write every ~10 s	relay_telegraf_def → influxdb_out4: dial tcp 10.1.0.3:8086: connect: no route to host, each carrying a 10 s timeout, for as long as a node is down.
Prometheus, by contrast, degrades honestly	up reports cc1=1, cc2=1, cc3=0 — the dead node is visible as a failed target while the survivors keep serving.

The distinction that matters for this story: the InfluxDB/Kafka path fails silently and returns plausible wrong numbers; the Prometheus path fails visibly. That is a property worth preserving deliberately, not a side effect.

Caveat on attribution. Only the leaderless partitions, the tooling timeout, and the Kapacitor replication errors are proven consequences of the node loss (RF=1 is structural). The dead Monasca route and the halted log ingestion were observed but not root-caused in this pass, and neither is a simple Kafka-topology consequence.

Correction (2026-09-05). The "only 1 of 2 survivors delivers metrics" row above was wrong and has been withdrawn — see the row for why. Telegraf on cc2 and cc3 was always fine: their lines are present in the telegraf-metrics topic, telegraf_persister shows zero lag on all six partitions, and querying InfluxDB per host returns current data for all three. Nothing was broken; the query was. The halted OpenSearch log ingestion is not part of that correction and still stands — there is no logs-<today>-* index at all, which is independent of how anything is parsed.

Follow-on this implies. Raising the replication factor on the metric/log topics is done in #1430 (see the Kafka criterion above). Giving the read path a way to say "degraded" instead of averaging over whatever it happens to have is still not filed — Prometheus/Thanos now degrade honestly, but hex_sdk stats_cluster_* over InfluxDB still does not.

Out of Scope (tracked elsewhere)

• Watcher's data source moving to Prometheus — [Story] Move Watcher Data Source from Monasca to Prometheus #673.

• Upgrading InfluxDB 1.x → 2.x — [Story] [Optional] Upgrade Kapacitor from 1.5.7 to 1.8.6 #648.

• Moving Kapacitor's alerting to Prometheus Alertmanager and removing Kapacitor — [User Story] Move Kapacitor Capabilities to Prometheus Alertmanager #767.

  Two notes for those two, from the [Spike] Telemetry on Epoxy: forward-port Monasca as a near-term bridge vs. replace with Prometheus #646 survey:

  Kapacitor is not only an alerting component. It is the write proxy in front of InfluxDB — every writer POSTs line protocol to Kapacitor :9092, not InfluxDB :8086 — and under HA it is also the replication layer, since InfluxDB 1.x OSS has no clustering and each relay_<db>_<rp> task fans out to all three nodes with influxDBOut(). Removing Kapacitor therefore also requires replacing the ingest path and the InfluxDB replication mechanism, which is a bigger change than "move the alerts".

  InfluxDB 2 does not by itself force Kapacitor out. What InfluxDB 2 removes is the subscriptions API, which is how Kapacitor stream tasks are normally fed. CubeCOS already does not use subscriptions — kapacitor.conf sets disable-subscriptions = true and writers post straight to Kapacitor's write API, which is precisely InfluxData's documented workaround for InfluxDB 2 ("to use Kapacitor stream tasks, write data directly to Kapacitor using the Kapacitor write API"). Batch tasks (join_ceph, alert_metric_write) use the 1.x-compatibility query API and "require no change". What a v2 hop actually needs is a config change, not a version bump: [[influxdb]] keeps username with the API token placed in password, and DBRP mappings are optional because InfluxDB auto-maps database+RP to a bucket for InfluxDBOut. There is no Kapacitor release that "adds InfluxDB 2 support" — compatibility is provided server-side by InfluxDB 2's 1.x-compatibility API. So dropping Kapacitor is a choice ([User Story] Move Kapacitor Capabilities to Prometheus Alertmanager #767), not a consequence of [Story] [Optional] Upgrade Kapacitor from 1.5.7 to 1.8.6 #648. Detail in #767's comment.

• Cross-replica Prometheus dedup (Thanos/Mimir). Not needed for parity — but parity is a low bar: the current pipeline is not HA (see below), so this should be judged against a real target, not against today.

Story Points: 8

High uncertainty concentrated in one place. Phases 1, 3 and 4 are well-understood packaging and repointing work; Phase 2 (per-VM metrics in lachesis) is unproven and carries the risk. Per the sizing guide this should be split — suggested: one story per phase, with Phase 2 sized only after a timeboxed spike on the collector.

Points	Effort / Resources	Description
1	Low	Low complexity; can be resolved easily.
3	Medium	Moderate workload; requires some focus, but risks are well-managed.
5	High	High complexity; difficult to implement, requiring significant time and effort.
8	Very High	Extreme complexity or high uncertainty. If possible, this should be divided into multiple user stories.

QA Verification

#	Command / steps	Expected outcome
1	On each node: systemctl is-active node_exporter	active
2	ss -lntp | grep -E ':9100|:<exporter-port>'	:9100 is haproxy; the exporter is on its own port, no conflict
3	curl -s "http://localhost:9091/prometheus/api/v1/query?query=up" | jq -r '.data.result[] | "\(.metric.job) \(.value[1])"'	every job, including prometheus itself, reports 1
4	curl -s --data-urlencode 'query=up{job="node"}' http://localhost:9091/prometheus/api/v1/query | jq -r '.data.result[].metric.fqdn'	one entry per node, each value equal to that node's hostname
5	curl -s --data-urlencode 'query=(100 - (avg by (fqdn)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100))' http://localhost:9091/prometheus/api/v1/query	a per-node CPU-used percentage within a few points of top on that node
6	Boot a VM, then query the per-instance CPU metric filtered on its UUID	a value is returned, labelled with that instance UUID
7	hex_sdk health_nova_check (and the other five converted checks) with the service up, then stopped	ok when up; a failure reported when stopped
8	Open each converted Grafana dashboard	panels render with data; no "No data" panels that previously had data
9	On a 3-node cluster, stop one node and re-run #3 on a survivor	remaining targets still 1; queries still answer
10	After Phase 4: systemctl list-units 'monasca-*' and ls /opt/openstack-antelope	no monasca units; the antelope venv is gone
11	hex_cli -c settings | grep -E 'influxdb\.(def|hc)\.rp|prometheus\.rp|prometheus\.thanos\.rp'	five tunings present: 14/7, 7/2, 30, 5, 90
12	journalctl -u prometheus | grep -E 'retention policy'	duration=30d and size=5GiB
13	curl -s http://$(hostname):10904/prometheus/api/v1/stores | jq -r '.data.store[].lastError'	null — the store gateway is serving the bucket
14	hex_sdk cmd -c -v "systemctl is-active thanos-compact"	exactly one node active, the others inactive
15	Stop thanos-store on one node, then hex_sdk health_thanos_check; echo $?	5; after hex_sdk health_thanos_repair it returns 0
16	openstack credential list -f value -c Data | grep -c '"access": *"admin"'	0 on a freshly bootstrapped cluster — the API no longer mints a credential named admin
17	openstack ec2 credentials delete admin then systemctl is-active cube-cos-api	active, and the cube-cos-api credential still present — the delete no longer breaks the API

• QA Status declared at Done — when this issue reaches Done, set the board's
  QA Status field to Not needed or Ready to QA. Never leave it empty on a Done
  ticket; In QA / Verified are QA's own transitions.

Output artifacts (Definition of Done)

Beyond code, docs, and config changes, completing this issue must also deliver:

• Handbook knowledge update — land the durable, team-readable knowledge from this
  work into the bigstack-handbook cubecos kb (kb/cubecos/…) via
  /bigstack-core:save-to-handbook (Topic / Runbook / Known-issue / ADR as fits).
  Diagnosed or fixed a failure? Ship the scenario sidecar (<note>.scenario.yaml)
  beside the note.

  Landed so far — bigstack-handbook #499, merged:

  • Topic — architecture/prometheus-grafana-packaging.md — how both components are packaged, the LTS rules that pick their versions, why Prometheus comes from EPEL, and where the exporters will come from.
  • Known-issue + scenario — known-issues/grafana-angular-panels-died-in-the-12-jump.md — twelve pie charts blank since a CVE bump crossed Angular's removal.
  • Known-issue + scenario — known-issues/prometheus-never-scraped-itself.md — the route-prefix self-scrape 404, and why the tempting fix is the wrong one.
  • Correction — blueprints/prometheus-telemetry-replacement.md — exporter sourcing repointed off the stale packagecloud repo onto EPEL and release tarballs.

  Landed for this phase — bigstack-handbook #512, merged:

  • Topic — architecture/metrics-pipeline-node-loss.md — how the pipeline survives losing a control node: Kafka RF as min(control nodes, 3), why kafka_topic_rf_reconcile is what carries 3.1.10-and-older clusters across the upgrade, the Thanos sidecar/querier layout and its replica dedup label, and the port/route-prefix asymmetry that guessing gets wrong.
  • Topic — architecture/health-repair-safety-rules.md — the failure mode four defects in this wave shared, and the three questions to ask of any health_*_repair. Also records why the operator is never gated, and why a strict check with another consumer is the wrong thing to weaken.
  • Known-issue + scenario — known-issues/influxdb-tsi-rebuild-outruns-start-timeout.md — the per-start series compaction that outruns TimeoutStartSec past ~25G and loops forever, with the measured rebuild times from cube4510.
  • Known-issue — known-issues/hex-sdk-nonexistent-function-dumps-usage.md — hex_sdk answers an unknown function with 23KB of usage on stdout, so a typo'd name in $(...) yields help text rather than an error.
  • Rewrite — known-issues/health-mysql-repair-unbounded-datadir-copies.md — was status: open and documented only the damage; now carries the trigger chain, the journal signature (repeated identical recovered positions), the four-part fix, and fixed-in:3.2.0.

  Landed for this phase — bigstack-handbook #564, merged:

  • Topic — architecture/metrics-retention-and-object-storage.md — why InfluxDB's two retention numbers add rather than being two windows, why Prometheus retention had to move into the config file, and the parts of the Thanos-to-RGW path that are invisible in the config: the sidecar's compaction precondition, retention being the compactor's rather than RGW's, the compactor singleton and its mask/unmask handshake, the commit-level ordering, and the per-consumer port bindings.
  • Known-issue + scenario — known-issues/cube-cos-api-shares-admin-ec2-credential.md — the shared-ownership problem that survived [Bug]: s3 credential secret key is empty, make Health page detail log Access Denied #703's fix, and the RGW project-owner mapping that makes a separate keystone user sufficient without a bucket policy.
  • Extension — known-issues/keycloak-discards-baked-build-options.md — two traps applying that fix to a cluster already crash-looping: a RollingUpdate StatefulSet will not replace a pod-0 that never becomes Ready, so the env var lands in the statefulset and not the pod; and fixing the flag does not undo a realm the flag already half-migrated, which hex_config repair_keycloak cannot repair because it never touches the schema.

  Landed for this phase — bigstack-handbook #566, merged:

  • Known-issue + scenario — known-issues/ceph-mgr-influx-bypasses-kapacitor.md — the one writer that broke the Kapacitor :9092 rule, and the two separate failures it caused: ceph metrics on one node of three, and a year-long block on binding InfluxDB off the wildcard. Records the parts not guessable from the code — that the module state lives in the mon quorum so an A/B switch carries it forward, that mgr/prometheus/server_addr must be scoped per daemon because a cluster-wide address is unbindable elsewhere, and that restarting another node's ceph-mgr unit from this node starts impostor mgrs, which produced two wrong conclusions during the work.
  • Update — architecture/monitoring-pipeline-as-built.md — the Monasca route is not dismantled yet, but three of this entry's statements stopped being true: the ceph-mgr influx collector and its relay pair are gone, the one documented exception to the Kapacitor write rule is closed, and InfluxDB no longer binds the wildcard. Each change is dated to its release so the entry stays usable against a 3.1.20 cluster.

  Landed for this phase — bigstack-handbook #567, merged:

  • Topic — architecture/prometheus-exporter-fleet.md — the fleet as built rather than as planned: the four services that already speak Prometheus and so needed no exporter, why the list is five and not nine, the provenance rule that checks the GitHub API rather than the prometheus.io catalogue, the directional bind rule, the fqdn label as a consumer contract, and the traps that cost live-cluster time (apache_exporter reporting apache_up 0 instead of an error, haproxy's two non-interchangeable stats listeners, a missing EnvironmentFile refused as Result: resources, and zookeeper's provider taking the service down).
  • Known-issue + scenario — known-issues/blackbox-http-version-pin-fails-http10-services.md — a valid_http_versions pin reports probe_http_status_code 200 beside probe_success 0, which reads as a sick service rather than an over-specified module. Octavia answers HTTP/1.0, and phase 3 moves the six health_*_check functions onto these series, so it would have left health_octavia_check permanently NG.
  • Extension — architecture/authoring-hex-config-modules.md — the file-mode rule for generated config files, after the same CodeQL alert was fixed a third time on this train. Corrects an earlier claim of ours that WriteFile was the only shape that clears it: it is that same open() factored out, same flags and mode. Records that cron silently refuses a group-writable /etc/cron.d entry, and lists the 22 alerts still open on develop by module.
  • Correction — architecture/prometheus-grafana-packaging.md — its exporter section was written in the future tense and now points at the as-built topic; the sourcing analysis it contained held.

  Landed for this phase — bigstack-handbook #579, merged. This one is the review findings rather than the fixes: four defects that review caught on fix(logging,network): bound log growth, and let a wedged bond recover itself #1433, feat(metrics): bound influxdb and prometheus retention, and keep the history in ceph s3 #1439, fix(metrics): take ceph-mgr off influxdb, and re-land the influxdb exposure change it blocked #1442 and hex#163, all of which shared a shape — each produced a system that parses, starts, and reports nothing wrong. Three of the four correct or complete statements this handbook already made, so they landed as extensions rather than new entries:

  • Extension — architecture/authoring-hex-config-modules.md — two authoring hazards. A mask/unmask handshake split across two modules is only safe where both of them commit, and CommitCheck predicates are per module: config_pacemaker masked openvswitch and thanos-compact while the unmasks sat in config_neutron and config_prometheus, so a commit carrying only pacemaker.modified, G_MOD(MGMT_IF) or G_MOD(IS_MASTER) masked with nothing left to unmask. Records that the trigger is an ordinary cluster-wide tuning change — ui → api → hex_config apply → hex_config commit, never bootstrap — so it would have masked openvswitch on every control and compute node at once, arming every later restart and the next boot to fail; and the trap in reading those predicates, since each module's modified is its own flag and cancelling the two against each other hides the main case. Second hazard: in a sectioned config format a new section emitted mid-section adopts the previous one's closing lines, which cost bind :80 its /api/ route and its default_backend while haproxy -c called both forms valid.
  • Extension — architecture/health-repair-safety-rules.md — a fourth rule, since the three existing ones assume the check measures something real. A check whose signal is unobtainable on a given topology reads as a permanent fault, and the repair behind it then fires forever on a healthy node. Records the fix as evidence-based rather than as weakening the check, and the corollary that "nothing to measure" is not evidence of health or of failure.
  • Extension — known-issues/bond-stops-carrying-traffic-with-healthy-links.md — the two guards the shipped watchdog was missing, with the scenario sidecar updated in the same commit so its shipped-fix description is not one review behind. The entry's own guard list read as though "a node with no target at all is reported healthy" covered this; it does not, because "the only target exists and never answers ICMP" falls on the other side of that test — and on a single-node cluster the default gateway is the only target there is.
  • Extension — known-issues/maxsize-inert-under-daily-logrotate.md — a second-order effect of that entry's own fix: logrotate runs postrotate before compression, so with delaycompress gone the trim could delete the file logrotate was about to open. Kept as the general shape rather than a syslog detail, because postrotate reads as "after the rotation is finished" and is not.

  Landed for this phase — bigstack-handbook #580, merged:

  • Topic — architecture/watcher-metrics-on-prometheus.md — the four metrics Watcher implements and where each comes from, why its hardcoded names mean metric_map.yaml cannot redirect them, the route-prefix mismatch that 404s a direct Prometheus target, the textfile collector and why not an exporter, the unit contracts, guest-used versus host RSS, and the two failure modes that are silent.
  • Correction — architecture/watcher-optimization-goals.md — repointed off Monasca and metric_map.yaml marked removed, plus a further correction: the entry had established that the host mappings do not resolve, which left the per-instance ones looking sound when in fact they had no producer at all on 3.1.20.
  • Extension — known-issues/masakari-lost-libvirt-python-in-caracal-hop.md — monasca's libvirt check is the same stranded-venv defect one service over, with no health check to catch it, so it sat undetected through a release. The Lesson is widened to "what is left in the old venv, and what does it import".
  • Correction — architecture/watcher-allocation-balance-strategy.md — reads no metric datasource at all, came through the move untouched, and is therefore the control for isolating datasource-caused regressions.

  Landed for this phase — bigstack-handbook #588, merged. The tail of the log-growth work, after @traviswu-bigstack asked on fix(logging,network): bound log growth, and let a wedged bond recover itself #1433 why a specific log grows so fast — the answer turned out not to be verbosity:

  • new known-issues/opensearch-flood-stage-retry-log-amplification.md (+ scenario) — every component behaves as designed and the loop is the defect. OpenSearch marks indices read-only at the 95% flood-stage watermark; the block reaches a client as TOO_MANY_REQUESTS, so it reads as backpressure rather than disk; logstash's opensearch output logs the retry at INFO once per document with the document inline (~60k lines/hour, and a 200k-line sample of the 1.4G generation was 100% that one message); and because path.data and /var/log are one filesystem in the shipped layout, that log eats the headroom needed to clear the block. It ran accept-3cc to 645.7 MB free. Leads with the discriminator between the two 429 sources, since the thread pools look nothing like it (rejected 0, no breaker trips, cluster green). Records that OpenSearch's refusal is deliberately unchanged — the defect was that reporting it cost more disk than the writes it refused. Cross-linked to the advisor's dead-transport entry, the same shape in another product.
  • Extension — known-issues/maxsize-inert-under-daily-logrotate.md — the three things review turned up about this entry's own fix. Hourly evaluation enforces the cap and also cuts the audit window: cubesys.log.default.retention is published in days while rotate counts generations, and every generated config inherits that one global (1 of 62 files sets its own), so a log tripping the cap hourly kept 14 hours. Records maxage as the fix and why maxage alone is not, the reason hex's years-old cron.hourly move never took effect (RHEL dropped that cron job for a systemd timer, and || true hid it), and the *.backup orphans that match no glob so nothing ever reclaims them — 5.04 GiB on one node, 14.3 GiB across three.

  Both entries also record what was nearly mistaken for a cause, which is the expensive part to re-derive: a kapacitor system-partition disk alert does exist and merely has no recipient registered on lab nodes; the 1 GB OpenSearch heap against 663 shards per node is a low ceiling but took no part here; and the obvious way to test retention is wrong, because logrotate never revisits generations beyond rotate.

  Landed for this phase — bigstack-handbook #630, merged. This one is @SekiXu's review findings on feat(monasca): move the last consumers onto prometheus and remove monasca #1457 rather than the removal itself:

  • new known-issues/chatty-health-check-voids-system-services.md — check_service passes each health check's stdout into the JSON array telegraf parses as system.services, so one check writing a stray line breaks the parse on the document: every service's status is lost and alert_host_services stops raising SRV00002W for any of them. A cluster in that state reports nothing and alerts on nothing, which is worse than the check failing, and nothing logs an error — telegraf's exec input just drops the sample. Records two independent ways in, both measured: a check calling a CLI that narrates when the service it queries is sick, and a redirect on the wrong stage of a pipeline (| sort | uniq 2>/dev/null guards uniq, which never writes to stderr, leaving the openstack client that does unguarded). Also records what is deliberately not the same bug, so the next reader does not churn twenty $CURL -sf lines that are fine.
  • Extension — known-issues/invisible-references-to-a-retired-service.md — a third witness, and the cleanest example of the class so far: the image shipped two accounts, monasca and mon-agent, and only the first is greppable. mon-agent was monasca-agent's unprivileged account, sits one line above monasca in all four account files and shares no substring with it, so grepping the service name finds one and not the other — which is what happened. Unlike the two existing witnesses it is latent rather than breaking, which is why it earns a place: it isolates the principle without the noise of a failure. Two grep-table rows go with it, the second from verifying rather than writing the removal — cli_cluster.cpp's service list is compiled into hex_cli, so a hand-landed lab with fresh sdk modules still reports the retired service until the binary is staged too.

  Still owed as later phases land: finish the pipeline topic as the Monasca route itself is dismantled (the ceph route is now done), and promote the blueprint from spike maturity.

---

Landed for this phase — bigstack-handbook #592, merged:

• Topic — architecture/per-instance-metrics-collection.md — the collector three subsystems now depend on: the twelve series, why they carry ceilometer_* names that have nothing to do with Ceilometer, why the counters are left cumulative, and the three libvirt memory figures that are not interchangeable — with the measured numbers that make balloon.unused, balloon.usable and balloon.rss three different answers to "how much memory is this VM using".
• Runbook — runbooks/retire-an-openstack-service.md — what the A/B partition swap does and does not take when a service leaves the build, the step that always gets skipped (grep for the retired service's ports, not its name), the line between remnants to delete and data stores to leave ageing out, and why the cleanup is a marker-guarded migrate_* function called from config_keystone's Commit() rather than a CLI command an operator has to remember.
• Correction — known-issues/kapacitor-stream-tasks-inert-without-subscriptions.md — was status: open and blamed disabled influx subscriptions for making stream tasks inert, concluding that any new per-VM alert must be a batch task. The discriminator is which node received the write, not the endpoint or the subscriptions; now fixed-in:3.2.0 with the three-write experiment that separates them. Owner left as travis.wu — the original finding and the subsystem are theirs.
• Correction — architecture/watcher-metrics-on-prometheus.md — three stale or wrong claims: the collector's module and function name after the rename, the cron's home (config_nova.cpp gated on IsCompute, and the old placement was not "deliberate" as the page defended it), and the memory claim above, left as a visible correction paragraph because anyone who tuned a threshold against the old sentence needs to see it change.
• Known-issue — known-issues/retired-service-port-left-in-a-health-probe.md — the 8070 probe, recorded as a class: a health check that probes bare port numbers takes a dependency on every service behind them, and no search for the retired service's name will find it.

No scenario sidecars: the two corrections are misdiagnoses rather than diagnose-to-fix arcs, and the 8070 finding's whole reproduction is one shell loop already in the entry.

Landed for this phase — bigstack-handbook #602, merged. Corrections to the three entries above, from finishing the removal:

• Broadened + renamed — known-issues/invisible-references-to-a-retired-service.md, was retired-service-port-left-in-a-health-probe.md. A second witness of the same class turned up: sdk_stats.sh named the monasca database through $MONASCA_DB in seven places — stats_topten_vm, stats_vm_chart, stats_inactive_vm_drop, which the UI's Top Instances, per-instance charts and series pruning are built on. No port, no service name, so both searches missed it, and deleting the variable left the queries as influx -database with no argument: not a shell error, not a syntax error, just blank panels. The sharp point is that the 59-check health sweep that caught the first witness could not catch the second — stats_* functions are not health checks, so the sweep was green while the UI was blank.
• Correction — runbooks/retire-an-openstack-service.md — step 3 grows from "grep for the ports" to a table of what to grep for (ports, the variable, database and measurement strings, Kafka topics, log paths) with where each hides, plus the rule that a shell variable is the nastiest of them: grep for it first, delete its definition last. The verify section stops implying the health sweep is sufficient.
• Extension — architecture/per-instance-metrics-collection.md — documents the collector's influx half, which grew from two measurements to eight to satisfy the above. The section worth reading is why the same numbers are published twice in two shapes: Prometheus has rate() so counters are right there, InfluxQL has none worth using so Monasca pre-divided its _sec series and every consumer on that side is shaped around receiving them that way.

⚠️ Obsolete — original story (forward-port Monasca)

Superseded by spike #646. The story below planned to keep Monasca by forward-porting its
metrics pipeline to Epoxy as a bridge, deferring Prometheus to 2027 H1. #646 reversed that:
forward-porting was found to be the more expensive option, and a falcon-3 incompatibility
already breaks monasca-api's error path on a shipping build. Kept verbatim for history.

Description

As a CubeCOS operator upgrading to Epoxy (2025.1) release, I want
existing metrics collection, the Watcher data source, and Grafana dashboards to keep working,
so that observability is preserved on Epoxy without waiting on the full telemetry redesign.

Monasca is now retired upstream (all components retired
2025-08-15; no release exists past Bobcat). Because a full move to Prometheus also requires
reworking the Watcher data source, alert path, dashboards, and UI, it is too large for the
Epoxy window. We therefore keep Monasca for this release by forward-porting the scoped
metrics pipeline (we do not use monasca-thresh), and defer the Prometheus replacement.

The long-term migration to Prometheus + openstack-exporter is tracked under epic
#881 (2027 H1). The decision and research behind this split are in spike #646.

Acceptance Criteria

Functional behavior and constraints, not code-level implementation.

• Forward-port the Monasca metrics pipeline to run on Epoxy's shared Python env:
  monasca-common (first), monasca-agent, monasca-persister, and the metrics path
  of monasca-api.
• Metrics flow end to end: agent → api → kafka → persister → InfluxDB.
• Watcher continues to read InfluxDB unchanged (noisy-neighbor / VMware DRS parity intact).
• Grafana dashboards continue to render against the existing data source.
• The fork and applied patches are documented for future re-porting.

Out of Scope (tracked elsewhere)

• Replacing Monasca with Prometheus + openstack-exporter — epic (Draft) [Epic] CubeCOS Simplicity & Predictability Refactor #881 (2027 H1).
• Retiring InfluxDB / Kapacitor and reworking the alert path and UI — epic (Draft) [Epic] CubeCOS Simplicity & Predictability Refactor #881.

Story Points: TBD

Pending the live-cluster validation in #646. Estimated 5 if the shared-env
oslo/eventlet/Python port is clean (~1–2 weeks); re-assess upward if that spike hits a hard
dependency wall.

Points	Effort / Resources	Description
1	Low	Low complexity; can be resolved easily.
3	Medium	Moderate workload; requires some focus, but risks are well-managed.
5	High	High complexity; difficult to implement, requiring significant time and effort.
8	Very High	Extreme complexity or high uncertainty. If possible, this should be divided into multiple user stories.

[bigstack-ci-bot]

🤖 Board reconciler applied 1 correction(s) to this ticket:

• Status: Todo → Backlog (S3 — Todo means picked into a sprint; set a Planned Sprint at sizing to bring it back)

_{Derived from ground truth (close reason · PR state · Planned Sprint · sprint calendar). If a correction is wrong, fix that ground truth — the reconciler follows it on the next sweep.}