feat(metrics): scrape the phase 1 prometheus exporters, with the fqdn label Watcher keys on - #1441
Merged
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
SekiXu
approved these changes
Sep 8, 2026
SekiXu
left a comment
SekiXu
left a comment
Contributor
There was a problem hiding this comment.
LGTM.
Went looking for the two things that usually bite in this area; neither is a problem here:
- Unauthenticated
/metricson the stats listeners is consistent with what the tree already does.ceph_prometheus_epproxies the ceph mgr/metricson VIP:9285 with no auth in the same file's srvlist table. _prometheus_targets_writeis a faithful generalization oflachesis_prometheus_targets: sametimeout 20 cubectl node list -j, same "empty output means etcd said nothing, so fail rather than truncate a live list", same.tmp+cmp -swrite. Ports appearing in both the module and the sdk is how 8086 / 9092 / 9095 / 2181 / 11211 already work here.
Commit()s IsUndef(s_eCubeRole) guard means the two unconditional SystemdCommitService(true, ...) calls only reach nodes that have a role, which is what the comment above them claims.
Establishing what does not need an exporter, on a live node rather than by assumption, is worth as much as the four that do. The zookeeper note in exporters.mk in particular saves the next person a day: "3.8 has a PrometheusMetricsProvider but Kafka's bundled distribution does not ship the jar" is exactly the kind of dead end that gets re-attempted.
… to 14d InfluxDB kept 364 days of metrics on the system partition -- the same partition as the OS, ceph and prometheus -- with a 35-day shard group. Because InfluxDB never deletes individual points but drops whole shard groups, and only once the entire group is older than the duration, a point written at the start of a group survived duration + shard duration: up to 399 days, reclaimed five weeks at a time. Nothing else bounded it; the influx-curator cron only drops series belonging to VMs that no longer exist. The four durations become TUNING_PUB tunings so a cluster can be sized without a rebuild, defaulting to def 14d/shard 7d and hc 7d/shard 2d. Duration and shard duration are exposed and documented as a pair on purpose -- shortening the duration alone leaves the old shard width, which caps how tight the real bound can get and keeps freeing disk in cliffs. The new pairs give a worst-case point age of 21d for def and 9d for hc, against 399d before. The two policies are a cardinality split, now written down in the code: 'def' carries series whose count scales with node count (host cpu/mem/disk, ceph_*), 'hc' carries sflow and vrouter.top, whose count scales with traffic -- one series per flow tuple -- routed there by telegraf's namepass and logstash's telegraf-hc-persister writing ?rp=hc. That is why the two get very different retention. CreateDBs clamps a shard duration wider than its policy and logs it, rather than letting the commit fail: InfluxDB rejects such a policy, and one that got through could never drop anything. UPGRADE NOTE. On the first commit after this change, expiry is recalculated as end_time + the new duration, and any shard group already past it is dropped WHOLE -- so an operator loses up to a shard-width more history than the new number suggests, in one step. On the test clusters that discarded a full 35-day group rather than only the excess over 14 days. Conversely the new shard width applies only to groups created after the change, so until the current group rolls over, def still holds up to 49 days despite reading 14d. Verified on jim-1cc and all three accept-3cc nodes: policies read def=336h/shard=168h and hc=168h/shard=48h, no data older than 14d remains, and /var/lib/influxdb went 125M->58M on jim-1cc and 279M->122M, 316M->147M, 265M->124M on accept-3cc -- about 54% on every node. Compiles and links clean in the build jail with all four tuning names present in the object. Signed-off-by: Jim Lin <jim.lin@bigstack.co> Co-authored-by: Eandalf <clinah@connect.ust.hk> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Retention was a hardcoded 90d with no size limit at all (storage.tsdb.retention.size=0B), which is the same unbounded shape influxdb had at 364d and on the same system partition -- shared with the OS, ceph and the influx TSDB. Time alone does not bound disk: if cardinality grows, 90d of series is however large it turns out to be. Both limits are now TUNING_PUB tunings, prometheus.rp.duration (30 days) and prometheus.rp.size (5 GiB), so a cluster can be sized without a rebuild. Both apply and whichever is reached first wins. Size 0 omits the setting entirely, which is prometheus's own way of saying no size limit -- writing 0 would cap storage at zero. Prometheus counts in powers of two, so the 5GB we emit is 5 GiB, and it echoes it back as '5GiB' in its startup log. Written into prometheus.yml's storage.tsdb.retention block rather than onto --storage.tsdb.retention.time/.size. 3.13 marks both flags [DEPRECATED] and names that block as the replacement, and a flag would take precedence over the block -- silently pinning the value and making the tunings look ineffective. The block requires prometheus 3.x: a 2.x prometheus does not ignore the unknown field, it refuses to parse the file and never starts. This branch ships 3.13 from EPEL, so that case cannot arise. Verified on jim-1cc and all three accept-3cc nodes, both on 3.13.1: prometheus logs 'TSDB retention updated duration=30d size=5GiB', the service is active with Restart=always and LimitNOFILE=65536, every scrape target including the self-scrape reports up, and thanos still sees all three sidecars. Signed-off-by: Jim Lin <jim.lin@bigstack.co> Co-authored-by: Eandalf <clinah@connect.ust.hk> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sidecar uploads blocks to object storage but nothing read them back:
the querier could only see what each prometheus still held locally, so
anything past the local retention window was unreachable despite being
safely in S3. thanos-store closes that gap.
thanos-compact enforces retention and downsampling in the bucket. Its unit
is installed but deliberately never enabled: two compactors against one
bucket corrupt it, so placement belongs to pacemaker, which runs exactly
one instance cluster-wide -- the same arrangement as cinder-volume.
Both run as prometheus, like the sidecar: no new system user, and they read
the same objstore credentials file. /var/lib/thanos/{store,compact} are
created and owned at rootfs install time.
Verified on accept-3cc: store active on all three nodes with
thanos_blocks_meta_synced{state="loaded"}=12 each, and the querier lists
the store gateway alongside all three sidecars.
Signed-off-by: Jim Lin <jim.lin@bigstack.co>
Co-authored-by: Eandalf <clinah@connect.ust.hk>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
thanos needs a bucket, credentials and an objstore.yml before any of its components can start. That is radosgw-admin and s3cmd work, so it belongs in the sdk rather than in config_prometheus; it is idempotent and runs on every commit. The rgw user is deliberately a native radosgw user rather than a keystone EC2 credential. The admin EC2 credential is contested: sdk_health's log-upload path deletes and recreates it while cube-cos-api recreates a deterministic one of its own, so each rotates the other out and anything caching the pair breaks. A native user is invisible to both and cannot be rotated out from under thanos. os_s3_bucket_quota already drives radosgw-admin directly, so native users alongside keystone-mapped ones are an established arrangement here. The endpoint is a required parameter, not discovered. os_endpoint.snapshot is the obvious source and the wrong one twice over: it is not written until cube_last, six commit levels after prometheus, and a master's first bootstrap skips it entirely, so at commit time it is routinely absent -- measured on accept-3cc, its mtime is 42 minutes after boot. Its "public" row also resolves to EXTERNAL, the wrong side of the appliance for a control-plane upload. config_prometheus already holds the control vip, so it passes <vip>:8888, the same internal endpoint config_swift publishes; that needs only rgw (ceph, L11) and haproxy's radosgw_proxy (L10), both already ahead of prometheus at L12. Keeping the sdk a pure consumer of that value avoids a second source of truth that could drift from the caller's. Verified on accept-3cc: endpoint with bucket and endpoint alone both produce bucket thanos at 10.1.0.100:8888; a call with no endpoint returns 1, writes no config and does not touch rgw. thanos-store, thanos-sidecar and thanos-query restart clean against the result, health_thanos_check passes, and all three sidecars report zero upload failures. Signed-off-by: Jim Lin <jim.lin@bigstack.co> Co-authored-by: Eandalf <clinah@connect.ust.hk> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes items 5 and 6 of the issue 672 comment. Prometheus keeps 30d locally
under a 5GiB cap; everything the sidecar has finished with now lives in a ceph
RGW bucket instead of being dropped, and 90d of it stays queryable.
Three pieces:
- The sidecar gains --objstore.config-file, so finished blocks are uploaded.
This also requires prometheus's own compaction to be off:
max-block-duration = min-block-duration = 2h. The sidecar validates that
itself and refuses loudly rather than shipping bad blocks -- observed as
"found that TSDB Max time is 3d and Min time is 2h. Compaction needs to be
disabled". Prometheus derives max as 10% of retention when unset, so this is
not a constant that could be left alone: the 30d retention change moved it to
3d and would have silently stopped every upload.
- thanos-store is configured and enabled, and added to the querier's endpoint
list. Without it the querier only ever sees what each prometheus still holds
locally, so anything past 30d would be unreachable despite being in S3.
- thanos-compact is configured but deliberately not enabled here. Two
compactors against one bucket corrupt it, so pacemaker owns placement.
Unmasking it once its config exists is this module's half of the handshake
config_pacemaker starts.
Retention in the bucket is a new tunable, prometheus.thanos.rp.duration,
defaulting to 90 days and applied to all three resolutions so the number means
"90d of history" whatever it has been downsampled to. It is enforced by the
compactor rather than by an RGW lifecycle rule: expiry that thanos does not
know about leaves dangling block metadata behind.
CONFIG_REQUIRES(prometheus, ceph) is what makes any of this work at bootstrap.
thanos_objstore_setup needs a live radosgw to create its user and bucket, and
without the dependency prometheus commits at L8 -- verified against hex_config
-d, cube_scan is L7 -- three levels ahead of ceph at L11. It now commits at L12.
The endpoint passed to the sdk is <shared_id>:8888 rather than something
discovered, because os_endpoint.snapshot is not written until cube_last.
While adding these listeners, every port this module writes was audited and
none is left on 0.0.0.0. A port binds the management address only where
something on another node calls it, and loopback otherwise:
9091 prometheus 127.0.0.1 + mgmt sidecar, self-scrape and the non-HA
haproxy backend are local; the health
check probes across nodes. The flag is
documented "Can be repeated".
10901 sidecar grpc mgmt every node's querier dials it
10902 sidecar http loopback nothing reads it
10903 query grpc loopback no querier fan-out, no thanos-rule
10904 query http mgmt haproxy backend, and the health check
10905 store grpc loopback only that node's own querier
10906 store http mgmt the health check probes across nodes
10907 compact http loopback nothing reads it; the singleton is
established through systemd, not HTTP
The compactor's scratch dir is /var/lib/thanos/compact, on the root filesystem
rather than /store: /store is the cross-version upgrade share, not a working
directory. thanos has no flag capping that directory; what bounds it is the
concurrency defaults, one compaction group at a time.
Verified on accept-3cc (3 control nodes): bucket created, objstore.yml valid on
all three, prometheus/sidecar/store/query all active, the querier lists three
sidecars and the store gateway with no error, zero "Compaction needs to be
disabled" messages and zero upload failures. The compactor completed full
cycles as a pacemaker singleton, producing level-2 blocks that each merge three
consecutive 2h level-1 blocks. Every cross-node path answers on all three nodes
and every loopback-only port is refused off-box; the self-scrape is up on all
three and grafana's route through the VIP returns 200. Verified on jim-1cc that
the non-HA path is unaffected: no thanos units active, prometheus healthy.
Signed-off-by: Jim Lin <jim.lin@bigstack.co>
Co-authored-by: Eandalf <clinah@connect.ust.hk>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two compactors against one thanos bucket corrupt it, so exactly one must run cluster-wide. That is placement, which is pacemaker's job -- the same shape as cinder-volume: a plain systemd resource with no colocation, so pacemaker runs it wherever it can. The unit is never enabled in systemd, or every control node would start one. That leaves a window during bootstrap where pacemaker exists but the compactor has no configuration yet, so this masks it before bringing pacemaker up and config_prometheus unmasks it once it has written the config. The masking precedent is openvswitch, which has used exactly this handshake; the resource itself is created from CommitLast at L18, well after both unmasks, so by the time pacemaker is told about thanos-compact the unit exists and is runnable. Commit order, from hex_config -d: pacemaker L9 masks, prometheus L12 unmasks thanos-compact, neutron L13 unmasks openvswitch, pacemaker_last L18 creates the resources. failure-timeout="60s" matters here specifically. Because the resource is masked when pacemaker first comes up, pacemaker will see it fail a few times on the way through a bootstrap, and without the timeout those failures never age out and the resource stays stopped. vaw already carries the same meta for the same reason. The openvswitch mask moves here from the top of bootstrap_cube_config so both masks live in one place. It is not a behaviour change on any path that matters: bootstrap_cube_config is a PROJ_BOOTSTRAP script and ran the mask on every boot whether or not a commit followed, while pacemaker and corosync are disabled in systemd -- verified on accept-3cc -- so nothing starts pacemaker except the hex_config commit pass that now performs the mask a few lines earlier. Commit() has already returned for any role other than control or compute, which is exactly the set config_neutron's OvnService unmasks openvswitch for, so the mask and the unmask cover the same nodes; the old unconditional mask in the boot script could leave other roles masked with nothing to unmask them. Verified on accept-3cc: pacemaker places thanos-compact on cc1 with cc2 and cc3 inactive, systemd reports it disabled on all three, and it logs complete compaction cycles -- initial sync, garbage collection, compaction, both downsampling passes and retention apply. Stopping it cluster-wide is reported by health_thanos_check as error 7 and recovers on pcs resource enable. The resource is created inside the HA branch of SetupCluster, so a non-HA node never creates it and simply leaves the unit masked. Signed-off-by: Jim Lin <jim.lin@bigstack.co> Co-authored-by: Eandalf <clinah@connect.ust.hk> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The existing thanos check covered the sidecar and the querier. Neither notices the two components this branch adds, and both fail silently. Errors 5 and 6 cover the store gateway, per node. It is what makes anything past local prometheus retention readable at all: with it down the querier still answers every existing probe, just from whatever is still on disk, so a cluster can quietly lose all its history in object storage while reporting healthy. Repair restarts it, alongside the sidecar and querier restarts already there. Error 7 covers the compactor, and is checked cluster-wide rather than per node because pacemaker places exactly one instance anywhere. Both directions are faults: two would corrupt the bucket, none means retention and downsampling have silently stopped and the bucket grows without bound. The compactor's repair deliberately does not restart the unit. Starting it directly would either fight pacemaker or, worse, bring up a second compactor against the bucket while pacemaker still believes its own instance is running -- the exact corruption the singleton exists to prevent. It asks pacemaker to replace the instance instead, with pcs resource cleanup, and only from the first control node so three nodes do not issue it at once. Verified on accept-3cc: with everything healthy the check returns 0. Stopping thanos-store on one node returns 5 and health_thanos_repair restores it, with the recheck back to 0. Disabling the compactor cluster-wide returns 7 and it clears once pacemaker is allowed to place it again. The cluster-wide count is taken with the sdk's own cmd -c -v, which reports cc1|0|active against cc2 and cc3 inactive. Signed-off-by: Jim Lin <jim.lin@bigstack.co> Co-authored-by: Eandalf <clinah@connect.ust.hk> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
cube-cos-api and hex_sdk were fighting over one EC2 credential. The API's accessKey setting is a keystone USER NAME, not an S3 key, and it defaulted to "admin", so newBucketSecret() minted a credential whose access key is literally "admin" for the admin user. sdk_health and cube_cluster_start_cluster then list admin's credentials, cache the first in /run/ec2.key and delete/recreate on an empty read -- so `ec2 credentials delete admin` removes exactly the credential the API created, and whichever side runs next silently adopts or destroys the other's key. Observed on jim-1cc, where the only credential in the cluster was the API's and hex_sdk was using it. Issue #703. The API now gets its own keystone user. Nothing in cube-cos-api changes: it already reads accessKey and secretKey from spec.resourceControl.aws, and the template simply stops leaving them at their defaults. Access to the shared "log" bucket survives the split for free. rgw runs with `rgw keystone implicit tenants = false`, so the S3 owner is the keystone PROJECT rather than the user, and `radosgw-admin bucket stats --bucket=log` reports its owner as the admin project id on both test clusters. A user holding an rgw-accepted role on that project therefore authenticates as the same S3 owner, so no bucket policy or ACL is involved. "member" is enough; the role is not what grants the access. hex_config also owns the credential rather than leaving it to the API, which creates it on startup but treats an HTTP 409 as success. Without this, the first time the configured secret changes -- a new seed, or the api.s3.secret tuning -- keystone would keep the old secret while the API signed with the new one, and every S3 request would be denied with nothing in either log explaining why. api_s3_user_setup reconciles instead: it leaves a matching credential alone and only deletes and recreates one that is absent or stale, which also makes it safe to run from every control node. It lives in sdk_api.sh rather than under os_ because the credential shape is the API's own -- the access key must equal the user name -- not a general one. Verified on jim-1cc: converges from no user at all, is idempotent (the credential id does not change on a re-run), and rotates correctly -- after a new secret the old one is rejected, the new one works, and exactly one credential remains. The API comes up active against it and round-trips an object through s3://log, while admin's own credential and hex_sdk's upload path are untouched. Verified on accept-3cc: run concurrently from all three control nodes it still leaves exactly one credential, all three reach the log bucket, all three API instances come up active, and health_api_check returns 0. (Reaching that state needed keycloak repaired first -- its realm was half-migrated behind the XA abort 92e9520 describes -- which is a separate pre-existing fault on that cluster, not something this change touches.) Signed-off-by: Jim Lin <jim.lin@bigstack.co> Co-authored-by: Eandalf <clinah@connect.ust.hk> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…path only
The mask half of the handshake lived in this module, unconditionally, while both
unmask halves live in the modules that own the units -- openvswitch in
config_neutron's OvnService (L13), thanos-compact in config_prometheus (L12).
Those modules' CommitCheck predicates are not this one's, so a commit could mask
without anything ever unmasking:
commits pacemaker but not neutron pacemaker.modified | G_MOD(MGMT_IF)
commits pacemaker but not prometheus pacemaker.modified | G_MOD(MGMT_IF)
| G_MOD(IS_MASTER)
So changing a pacemaker tunable on its own -- or the management interface, or the
master flag -- left both units masked. A mask does not stop a running daemon, so
nothing breaks at the time; what breaks is every later restart. `systemctl
restart openvswitch` fails for as long as the mask stands, the repair paths
included, and because the mask sat above the `if (enabled)` block it landed on
compute nodes too. For thanos-compact the cost is a compactor pacemaker can no
longer start.
Gating on IsBootstrap() is what makes the handshake safe rather than lucky: every
CommitCheck in the pass short-circuits to true under IsBootstrap(), so that is
exactly the set of commits where both unmasks are guaranteed to follow. It is
also the only path where the window exists -- on any later commit both units are
already configured, so there is nothing to protect them from. This restores the
scope the mask had at the top of bootstrap_cube_config, a PROJ_BOOTSTRAP script,
before it moved into this module, with the one difference that Commit() has
already returned for any role other than control or compute -- the same set
OvnService unmasks openvswitch for.
Reported by SekiXu in review on #1439, who derived the whole path from the two
predicates and noted that IsBootstrap() short-circuits both, which is what makes
this the minimal fix. Pairing the mask inside the `if (enabled)` block instead
would not have closed it: a commit that flips pacemaker.enabled to true still
masks and still never reaches an unmask.
Verified on jim-1cc as an A/B of two binaries built from this same source, one
with the guard and one without, each running the identical
`hex_config commit <settings> pacemaker` -- a non-bootstrap commit of the
pacemaker module alone, which is the reviewer's case exactly. WriteLogRotateConf
is Commit()'s last statement, so /etc/logrotate.d/pacemaker is the witness that
the module ran rather than returning at its CommitCheck; the delta was recreated
from a pristine settings copy before each run, because a commit writes its
settings back and the second run would otherwise find nothing modified and prove
nothing.
without the guard witness written, openvswitch/thanos-compact -> masked/masked
with the guard witness written, openvswitch/thanos-compact -> unchanged
Also measured what the leaked mask costs: with both units masked, `systemctl
restart openvswitch` fails with "Unit openvswitch.service is masked" and
thanos-compact will not start, while ovs-vswitchd stays active -- which is why
this would have gone unnoticed. Node left with both units unmasked, settings and
pacemaker enablement as found, ovs-vswitchd and ovn-controller active, br-int
and provider present, and the management IP held.
Refs #1439
Signed-off-by: Jim Lin <jim.lin@bigstack.co>
Co-authored-by: Eandalf <clinah@connect.ust.hk>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sourcing is split three ways, and the largest part of it is discovering how much does not need packaging at all. Four services already speak Prometheus, checked on a live node rather than assumed: haproxy 2.8 is built +PROMEX and carries the exporter as an internal service, rabbitmq 3.11 ships rabbitmq_prometheus, influxdb 1.12 already serves /metrics on :8086, and zookeeper 3.8 has PrometheusMetricsProvider built in. So there is no haproxy_exporter here, and none of the other three either. Of what is left, EPEL carries node-exporter at the current upstream 1.12.1 and none of the others, and no upstream exporter publishes an rpm at all -- every one ships a linux-amd64.tar.gz and nothing else. That is one ROOTFS_DNF line plus the fetch pattern core/thanos already uses, with the fetch in core/exporters/Makefile so the tarballs land in the build tree like any other artifact and a rootfs install is a copy. Provenance was checked against the GitHub API, not assumed from the prometheus.io catalogue, which still lists two of these at their old homes: blackbox_exporter and memcached_exporter are prometheus/, ipmi_exporter is prometheus-community/ (moved off soundcloud/), node_exporter is prometheus/ via EPEL. apache_exporter is the exception -- Lusitaniae/, not org-backed. It is the exporter the official catalogue lists for Apache, actively maintained, and has no org-backed alternative; it feeds Grafana middleware panels only, so it is the one entry that can be dropped without breaking a consumer. danielqsj/kafka_exporter was rejected on the same test: single maintainer and five months stale, where prometheus/jmx_exporter covers Kafka from the official org. node_exporter does not get its registered :9100 because CubeCOS haproxy already binds it for the stats listener, and moving that would change an operator-facing URL. The blackbox module deliberately does not constrain valid_http_versions. Octavia's API answers HTTP/1.0, and a 1.1/2.0 list reported probe_http_status_code 200 alongside probe_success 0 -- a false failure that would have left health_octavia_check permanently NG once it moves onto this series in phase 3. valid_status_codes accepts 200, 300 and 401 because unauthenticated OpenStack API roots disagree: nova and octavia answer 200, glance and cinder 300, and a service demanding a token answers 401. All three prove it is serving, which is what monasca's http_status meant. Verified in the jail: make -C core/exporters fetches and digest-verifies all four and leaves four ELF x86-64 binaries in the build tree. Signed-off-by: Jim Lin <jim.lin@bigstack.co> Co-authored-by: Eandalf <clinah@connect.ust.hk> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The exporter units all carry EnvironmentFile=/etc/default/<name>, and nothing in the package creates those files -- rpm -qf reports the four tarball exporters' as owned by no package. Only config_prometheus.cpp writes them, which is right: the ARGS carry the node's management address, so they cannot be baked into an image. Worth recording why the file must exist by the time a unit starts, since the failure is misleading: a missing EnvironmentFile does not mean "start with no arguments". systemd refuses the unit with Result: resources and says nothing about the cause. config_prometheus already writes each file before enabling the corresponding unit, so the shipped path is ordered correctly and this only guarantees the directory it writes into. Signed-off-by: Jim Lin <jim.lin@bigstack.co> Co-authored-by: Eandalf <clinah@connect.ust.hk> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The exporters run on every node, but only control nodes are in cubesys.control.addrs -- so config_prometheus cannot know the compute and storage membership at commit time, and a node joining must not need a hex_config commit before it is scraped. file_sd solves both, and lachesis already established the pattern here: a generator run from cron, writing one target file per job. Every target carries fqdn as well as instance, and that label is the point of the change rather than a detail. Monasca's per-host series were dimensioned hostname, and Watcher's metric_map keys host_cpu_usage and host_ram_usage on it -- so the replacement has to answer the same question with the same key. Its value is the node's hostname exactly as cubectl node list reports it, never an address. Roles are split the way the services are: node_exporter and ipmi_exporter everywhere, since every role has a kernel and possibly a BMC worth reporting on; memcached and apache only on control, where those services exist. Two behaviours inherited from the lachesis generator because both are deliberate. Empty cubectl output is treated as failure rather than as a zero-node cluster -- it means etcd did not answer, and truncating a live list to nothing would drop every target. And a removed node's target is dropped rather than retained-as-down, because removal is deliberate and a permanent up==0 for it is only noise. Verified on jim-1cc and accept-3cc: the generator is idempotent, and on the 3-node cluster each of the four files enumerates all three nodes with fqdn=cc1/cc2/cc3 against the right management address and port. Signed-off-by: Jim Lin <jim.lin@bigstack.co> Co-authored-by: Eandalf <clinah@connect.ust.hk> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
haproxy needs no exporter. 2.8 is built +PROMEX and reports "Available services : prometheus-exporter", so one use-service line on the stats listener it already binds is the whole of what haproxy_exporter would have been -- and it reports haproxy's own counters rather than re-parsing the CSV stats page. Both listeners get it, because there are two haproxies and they are not interchangeable: haproxy.cfg runs on every control node and fronts that node's own traffic, with stats on 9100, while haproxy-ha.cfg is the pacemaker singleton that owns the VIP, with stats on 9000. Scraping one tells you nothing about the other. Guarded on the request path so the stats UI is untouched, and deliberately outside stats auth: the scrape is local or over the management network, and Prometheus has nowhere to put a password. Verified on jim-1cc and accept-3cc: /metrics returns 1465 series on the local listener of all three control nodes and on the VIP's, haproxy -c accepts the generated config, and /haproxy_stats still answers 200 with its existing credentials. Signed-off-by: Jim Lin <jim.lin@bigstack.co> Co-authored-by: Eandalf <clinah@connect.ust.hk> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Writes each exporter's ARGS, enables it per role, and generates the scrape jobs. The exporters are deliberately not gated on the module's existing enabled flag, which is IsControl. Prometheus itself only runs on control nodes, but node_exporter and ipmi_exporter report on compute and storage too -- that is the whole point of them -- so their config and their units are handled on every role, and only the scrape side is control-only. Ports follow their registered defaults with one exception. node_exporter's 9100 is taken by CubeCOS haproxy's stats listener, and moving haproxy would change an operator-facing URL, so the exporter moves. Each binds one address rather than 0.0.0.0, the same rule the thanos ports follow: the management address where a Prometheus on another node scrapes it, loopback where only the local one does -- which for blackbox is the whole exporter, since it is the probe that crosses the cluster, not the scrape. Two things here were wrong until a live cluster said so, and both would have shipped silently broken. apache_exporter was pointed at port 80, which is the front-end proxy: it answers /server-status with a 302 to https, and the exporter reports that as apache_up 0 rather than as an error. httpd's own vhost is 8080, and config_apache2 already ships the Location block. And the haproxy job assumed one stats listener when there are two -- the per-node local one on 9100 and the pacemaker singleton on the VIP on 9000 -- so it is now two jobs, the per-node one listed from cubesys.control.addrs. The blackbox job probes the same service ports config_haproxy fronts, through the VIP, because that is what monasca's http_check watched -- these are the series the six health_*_check functions move onto in phase 3. It needs relabel_configs rather than plain targets: __address__ has to become the exporter while the original target survives as a label, or every series would be labelled with the exporter instead of the service. Verified on jim-1cc, 14 of 14 targets up, and accept-3cc, 27 of 27 across three control nodes. The fqdn label is present and correct on all four per-node jobs. Watcher's two host metrics are answerable from node_exporter keyed on fqdn: host_ram_usage reads 27.8 GiB on cc1 against free -g reporting 27 used. ipmi_up is 0 on every node, as expected on VMs with no BMC and the same behaviour monasca's ipmi_sensors plugin had there; it needs real hardware to validate. Signed-off-by: Jim Lin <jim.lin@bigstack.co> Co-authored-by: Eandalf <clinah@connect.ust.hk> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
rabbitmq needs no exporter -- rabbitmq_prometheus ships with 3.11 and reports the broker's own counters. It ships DISABLED, though, so this is not a no-op: enabling it alongside rabbitmq_management is the whole of what a rabbitmq exporter would have been. Bound to the management address rather than its default 0.0.0.0:15692, the same rule the prometheus and thanos listeners follow. influxdb is scraped for what it will report rather than what it reports today. 1.12's /metrics is 116 lines of go, process and promhttp series with not one influx-specific metric -- no shards, no series counts, no writes, no queries, because 1.x keeps those in the _internal database. 2.x exposes them properly and issue #648 moves us there, so wiring the job now means that lands with no further change here. Zookeeper was attempted and backed out, which is worth recording so it is not tried again. 3.8 does carry a PrometheusMetricsProvider, but the class lives in zookeeper-prometheus-metrics.jar and Kafka's bundled distribution ships only zookeeper.jar and zookeeper-jute.jar -- setting metricsProvider.className took zookeeper down with status=2/INVALIDARGUMENT on jim-1cc. It needs prometheus/jmx_exporter, as does kafka, which has no native endpoint at all; one artifact covers both and it attaches to the JVM rather than being a scrape target, so it is a separate change. Verified on accept-3cc: the plugin enabled on all three nodes with a rolling restart, each binding its own management address, cluster_status still listing all three running nodes, health_rabbitmq_check and health_kafka_check both 0, and rabbitmq_identity_info reporting from cc1, cc2 and cc3. 33 of 33 scrape targets up across twelve jobs. On jim-1cc, 16 of 16. Signed-off-by: Jim Lin <jim.lin@bigstack.co> Co-authored-by: Eandalf <clinah@connect.ust.hk> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
CodeQL flagged the fopen in WriteExporterTargetsCronJob as cpp/world-writable-file-creation on PR 1441 -- the mode is left to the process umask rather than stated. Same finding and same fix as b9416d3 did for the thanos config files, and 61d0ed8 for config_apache2 before them: WriteFile() from filesystem.hpp opens with an explicit S_IRUSR|S_IWUSR|S_IRGRP|S_IROTH. It matters beyond the static analysis for a cron.d file specifically. cron refuses to run an entry whose file is group- or world-writable, so a umask that ever let this out at 0664 would leave the exporter target lists silently frozen at whatever the commit seeded, with membership changes never picked up and nothing logged. This is the only alert on the branch, and the only review comment on the PR. Verified in the build jail: clean CXX config_prometheus.o under -Wall -Werror and a clean LD hex_config, with the cron literal unchanged in the object -- '* * * * * root /usr/sbin/hex_sdk prometheus_exporter_targets'. Signed-off-by: Jim Lin <jim.lin@bigstack.co> Co-authored-by: Eandalf <clinah@connect.ust.hk> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eandalf-Bigstack
force-pushed
the
jim.lin/feat/replace-monasca-3
branch
from
September 8, 2026 16:26
5a0cf64 to
0bc2f6a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
What this PR does / why we need it
Phase 1 of replacing Monasca: get the metrics Monasca supplied into Prometheus, from exporters the community maintains, so the downstream consumers keep working. No collector of our own is written here.
+PROMEX, rabbitmq 3.11 shipsrabbitmq_prometheus, influxdb 1.12 already serves/metrics, zookeeper 3.8 has aPrometheusMetricsProvider. So there is nohaproxy_exporterand none of the other three. What remains isnode_exporterfrom EPEL plus four tarball exporters fetched and digest-verified incore/exporters/Makefile, installed into the rootfs byexporters.mkARGSis written per node, the units are enabled per role, and the scrape jobs are generated.node_exporterandipmi_exporterrun on every role because that is the point of them; memcached and apache only where those services exist. Every listener binds one address rather than0.0.0.0file_sdfiles, because compute and storage are not incubesys.control.addrsand a joining node must not need ahex_configcommit before it is scraped. Each target carriesfqdnas well asinstance, matching the node's hostname exactly ascubectl node listreports it — Monasca's per-host series were dimensionedhostnameand Watcher'smetric_mapkeyshost_cpu_usage/host_ram_usageon itProvenance was checked against the GitHub API rather than the prometheus.io catalogue, which still lists two of these at their old homes.
apache_exporteris the one entry that is not org-backed (Lusitaniae/); it is what the official catalogue lists for Apache, has no org-backed alternative, and feeds Grafana middleware panels only, so it is the one that could be dropped without breaking a consumer.danielqsj/kafka_exporterwas rejected on the same test — single maintainer, five months stale — in favour ofprometheus/jmx_exporter, which covers Kafka from the official org and is a separate change because it attaches to the JVM rather than being a scrape target.Which issue(s) this PR fixes
Special notes for your reviewer
Three things were wrong until a live cluster said so, and all three would have shipped silently broken.
apache_exporterwas pointed at port 80, which is the front-end proxy — it answers/server-statuswith a 302 to https and the exporter reports that asapache_up 0rather than as an error; httpd's own vhost is 8080. The haproxy job assumed one stats listener when there are two, the per-node local one on 9100 and the pacemaker singleton on the VIP on 9000, so it is two jobs. And the blackbox module pinnedvalid_http_versions, which made Octavia — whose API answers HTTP/1.0 — reportprobe_http_status_code 200alongsideprobe_success 0, a false failure that would have lefthealth_octavia_checkpermanently NG once it moves onto this series in phase 3.Why zookeeper is not here. 3.8 does carry a
PrometheusMetricsProvider, but the class lives inzookeeper-prometheus-metrics.jarand Kafka's bundled distribution ships onlyzookeeper.jarandzookeeper-jute.jar— settingmetricsProvider.classNametook zookeeper down withstatus=2/INVALIDARGUMENTonjim-1cc. It needsprometheus/jmx_exporter, as does kafka. Recorded so it is not tried again.Why influxdb is scraped for what it will report, not what it reports today. 1.12's
/metricsis 116 lines of go, process and promhttp series with not one influx-specific metric — no shards, no series counts, no writes, no queries, because 1.x keeps those in the_internaldatabase. 2.x exposes them properly and #648 moves us there, so wiring the job now means that lands with no further change here.Why
node_exporterdoes not get its registered port. CubeCOS haproxy already binds 9100 for the stats listener, and moving that would change an operator-facing URL, so the exporter moves instead.Why a missing
EnvironmentFilehad to be ruled out. The units carryEnvironmentFile=/etc/default/<name>and nothing in the packages creates those files —rpm -qfreports the four tarball exporters' as owned by no package. A missing one does not mean "start with no arguments": systemd refuses the unit withResult: resourcesand says nothing about the cause.config_prometheuswrites each file before enabling its unit, so the shipped path is ordered correctly; the build only guarantees the directory.Why the blackbox exporter is the one bound to loopback. For the others the management address is where a Prometheus on another node scrapes them. For blackbox it is the probe that crosses the cluster, not the scrape, so the whole exporter is local. Its job needs
relabel_configsrather than plain targets:__address__has to become the exporter while the original target survives as a label, or every series would be labelled with the exporter instead of the service.Additional documentation
Verified on
accept-3cc(3-node HA) andjim-1cc(single control) unless stated otherwise.For the requirement "the exporters are sourced reproducibly", in the jail:
For the requirement "every target is scraped", from Prometheus's own target list:
For the requirement "the label Watcher keys on is present and correct", on
accept-3cc:For the requirement "Watcher's host metrics are answerable from node_exporter":
For the requirement "haproxy reports its own counters, both listeners":
For the requirement "enabling rabbitmq_prometheus does not disturb the cluster", on
accept-3ccwith a rolling restart:ipmi_upis 0 on every node on both clusters, as expected on VMs with no BMC — the same behaviour Monasca'sipmi_sensorsplugin had there. It needs real hardware to validate.The target generator is idempotent on both clusters: a second run produces byte-identical files.