feat(watcher): move the decision engine onto prometheus and supply the instance metrics it needs

[Eandalf-Bigstack]

What type of PR is this?

• feature
• bug

What this PR does / why we need it

Watcher is the last consumer that still reads Monasca. This moves its decision engine onto Prometheus and supplies the only metrics that move did not already have, so Monasca can be deleted without taking Watcher with it. This is issue #673.

• Backport the Prometheus datasource into Caracal. Watcher 12.1.0 predates it; it landed upstream in 2025.1. The two upstream files are taken verbatim, so the backport stays trivially re-portable and drops out entirely on Epoxy
• Publish Prometheus where the datasource can actually reach it. It builds host:port/api/v1/... and python-observabilityclient accepts only a host and a port — there is nowhere to put our /prometheus route prefix, and Watcher validates the host against a hostname regex so one cannot be smuggled in. A loopback haproxy listener prefixes the path
• Supply the two missing series. ceilometer_cpu and ceilometer_memory_usage come from a node_exporter textfile written by hex_sdk watcher_instance_metrics, cronned by config_nova.cpp on compute-capable nodes. No new service, port or scrape job
• Retire the Monasca-shaped patches. clients.py, datasources/monasca.py and metric_map.yaml all go, along with PR #548's unit conversion
• Carry what remains as reviewable diffs, the convention core/nova and core/masakari already use

Which issue(s) this PR fixes

• Fixes [Story] Move Watcher Data Source from Monasca to Prometheus #673 — Watcher reads Prometheus end to end
• Refs [Story] Keep CubeCOS telemetry working by replacing Monasca with Prometheus and exporters #672 — the Watcher item of phase 3, and it answers what phase 2 actually had to deliver

Special notes for your reviewer

The first two commits are a pair and should not be split across releases. The datasource move alone leaves instance_cpu_usage and instance_ram_usage resolving to None, and the failure is silent: get_backend only rejects a datasource when a metric maps to None, but ceilometer_cpu is mapped — it is simply absent from the store. So audits still SUCCEED, still produce action plans, and decide from no per-instance data with nothing logged. The unit change below has the same coupling in the other direction.

Watcher's metric names are not configurable, and metric_map.yaml cannot help. METRIC_MAP hardcodes instance_cpu_usage -> ceilometer_cpu and instance_ram_usage -> ceilometer_memory_usage, and _build_prometheus_query then dispatches on the meter name, raising Cannot process prometheus meter for anything it does not recognise. metric_map_path does override the map (manager.py:182), but renaming a Prometheus meter only breaks the dispatch. That is why the collector emits the ceilometer names rather than our own: it keeps the backported datasource byte-identical to upstream. metric_map.yaml is removed rather than repointed, and the option's default path is safe to leave dangling because upstream's load_metric_map guards with os.path.exists.

Why × 1024.0 had to go, and why it is coupled. PR #548 added it because Monasca's mem.used_gb is GiB while workload_balance compares against node.memory in MiB. ceilometer_memory_usage is already MiB. Measured on the same node: 424.446 % with the multiply still deployed, 0.4145 % without. Land the datasource without the strategy change and memory reads 1024× high; land the strategy change without the datasource and the Monasca era reads 1024× low. Neither fails loudly — both just migrate wrongly.

Memory is the guest's own figure, not the host resident set. available - unused, which is exactly what Ceilometer's inspect_memory_usage computes for memory.usage and what Monasca's libvirt check computes for vm.mem.used_gb — the series metric_map.yaml pointed instance_ram_usage at, so an existing threshold carries over unchanged. balloon.rss is a different quantity that Monasca publishes separately as vm.mem.resident_gb; on a 128 MiB guest it measured 133 MiB against a guest-used 28 MiB. Where a guest has no virtio-balloon neither field is reported and the series is omitted, as Monasca omits it, rather than silently substituted.

Why a textfile and not an exporter. No libvirt exporter passes the provenance bar the phase 1 fleet was chosen under: the upstream one has had no commit since June 2021 and prometheus-community declined to adopt it — a harder fail than the kafka_exporter already rejected. The textfile collector needs no new service, port or scrape job, and the node's node_exporter target already carries the fqdn label. No nova call is needed for the instance label either: nova sets the libvirt domain UUID to the instance UUID, so virsh list --uuid is already the value instance_uuid_label names.

Addressing review — where the cron lives. It now sits in config_nova.cpp, gated on IsCompute, because it reads the local libvirt domains and so belongs to the module that runs where the domains are — the same gate nova's own compute services use a few lines below. It was in config_prometheus.cpp for a weaker reason: config_watcher.cpp returns early unless IsControl while the instances live on compute, and config_prometheus.cpp's Commit() happened to run on every role. That made it work rather than made it right, and it wrote a cron on control-only and storage nodes where the generator has no virsh. config_prometheus.cpp keeps only what is genuinely its own: node_exporter's --collector.textfile.directory flag and the scrape. The writer stays in sdk_watcher.sh, since Watcher is its only consumer and hex_sdk resolves the module from the function's first token.

Addressing review — the patch format. The three modified files are now unified diffs at <rel>.py.patch beside the pristine <rel>.py.orig, with the three brand-new files installed verbatim: the core/nova and core/masakari convention. The old cp -rf overlay took 617 lines of whole-file copies to express 12 hunks, and would have silently overwritten a future 12.2.0 file with 12.1.0 content on a version bump. One deliberate difference from nova and masakari: both guard the apply with || exit 1 and rely on --forward for idempotence, but --forward only skips the hunks — measured here, it still exits 1 when every hunk is already applied, so a re-run against an already-patched tree aborts the build. It never fires for them because each build reinstalls the venv from pip first; it would fire in an incremental workspace. A --dry-run --reverse pre-check detects that case and skips it. Say the word if you would rather match nova byte-for-byte.

What upstream is doing, for the record. The Prometheus datasource is deprecated as of 2026.1 in favour of Aetos, a Keystone-authenticated RBAC proxy in front of Prometheus, and the default watcher_datasources has changed accordingly. Backporting the deprecated datasource is still right for us — Aetos is an additional service plus a Keystone endpoint, and our path is yoga → antelope → caracal → epoxy — but the next Watcher uplift should revisit it. Upstream also assumes Ceilometer + sg-core produce the instance metrics, which is why the names are ceilometer_*; we deliberately do not adopt those two services for two metrics.

Additional documentation

Verified on accept-3cc (3-node HA) and jim-1cc (single control) unless stated otherwise.

For the requirement "the backport loads in Caracal":

datasources registered: ['gnocchi', 'ceilometer', 'monasca', 'grafana', 'prometheus']
check_availability:     available

For the requirement "the datasource can reach Prometheus at all", which is what the haproxy listener exists for:

127.0.0.1:9091/api/v1/query             -> HTTP 404   (route prefix)
10.1.0.1:10904/api/v1/query             -> HTTP 404   (thanos querier, same)
127.0.0.1:9091/prometheus/api/v1/query  -> HTTP 200
127.0.0.1:9099/api/v1/query             -> HTTP 200   (the listener)

For the requirement "host metrics resolve", which needed no new collection:

jim-1cc      host_cpu_usage 45.21 %   host_ram_usage 32090844 KiB   (32 GB box)
accept-3cc   cc1 50.81 % / 28.92 GiB   cc2 34.89 % / 27.57 GiB   cc3 32.56 % / 25.61 GiB
             cc1 free -g: 28 GiB used

All three fqdn labels are discovered through the thanos queriers on the HA cluster.

For the requirement "the collector produces the two missing series", on the node hosting the instance:

ceilometer_cpu{resource="e3b8df03-68bb-49af-94a5-7df427079253"} 219945287000
ceilometer_memory_usage{resource="e3b8df03-68bb-49af-94a5-7df427079253"} 28

and nothing on the nodes with no domains. Cross-checked against libvirt directly:

virsh dommemstat -> available=112676 KiB  unused=83196 KiB
                    (112676-83196)/1024 = 28 MiB          == the emitted value
cpu.time         -> 439926000 ns over 181 s on 1 vcpu
                    = 0.243 %                              == what the datasource returned

For the requirement "the strategies behave as they did on Monasca", the same three audits run on both datasources. Node 26e1681d is the one hosting the instance:

                     host_cpu_usage_percent   host_ram_usage_percent
monasca baseline     0.0 (idle VM)            0.0594
prometheus           0.0244                   0.0873

Both memory figures are guest-used; they differ only by sample moment (19.1 MiB vs 28 MiB of a 128 MiB guest). The other two nodes report 0.0 throughout, having no instances. Audit outcomes, identical across both datasources and unchanged after the review refactor:

workload_balance  metrics=instance_cpu_usage   SUCCEEDED
workload_balance  metrics=instance_ram_usage   SUCCEEDED
allocation_balance                             SUCCEEDED, action plan RECOMMENDED (1 migration planned)

allocation_balance is unaffected by the move by construction — its own header reads "compute model only, no metrics datasource".

For the requirement "removing metric_map.yaml breaks nothing", with the file deleted on all three nodes:

decision engines:        active on cc1, cc2, cc3
metric_map complaints:   0 on all three
the three audits:        SUCCEEDED
readings:                unchanged at 0.0873 / 0.0244

For the requirement "the patch format applies and re-applies", against a tree reconstructed from the .orig files in the jim build container:

first run:   all three patches apply cleanly              rc=0
second run:  every hunk reported already applied, skipped rc=0
verbatim:    prometheus.py, prometheus_client.py, allocation_balance.py installed

For the requirement "it builds": clean CXX under -Wall -Werror and a clean LD hex_config for config_nova.o, config_prometheus.o, config_watcher.o and config_haproxy.o, with the cron literals now in config_nova.o and zero references remaining in config_prometheus.o.

Not verified here, in two places. An instance without virtio-balloon, where the memory series is omitted by design — the lab guests are Cirros images that all report balloon statistics. And the control-only path where the cron should be absent: every node on both lab clusters is control-converged, so IsCompute matches everywhere and no topology available here reaches the unlink branch.

[traviswu-bigstack]

Let's use .diff for all following openstack patches and drop the .py file

[SekiXu]

Verified the three claims this PR rests on against the sources rather than taking them from the body.

The patch convention checks out, byte for byte.

• All three .orig files are byte-identical to upstream python-watcher 12.1.0 (watcher/conf/__init__.py, watcher/decision_engine/datasources/manager.py, watcher/decision_engine/strategy/strategies/workload_balance.py), so "pristine" is literally true and a 12.2.0 bump will conflict loudly instead of being absorbed.
• datasources/prometheus.py and conf/prometheus_client.py are byte-identical to upstream stable/2025.1. The "taken verbatim" claim holds, which is what makes the backport droppable on Epoxy.
• All three .py.patch files apply to their .orig at rc=0. On a second run the patch --dry-run --reverse pre-check in core/watcher/watcher.mk correctly reports every one as already applied and skips it — so the deliberate divergence from core/nova's || exit 1 + --forward does what the comment says it does, and an incremental workspace no longer aborts the build.
• Applying the three patches reproduces develop's whole-file copies with exactly one difference: the * 1024.0 removal in workload_balance.py. Nothing else changed under cover of the format switch.

python-observabilityclient is pinned — ===0.1.1 is in core/heavyfs/os-caracal-pip-upper-constraints.txt, so the -c on the pip install covers it and the unpinned-looking name in watcher.mk is not actually unpinned.

Three non-blocking notes. The first two are already fixed by #1457, so they only surface if this ships on its own:

1. core/sdk_sh/modules/sdk_watcher.sh — the # HELP ceilometer_memory_usage Resident memory of the instance in megabytes. line contradicts both the code (available - unused) and the file's own header, which explicitly rejects balloon.rss because it is the resident set. It also says megabytes where the header says MEBIBYTES. #1457 rewrites this line correctly.
2. Same file — $(( (avail - unused) / 1024 )) truncates to whole MiB (28.8 → 28). On the 128 MiB lab guest that is nearly a percentage point. #1457's awk does this in floating point.
3. Still present in both: a virsh list timeout is indistinguishable from "no running domains" — the for loop just yields nothing, and the good textfile is then overwritten with headers only. rate()/avg_over_time absorb a one-sample gap so the impact is small, but a failed read and an empty host are worth telling apart.

Style nit: the watcher_prometheus frontend is emitted above the comment block in config_haproxy.cpp that says "Add new sections here, at the end." It parses fine and it is after both cube_cos frontends, so it does not hit the hazard that comment describes — but the instruction and the placement now disagree.

LGTM. This one should merge first; #1457 is stacked on it.