Collect and report per-VM and per-pool storage usage - #1450
Open
traviswu-bigstack wants to merge 1 commit into
Open
traviswu-bigstack wants to merge 1 commit into
traviswu-bigstack wants to merge 1 commit into
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
traviswu-bigstack
force-pushed
the
travis.wu/storage-usage-collector
branch
6 times, most recently
from
September 15, 2026 03:27
35bcbeb to
27ab3db
Compare
This was referenced Sep 15, 2026
Collaborator
Author
Companion PRsThis collector is one of five pieces; the other four are open as drafts:
Merge order is spec → api → ui, with skyline-console independent. This PR carries the Worth flagging from lab validationAll VM metrics were already broken on Prometheus-migrated clusters, not just the new one. Six statements in cube-cos-api still queried Block allocation is not filesystem usage, and cannot be substituted for it. Measured here: the block layer reported 85% on an instance whose guest reported 14.8%. Running |
traviswu-bigstack
force-pushed
the
travis.wu/storage-usage-collector
branch
9 times, most recently
from
September 15, 2026 11:10
82779bd to
3ab1067
Compare
Adds a sampled collector for storage usage, a CLI surface for it, and alarms
for pool capacity and VM filesystem fullness.
Three numbers are collected and deliberately kept separate, because they
differ by large factors in both directions and cannot be blended into one
"used" column: provisioned (nova/cinder), physically allocated (the storage
backend), and in-guest filesystem used (qemu-guest-agent). On one measured VM
those read 171.8G, 37.8G and 21.1G respectively.
Adapters, one per backend, all emitting the same record shape:
ceph rbd du across the four pools CubeCOS actually creates -- ephemeral-vms
(nova root/ephemeral), cinder-volumes, glance-images (shared overhead,
never charged to a VM, or COW clones double-count their parent) and
k8s-volumes -- plus ceph df detail for stored vs raw and the
compression counters. The replication factor is DERIVED per pool, not
assumed: pools legitimately differ, and a hardcoded 3 makes every
figure on a size=2 pool 50% wrong.
nfs qemu-img on the share; sparse raw, not qcow2, per the builtin model.
cinder every driver already publishes pool capacity, so a backend with no
adapter of its own still reports honest totals instead of a blank.
guest qemu-guest-agent guest-get-fsinfo, deduped by device (an entry is
returned per mountpoint, so a twice-mounted filesystem would double)
and with pseudo filesystems dropped.
Attribution enumerates per project: no openstack list command exposes a
volume's or server's owning project at any --long level and -c Project is
silently dropped, so the obvious join always yields null.
Per-VM compression ratio is not obtainable on Ceph or NFS -- BlueStore's
counters are per-pool and there is no per-image stat -- so reduction ratios are
pool-level only. No Cinder driver is patched.
The orchestrator is a telegraf exec input on the VIP-owning control node
only. Every other periodic hex_sdk sampler on the node is one -- twelve of
them, one already with a 5m timeout -- and the delivery path they share
(kafka -> logstash -> the kapacitor write proxy) is the one the alert
templates read. So this prints line protocol on stdout under `-f line`, the
way network_virtual_router_stats does, and telegraf delivers it; nothing here
posts to a database itself. A cron entry would have been a second scheduling
mechanism for no reason other than this feature.
The lock records its holder's pid. A run killed before it can clean up -- a
telegraf timeout, a SIGPIPE from a piped read, any SIGKILL -- leaves the
directory behind, and a lock that is never released silences the sampler for
good. Found exactly that way: piping a manual run into `head` killed it, and
every run for the next five minutes did nothing and said nothing.
The orchestrator runs on the VIP-owning control node only. This
cannot use is_vip_active(), which reports the resource Started anywhere and is
therefore true on every control node; ownership is a local test against the
node name, and a node with no pacemaker counts as owner.
Writes go to the Kapacitor write proxy on :9092 rather than to InfluxDB
directly. That one ingress both relays the write out to the peer control nodes
and feeds the stream tasks the alert templates are built on, so the collector
needs no fan-out of its own; writing to :8086 reaches neither, and is the
bypass that left ceph metrics on a single node (#672). `influx -execute
"INSERT ..."` is unusable regardless -- it takes exactly one line of line
protocol, so a batch writes only its first line.
Everything lands in telegraf.def, alongside the cpu and memory series the
monasca retirement moved there (#672 phase 3/4, #1457). vm.disk.usage_perc is
in-guest filesystem percent from the guest agent, distinct from that
collector's libvirt-derived vm.io.*_bytes_sec -- different source, different
meaning, no name collision.
_storage_usage_influx_write duplicates the POST that sdk_instance.sh's
_instance_metrics_ship makes. Sharing it would mean promoting a helper into
modules.pre, because hex_sdk resolves a module by globbing sdk_<MOD>*.sh and a
storage_* function cannot see a private function in sdk_instance.sh. Left as a
follow-up rather than editing a freshly merged module from this PR.
The NFS adapter falls back to reading nfs_shares_config out of the backend's
own ext_storage conf when cinder_get_storage fails. That call fails outright
for a backend with no extra-config ownership file, which the storage API
writes -- so a backend created any other way has none, and the entire backend
went unreported with nothing said. Found by exercising the adapter against a
real NFS-backed volume rather than a mock.
Alarms fire on threshold crossings with hysteresis, never per sample, or a
15-minute timer produces ~96 events a day and the event table gets ignored.
Over-subscription is a separate alarm from fullness, because a pool 60% full
with 4x provisioned is the one that ends in an outage. The VM filesystem alarm
evaluates per filesystem rather than on the per-VM sum -- an empty 500G /data
beside a full 10G / averages to ~2% while the VM is broken -- and floors out
filesystems too small to be actionable, deliberately below /boot.
The VM disk alarm streams one point per thing that can fill up, carrying the
best reading available for it: the guest filesystem where the agent answers,
the block layer where it does not. Without the fallback an instance with no
agent raises no disk alarm at all -- it never appears, and nothing says so.
Verified on a cluster: an agentless CirrOS instance went from zero alarm
points to DSK00005W at 88%.
Block rows are emitted only for an instance with no guest reading at all,
never beside one: the two measure different things and a disk reported twice
would alarm twice. A `source` tag says which layer answered, and a block row
carries its disk in mountpoint -- the field the alarm groups on and names --
so one template serves both.
The VM alarm ships as a kapacitor stream task on telegraf.def, the same shape
and ingress as the shipped cpu and memory templates. Subscriptions are
disabled on every influxdb target deliberately -- Kapacitor sits in the write
path instead of subscribing -- so a task is fed by what arrives at the write
proxy, which is why the collector posts there.
Tunables are read through hex_tuning at the point of use, the way sdk_power
and sdk_banner read theirs, rather than through a file of our own that
hex_config writes and the sampler sources. settings.txt records only what was
changed from a default, so each fallback in the shell is the default declared
in config_cinder.cpp -- two copies of one number, against a second mechanism
for something the tuning system already does, and one fewer root-written file
to get the mode wrong on.
That file was also hiding a dead tunable: storage.usage.exact is read by
storage_usage_ceph_disks, which runs before storage_usage_thresholds, and the
env file was sourced only by the latter. Setting it had no effect at all.
Measured after the change: 12s for an exact pass, 2s with it off.
One collection, two outputs. Grafana reads Prometheus, so the same pass also
writes a node_exporter textfile -- the influx line protocol exists only because
Kapacitor cannot read Prometheus and the alarm needs a stream. This follows
instance_metrics_collect's shape: write a .tmp and rename, because node_exporter
reads the directory on its own schedule and would otherwise serve a half-written
file as real samples. A null is omitted rather than written as 0: an absent
series reads as "not measured", a 0 reads as "measured, empty".
Ceph pool figures come from ceph-mgr's own exporter rather than a second
`ceph df detail`. Prometheus already scrapes it as job "ceph" -- 610 ceph_*
families, including stored, bytes_used, max_avail and the compression counters
-- and the numbers agree to three decimals with what this used to recompute.
So the collector reads ceph_pool_* and stops emitting cube_storage_pool_* for
ceph pools, which were the same numbers under a second name. The metadata
series carries the pool name that the value series lacks, and joining them in
PromQL needs a group_left this data does not match uniquely, so the two are
fetched separately and joined on pool_id.
It still falls back to `ceph df detail` when that query fails. A capacity
alarm that goes quiet exactly when the metrics stack is the thing that broke
is worse than a second read. Backends that are not ceph have no exporter at
all and keep reporting through the collector.
The shipped Storage dashboard gains a "Pool Usage" row -- pool stored vs raw
and pool provisioned over capacity -- collapsed by default, as the OSDs and
Pools rows beside it already are, so the landing view stays cluster health. It covered ceph cluster and OSD health and
nothing per-pool. Per-VM figures are deliberately not here: this dashboard is
read per pool and per OSD, and the Instance dashboard already answers the
per-VM question against the VM the operator has open.
205 assertions across 9 self-contained test suites; they mock rbd, ceph,
qemu-img, virsh, openstack and curl, so they need no cluster.
The guest pass also rolls up per cinder volume. A guest filesystem is tied to
its volume through the uuid nova writes into the disk's libvirt <serial>, which
the agent returns on each filesystem's disk[].serial -- the only thing that
maps a guest mountpoint back to a volume. An ephemeral root disk carries no
serial and so produces no volume row, rather than inventing one. This is what
lets a volume be shown by what its filesystem actually uses instead of by what
the pool has allocated, which on a written-once volume reads 100% while the
filesystem is nearly empty.
Ceph usage is read with rbd du --exact, and storage.usage.exact turns that
off. The object map counts an object at its full 8M once anything has been
written into it, and a discard empties an object without removing it from the
map, so the map reports a high-water mark that never falls. On a lab volume
the map said 34G against 1.5G exact, and summed over a pool 87G against the
11.4G ceph df says is stored -- a volume written across once and later trimmed
reads as nearly full forever. Sampling its objects shows why: 11 of 12 exist
at 0 bytes. The scan costs about 18x (6s against 0.3s for a 709G pool), so a
cluster with millions of objects can turn it off and accept the high-water
mark; and a scan that outruns the timeout falls back to the map for that pool
rather than dropping its disk rows, reporting confidence estimated.
Per-instance series are tagged resource_id on the influx side and labelled
resource / project / project_name / name on the Prometheus side, rather than
instance_id / vm_name. That is the convention every other per-instance series
in the fleet already uses, and it is load-bearing in two places: the API's
parseResourceId reads resource_id, and the shipped Instance dashboard binds its
$UUID variable to the resource label. Named the other way the series are
syntactically fine and join to nothing.
The shipped Instance dashboard carries the per-VM figures in its existing
Disks section -- provisioned, allocated, guest filesystem used, thin ratio,
snapshots, filesystem count, per-filesystem usage and usage over time -- every
panel filtered to the dashboard's own $UUID, so they appear on the page an
operator already opens for one VM. They sit beside disk bandwidth and IO rate
rather than in a section of their own, because all of it is that VM's disks.
Its CPU / Memory graph gains a disk series and is renamed to match. Guest
filesystem percent shares the panel's percent axis with cpu and memory, so the
three headline utilisations of a VM are read together on one graph rather than
one of them living somewhere else.
skyline.yaml.in pointed at http://localhost:9091 while Prometheus runs with
--web.external-url=.../prometheus/, so its API is served under that route
prefix and the bare path 404s. Skyline concatenates the endpoint with
/api/v1/query, so every Skyline Prometheus query failed, not just the new
ones. Pre-existing, unrelated to this feature, fixed here because the VM disk
usage column is the first thing to depend on it.
Signed-off-by: Travis Wu <travis.wu@bigstack.co>
traviswu-bigstack
force-pushed
the
travis.wu/storage-usage-collector
branch
from
September 16, 2026 06:19
3ab1067 to
d86f8cd
Compare
traviswu-bigstack
marked this pull request as ready for review
September 16, 2026 06:23
Eandalf-Bigstack
left a comment
Eandalf-Bigstack
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, we would be merging this PR after v3.1.20, into v3.2.0 instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind feature
What this PR does / why we need it
Samples storage usage on a tunable interval, exposes it through
hex_cli, and alarms on poolcapacity and VM filesystem fullness.
Three numbers are collected and deliberately kept as three, because they measure different things
and disagree in both directions: provisioned (nova/cinder), physically allocated (the
backend), and in-guest filesystem used (qemu-guest-agent). Thin provisioning never shrinks, so
allocated stays high after a guest deletes files unless discard passed through end to end; the guest
figure counts nothing outside a mounted filesystem. On one measured VM those read 171.8G / 37.8G /
21.1G — a single blended "used" column would be neither comparable between VMs nor summable against
the cluster total.
One adapter per backend, all emitting the same record shape:
rbd duacross the four pools CubeCOS creates (ephemeral-vms,cinder-volumes,glance-images,k8s-volumes), plusceph df detailfor stored-vs-raw and the compressioncounters.
glance-imagesis shared overhead and never charged to a VM, or COW clonesdouble-count their parent.
qemu-imgon the share (sparse raw, per the builtin model).own still reports honest totals rather than a blank.
guest-get-fsinfo, deduped by device and with pseudo filesystems dropped.Per-VM compression ratio is not obtainable — BlueStore's counters are per-pool and Ceph has no
per-image stat — so reduction ratios are pool-level only. No Cinder driver is patched.
Which issue(s) this PR fixes
Fixes #1449
Special notes for your reviewer
Five decisions worth a look, each of which fails silently if done the obvious way:
sdk_storage.shrather than a new module file.hex_sdkresolves a module byglobbing
sdk_<first-token>*.sh; a second matching file makes[ -e f1 f2 ]error and returnfalse, which would push every
storage_*call — including the existing multipath anddisk-selection ones — down the source-all-modules fallback.
hardcoded 3 makes every figure on a
size=2pool 50% wrong with no error.openstacklist command exposes a volume's orserver's owning project at any
--longlevel, and-c Projectis silently dropped, so theobvious join always yields null.
is_vip_active()cannot express this — it reports the resource Started anywhere, true on every control node, so
using it would run the collector three times over.
:9092, not to InfluxDB. That one ingress bothrelays the write to the peer control nodes and feeds the stream tasks the alert templates are
built on; writing to
:8086reaches neither and repeats the bypass that left ceph metrics on asingle node ([Story] Keep CubeCOS telemetry working by replacing Monasca with Prometheus and exporters #672).
influx -execute "INSERT ..."is unusable regardless — one line per call.Rebased onto the monasca retirement. #1457 removes
MONASCA_DBand its relay task, so theper-filesystem series now lands in
telegraf.defand the alarm is a stream task there — thesame shape and ingress the cpu/memory templates move to in that PR, rather than a parallel
mechanism. Verified on a 3-node lab after the change: 24 disk rows and 30 filesystem rows landing
identically on all three nodes, nothing written to
monasca, and the stream task consuming them(
processed=30, 27 past the tenant filter, alerts delivered).Merge #1457 first — both PRs touch
core/main/cube.mk, and this one assumes the retirement.Both original blockers are now cleared.
hex_configwas built at the cluster's own sha(
d2ee24c5, notdevelop) and hot-deployed withcluster checkunchanged at 27ok; and the NFSadapter has run against a real NFS-backed Cinder volume — which is how the
cinder_get_storagesilent failure below was found.
Why pool thresholds are tunable and VM ones are not. The VM disk alarm hardcodes 75/85/95 in
the tick, exactly like
tpl_alert_vm_cpu,tpl_alert_vm_memand the host disk alarm — none ofthose declare threshold vars, and
alert_vm_event_listscrapes the literals back out for display.The pool alarms (
STO0000x) do take tunables, deliberately: "filesystem 95% full" means the samething everywhere, "pool 85% full" does not, and the over-subscription ratio has no natural constant.
Changed under review. Ceph pool figures come from ceph-mgr's exporter (
ceph_pool_*) ratherthan a second
ceph df detail, and the duplicatecube_storage_pool_*series for Ceph are gone.The sampler is a telegraf exec input, not a cron entry, printing line protocol under
-f linelikenetwork_virtual_router_stats. Tunables are read throughhex_tuningat the point of use, so thebespoke env file is gone — it was also hiding a dead tunable,
storage.usage.exactnever tookeffect.
rbd dunow runs--exact: the object map over-reported 87 GiB against the 11.4 GiBceph dfsays the pool stores. The VM disk alarm falls back to the block layer when no guest agentanswers, so an agentless instance can alarm at all.
Grafana, added after review feedback. Grafana reads Prometheus, so the same collection pass now
also writes a node_exporter textfile — the influx line protocol exists only because Kapacitor cannot
read Prometheus and the alarm needs a stream. Same shape as
instance_metrics_collect:.tmpthenrename, and a null is omitted rather than written as
0(an absent series reads as "not measured",a
0reads as "measured, empty").The shipped
Storagedashboard covered ceph cluster and OSD health and nothing per-pool or per-VM.It gains a VM and pool usage row: top 10 VMs by allocated storage, top 10 filesystems by usage,
top 10 VMs by thin-provisioning gap, pool stored vs raw, and pool provisioned over capacity.
Verified on a 1cc lab: 226 series in the textfile, and Prometheus serving them —
topk(10, sum by (vm_name) (cube_storage_disk_allocated_bytes))returns real VMs.166 assertions across 9 self-contained suites; they mock
rbd,ceph,qemu-img,virsh,openstackandcurl, sobash core/sdk_sh/tests/test_storage_usage_*.shneeds no cluster.Additional documentation