feat(metrics): bound influxdb and prometheus retention, and keep the history in ceph s3

[Eandalf-Bigstack]

What type of PR is this?

• feature
• bug

What this PR does / why we need it

A metrics database should not be able to fill the system partition, and it should not lose history when it is capped. This PR does both halves of that, and fixes the credential collision that surfaced while wiring the second half up.

• Bound both time-series stores. InfluxDB kept 364d + 35d on the def retention policy — worst case an InfluxDB 1.x shard survives duration + shardGroupDuration, because whole shard groups are dropped and a group only expires once its end time plus the duration has passed. Prometheus kept 90d with no size cap at all. Both are now tunable in hex_config rather than compiled-in: def drops to 14d/7d, hc to 7d/2d, prometheus to 30d with a 5 GiB cap
• Stop throwing the capped history away. A 30d cap only makes sense if what falls off the end still exists somewhere, so the thanos sidecar now ships finished blocks to a ceph RGW bucket, a store gateway serves them back to the querier, and a compactor downsamples and enforces 90d retention in the bucket. Retention is thanos's, not an RGW lifecycle rule: expiry that thanos does not know about leaves dangling block metadata behind
• Give cube-cos-api its own RGW identity. The API's accessKey setting is a keystone user name, not an S3 key, and it defaulted to admin — so it minted an EC2 credential whose access key is literally "admin", while sdk_health and cube_cluster_start_cluster list admin's credentials, cache the first in /run/ec2.key and delete/recreate on an empty read. ec2 credentials delete admin removes exactly the credential the API created. This is the residue of [Bug]: s3 credential secret key is empty, make Health page detail log Access Denied #703

Which issue(s) this PR fixes

• Refs [Story] Keep CubeCOS telemetry working by replacing Monasca with Prometheus and exporters #672 — items 1, 2, 5 and 6 of the retention comment
• Refs [Bug]: s3 credential secret key is empty, make Health page detail log Access Denied #703 — the admin EC2 credential is no longer shared

Special notes for your reviewer

Why the two InfluxDB numbers add up. They are not two windows, they are duration and shard-group duration. InfluxDB 1.x never deletes points, only whole shard groups, and a group's expiry_time is its end time plus the retention duration — so the oldest point still on disk can be duration + shardGroupDuration old. 364d/35d was a worst case of 399d. The new pairs keep that arithmetic visible: 14d/7d and 7d/2d.

Why prometheus retention moved into the config file. 3.13 marks both --storage.tsdb.retention.time and .size [DEPRECATED] and points at the storage.tsdb.retention block, and a flag set on the command line takes precedence over that block — which would silently pin the value and make the tunings look inert. This is 3.x-only: 2.55 does not know the field and refuses to parse rather than ignoring it, which took all three accept-3cc nodes to activating when it was tried there before the upgrade.

The sidecar will not upload unless prometheus stops compacting. --storage.tsdb.max-block-duration must equal --min-block-duration. Prometheus derives max as 10% of retention when unset, so this is not a constant that could be left alone — cutting retention to 30d moved it to 3d and would have silently stopped every upload. The sidecar validates this itself and refuses loudly ("found that TSDB Max time is 3d and Min time is 2h. Compaction needs to be disabled") rather than shipping bad blocks.

Why the compactor is a pacemaker resource and not a systemd unit. Two compactors against one bucket corrupt it, so exactly one must run cluster-wide — the same shape as cinder-volume. That leaves a bootstrap window where pacemaker exists but the compactor has no config, so config_pacemaker masks it before bringing pacemaker up and config_prometheus unmasks it once the config is written. The openvswitch mask moves out of bootstrap_cube_config into the same place.

The mask is gated on IsBootstrap(), and that qualifier is the whole safety of the handshake. The mask lives in config_pacemaker while both unmasks live in the modules that own the units — openvswitch in config_neutron's OvnService, thanos-compact in config_prometheus — and those modules' CommitCheck predicates are not this one's:

commits pacemaker but NOT neutron     pacemaker.modified | G_MOD(MGMT_IF)
commits pacemaker but NOT prometheus  pacemaker.modified | G_MOD(MGMT_IF) | G_MOD(IS_MASTER)

Without the gate, a settings commit carrying any of those masks both units and nothing ever unmasks them. A tuning change is applied cluster-wide, and it lands on exactly this path: cube-cos-ui to cube-cos-api, whose ApplyTuning() runs hex_config apply <dir> on each node, which translates the policy and then spawns hex_config commit <settings> — a settings commit, so IsBootstrap() is false. One tuning change therefore masks openvswitch on every control and compute node at once, compute included because the mask sits above the if (enabled) block. A mask does not stop a running daemon, so nothing looks wrong at the time; what it does is arm every later restart to fail, the repair paths and the next boot included. Every CommitCheck in the pass short-circuits to true under IsBootstrap(), so that is exactly the set of commits where both unmasks are guaranteed to follow — and it is the only path where the window exists at all, since on any later commit both units are already configured. Caught in review by @SekiXu; see the verification below.

Why the endpoint is passed to the sdk rather than discovered. os_endpoint.snapshot is the obvious source and unusable at commit time — its only commit-path writers are cube_last (six levels after prometheus) and keystone, which skips it on a master's first bootstrap. Measured on accept-3cc, the file's mtime is 42 minutes after boot. Its public row also resolves to EXTERNAL, the wrong side of the appliance for a control-plane upload.

Why a separate keystone user is enough for the shared log bucket. rgw runs with rgw keystone implicit tenants = false, so the S3 owner is the keystone project, not the user — and radosgw-admin bucket stats --bucket=log reports its owner as the admin project id on both clusters. A user with an rgw-accepted role on that project authenticates as the same S3 owner, so no bucket policy or ACL is involved. member suffices; the role is not what grants the access.

Why hex_config owns that credential. cube-cos-api creates it on startup but treats an HTTP 409 as success. Without reconciliation, the first time the configured secret changes — a new seed, or the api.s3.secret tuning — keystone would keep the old secret while the API signed with the new one, and every request would be denied with nothing in either log saying why.

Additional documentation

Verified on accept-3cc (3-node HA) and jim-1cc (single control) unless stated otherwise.

For the requirement "InfluxDB is bounded", applied live after the tunings landed:

jim-1cc     125M -> 58M
accept-3cc  cc1 279M -> 122M    cc2 316M -> 147M    cc3 265M -> 124M

For the requirement "prometheus honours both caps", from prometheus's own startup log rather than the flags:

level=INFO msg="Time-based retention policy" duration=30d
level=INFO msg="Size-based retention policy" size=5GiB

accept-3cc was upgraded 2.55.1 -> 3.13.1 first, since the config-file form does not exist in 2.x.

For the requirement "blocks reach S3 and are readable back", on all three control nodes:

[cc1 ~]# systemctl is-active prometheus thanos-sidecar thanos-store thanos-query
active active active active
[cc1 ~]# curl -s http://10.1.0.1:10904/prometheus/api/v1/stores | jq -r ...
  sidecar  10.1.0.1:10901   err=None
  sidecar  10.1.0.2:10901   err=None
  sidecar  10.1.0.3:10901   err=None
  store    127.0.0.1:10905  err=None
[cc1 ~]# curl -s http://10.1.0.1:10906/metrics | grep 'blocks_meta_synced{state="loaded"}'
thanos_blocks_meta_synced{state="loaded"} 12

Zero "Compaction needs to be disabled" messages and thanos_shipper_upload_failures_total 0 on every node.

For the requirement "the compactor is a singleton and does real work":

[cc1 ~]# pcs status resources | grep thanos-compact
  * thanos-compact   (systemd:thanos-compact):  Started cc1
[cc1 ~]# hex_sdk cmd -c -v "systemctl is-active thanos-compact"
cc1|0|active
cc2|3|inactive
cc3|3|inactive

A full cycle in its log — initial sync, garbage collection, compaction, both downsampling passes, retention apply — and the bucket shows the result: three level-2 blocks each spanning 6h with 1,344,240 samples, exactly 3 x 448,080, i.e. three consecutive 2h level-1 blocks merged. 18 blocks, 10,740,568 samples, continuous coverage with no gap.

For the requirement "nothing is offered on an interface that does not need it", every listener this module writes was audited; none is left on 0.0.0.0:

9091  prometheus   127.0.0.1 + mgmt   sidecar/self-scrape/non-HA haproxy are local; health check is cross-node
10901 sidecar grpc mgmt               every node's querier dials it
10902 sidecar http loopback           nothing reads it
10903 query grpc   loopback           no querier fan-out, no thanos-rule
10904 query http   mgmt               haproxy backend, and the health check
10905 store grpc   loopback           only that node's own querier
10906 store http   mgmt               the health check probes across nodes
10907 compact http loopback           nothing reads it; the singleton is established through systemd

Confirmed on all three nodes: the four cross-node paths answer, the four loopback-only ports are refused off-box, the self-scrape is up, and grafana's route through the VIP (https://<vip>/prometheus/graph) returns 200.

For the requirement "commit order is correct", from hex_config -d on the built binary:

L9   pacemaker    <- masks openvswitch and thanos-compact, then starts pacemaker
L11  ceph         <- radosgw up
L12  prometheus   <- ceph cube_scan first        (was L8 without CONFIG_REQUIRES(prometheus, ceph))
L13  neutron      <- unmasks openvswitch
L16  api          <- apache2 cube_scan cyborg first influxdb keycloak keystone mongodb
L18  pacemaker_last <- creates the pcs resources

For the requirement "a settings commit cannot leave openvswitch masked" — an A/B of two binaries built from the same source, one with the IsBootstrap() gate and one without, each running the identical non-bootstrap, pacemaker-only commit on jim-1cc. WriteLogRotateConf is Commit()'s last statement, so /etc/logrotate.d/pacemaker witnesses that the module ran rather than returning at its CommitCheck:

[cc1 ~]# hex_config commit <settings with one added pacemaker.enabled line> pacemaker
without the gate   witness written   openvswitch/thanos-compact -> masked/masked
with the gate      witness written   openvswitch/thanos-compact -> unchanged

The delta has to be recreated from a pristine settings copy before each run: a commit writes its settings back, so a second run finds nothing modified, returns at CommitCheck and proves nothing — the first attempt at this measured exactly that false pass.

What the leaked mask costs, measured with both units masked:

[cc1 ~]# systemctl restart openvswitch
Failed to restart openvswitch.service: Unit openvswitch.service is masked.
[cc1 ~]# systemctl start thanos-compact
Failed to start thanos-compact.service: Unit thanos-compact.service is masked.
[cc1 ~]# systemctl is-active ovs-vswitchd
active                    # <- still up, which is why this would go unnoticed

For the requirement "health notices when either new component fails":

[cc1 ~]# hex_sdk health_thanos_check ; echo $?
0
[cc1 ~]# ssh cc3 systemctl stop thanos-store ; hex_sdk health_thanos_check ; echo $?
5
[cc1 ~]# hex_sdk health_thanos_repair ; hex_sdk health_thanos_check ; echo $?
0
[cc1 ~]# pcs resource disable thanos-compact ; hex_sdk health_thanos_check ; echo $?
7

For the requirement "cube-cos-api no longer shares admin's credential" — jim-1cc, converge from no user at all, then idempotence, then rotation:

[cc1 ~]# openstack user delete cube-cos-api
[cc1 ~]# hex_sdk api_s3_user_setup cube-cos-api SECRET_ONE ; echo $?
0
    access=cube-cos-api secret=SECRET_ONE     role on admin project: member
[cc1 ~]# hex_sdk api_s3_user_setup cube-cos-api SECRET_ONE     # credential id unchanged
[cc1 ~]# hex_sdk api_s3_user_setup cube-cos-api SECRET_TWO     # rotation
    old secret rejected: yes    new secret works: ok    credentials: 1

And the isolation that #703 was about — the delete that used to break the API:

[cc1 ~]# openstack ec2 credentials delete admin
[cc1 ~]# openstack credential list --user cube-cos-api --type ec2 -f value -c ID | wc -l
1
[cc1 ~]# systemctl is-active cube-cos-api
active

On accept-3cc, run concurrently from all three control nodes it still converges to exactly one credential, all three reach s3://log, all three API instances come up active, and health_api_check returns 0. Reaching that state needed keycloak repaired first: its realm was half-migrated behind the XA abort described in 92e9520a, which is a pre-existing fault on that cluster and not something this PR touches. Two details worth recording — a StatefulSet RollingUpdate will not replace a pod-0 that never becomes Ready, so the XA env var sat in the statefulset while the pod ran without it until the pod was deleted; and once MIGRATION_MODEL is empty while the master realm already holds its default client scopes, no amount of restarting recovers it.

[SekiXu]

The thanos-compact-as-pacemaker-singleton shape looks right, and gating the pcs resource on
if (ha) while config_prometheus only unmasks under thanosEnabled keeps non-HA free of a
resource pointing at a masked unit.

One thing about the mask/unmask handshake: the two halves live in modules whose CommitCheck()
predicates are not the same, so a commit can mask without ever unmasking.

pacemaker: modified | s_bCubeModified | s_bNetModified | G_MOD(IS_MASTER)
           | G_MOD(MGMT_IF) | G_MOD(MGMT_ADDR) | G_MOD(SHARED_ID)
neutron:   modified | s_bMqModified | s_bNovaModified | s_bCubeModified
           | s_bKeystoneModified | s_bNetModified | G_MOD(IS_MASTER)
           | G_MOD(MGMT_ADDR) | G_MOD(SHARED_ID) | G_MOD(OVERLAY_ADDR)
           | G_MOD(PROVIDER_IF)

pacemaker's own modified - i.e. pacemaker.enabled itself changed - is not in neutron's set.
On that commit config_pacemaker::Commit() masks openvswitch and config_neutron::Commit()
returns at its CommitCheck before reaching OvnService(), which is the only
systemctl unmask openvswitch in the tree (config_neutron.cpp:620). The mask does not stop a
running daemon, but until the next full commit any systemctl restart openvswitch fails,
including the repair paths - and since the mask sits above the if (enabled) block it lands on
compute nodes too. thanos-compact has the same shape against config_prometheus's narrower
predicate, though there the consequence is only a stopped compactor.

IsBootstrap() short-circuits both, so the bootstrap path this replaces is unaffected - which is
why gating the mask on IsBootstrap(), or pairing it inside the if (enabled) block right
before pacemaker is started, keeps the window you want to close and drops the rest.

No comment on the secret in api_s3_user_setup's argv or the timeout 0 - both are how the tree
already does it (config_barbican.cpp:135 and config_cyborg.cpp:136 pass passwords the same
way, and HexUtilSystemF(0, 0, ...) is 452 of 454 call sites in core/modules).

[Eandalf-Bigstack]

@SekiXu thank you — and this one deserves more than a routine thanks. You stopped a routine cluster-wide tuning change from turning into a cluster-wide network blackout, with openvswitch masked on every node at once. Fixed in 3b56128.

I traced the path end to end after your comment, and it lands exactly where you said it would:

cube-cos-ui -> cube-cos-api  ApplyTuning()
             -> hex_config apply <dir>        (per node)
             -> hex_config commit <settings>  <- a settings commit, so IsBootstrap() is false
             -> config_pacemaker::Commit()    <- masks openvswitch + thanos-compact
                config_neutron::Commit()      <- returns at CommitCheck, never reaches OvnService
                config_prometheus::Commit()   <- same

So the trigger is not an exotic one. It is the normal way a tuning gets applied, it runs on every control and compute node — compute included, because as you noted the mask sits above the if (enabled) block — and every one of those nodes ends up with openvswitch masked and nothing left in the pass to unmask it.

The part that makes it nastier than it first looks is that it is silent. A mask does not stop a running daemon, so the cluster keeps forwarding and nothing looks wrong at the time:

[cc1 ~]# systemctl restart openvswitch
Failed to restart openvswitch.service: Unit openvswitch.service is masked.
[cc1 ~]# systemctl start thanos-compact
Failed to start thanos-compact.service: Unit thanos-compact.service is masked.
[cc1 ~]# systemctl is-active ovs-vswitchd
active                    # <- still up, which is why this would have gone unnoticed

What it actually does is arm the blackout: the next thing to restart openvswitch on any of those nodes fails — the repair paths, a rolling restart, or simply the next boot, where a masked unit does not come up at all. A cluster that had been told nothing more than "change this tuning" would then lose its datapath, everywhere, at whatever later moment someone rebooted.

I confirmed your predicate analysis and it holds, with one addition — G_MOD(MGMT_IF) is in pacemaker's set and in neither of the others, so a management-interface change reaches the same place:

commits pacemaker but NOT neutron     pacemaker.modified | G_MOD(MGMT_IF)
commits pacemaker but NOT prometheus  pacemaker.modified | G_MOD(MGMT_IF) | G_MOD(IS_MASTER)

I took your IsBootstrap() suggestion rather than the other one, for the reason your own comment implies: pairing the mask inside the if (enabled) block would not have closed it, since a commit that flips pacemaker.enabled to true still masks and still never reaches an unmask. Gating on IsBootstrap() works because every CommitCheck in the pass short-circuits to true there, so it is exactly the set of commits where both unmasks are guaranteed to follow — and it is the only path where the window exists at all, since on any later commit both units are already configured.

Verified on jim-1cc as an A/B of two binaries built from the same source, one with the gate and one without, each running the identical non-bootstrap pacemaker-only commit:

[cc1 ~]# hex_config commit <settings + one pacemaker.enabled line> pacemaker
without the gate   witness written   openvswitch/thanos-compact -> masked/masked
with the gate      witness written   openvswitch/thanos-compact -> unchanged

WriteLogRotateConf is Commit()'s last statement, so /etc/logrotate.d/pacemaker is the witness that the module really ran rather than returning at its CommitCheck. Worth recording that the first attempt at this measured a false pass: a commit writes its settings back, so the second run of the pair found nothing modified, returned at CommitCheck, and reported "no mask" for entirely the wrong reason. The delta now gets recreated from a pristine settings copy before each run.

One thing you raised that this PR does not change: config_prometheus::CommitCheck() does not include its own modified at all, so a prometheus.* tuning change alone never commits that module. That is pre-existing and independent of the masking bug, so I have left it out of this PR rather than widen it — happy to take it as a follow-up if you think it should move sooner.

Thanks also for clearing the two things you explicitly did not object to — the secret in api_s3_user_setup's argv and the timeout 0 — with the precedents counted. Saved a round trip.

[SekiXu]

3b56128 closes it, and the reasoning it rests on holds where it has to.

Both unmasks are guaranteed inside a bootstrap pass, which is what the IsBootstrap() gate needs to be true:

• all three CommitChecks do short-circuit under it (config_pacemaker.cpp:178, config_neutron.cpp:826, config_prometheus.cpp:428)
• OvnService() is called unconditionally in neutron's Commit() after its own early return (config_neutron.cpp:911), and it unmasks openvswitch for control|compute (config_neutron.cpp:617-621) - the same role set pacemaker's Commit() gates itself on (config_pacemaker.cpp:193), so there is no node where the mask lands and the unmask cannot reach

One residual worth having on the record rather than acted on. thanos-compact's unmask is gated on thanosEnabled = IsControl && s_ha (config_prometheus.cpp:448,486) while the mask runs for control|compute, so the unit stays masked after bootstrap on a non-HA control and on every pure compute node. The non-HA control heals itself the moment it matters: a cubesys ha change sets s_bCubeModified (config_prometheus.cpp:405, IsModifiedTune(1)), which is in prometheus's CommitCheck, so the commit that turns HA on is also the one that unmasks. The compute node keeps the mask, and the compactor resource carries only INFINITY location prefers on the control hosts (config_pacemaker.cpp:104,118) while pure computes join as pacemaker remotes (config_pacemaker.cpp:327), so placement there is possible and would find the unit unstartable rather than transiently failing. That is the same exposure cinder-volume already has one line above, so it is not something this PR introduces.