fix(logging,network): bound log growth, and let a wedged bond recover itself

[Eandalf-Bigstack]

What type of PR is this?

• bug

What this PR does / why we need it

Two unrelated-looking outages on accept-3cc turned out to be one theme: the system could not see what was wrong, and in one case its own recovery machinery was the cause.

• Stop one log from being able to fill the root partition. logstash.log.1 reached 1.4G under a maxsize 128M cap, /var/log held 8.5G on cc1 and 6.1G on cc2 of an 88G root shared with the OS, ceph and both metric stores, and ceph was warning mon cc2 is low on available space. The growth was not a busy service: 100% of a 200k-line sample of that generation was the opensearch output logging one INFO line per retried document, with the whole document inline, at ~60k lines an hour, while OpenSearch refused writes at its flood-stage watermark. That logger is now pinned to warn, which removes the amplifier itself. An earlier revision also moved the logrotate timer to hourly so the maxsize cap would be evaluated more than once a day; that has been reverted on the product owner's call and the schedule stays daily — see the note below
• Recover a bond that has stopped carrying traffic. A control node sat unreachable for 14 hours with every conventional signal healthy — both slaves UP/LOWER_UP at 1000Mbps, MII Status: up, the provider bridge holding the management IP, OVS forwarding with a NORMAL flow and 493M packets counted. health_link_check only pings peers, which cannot tell "the peer is down" from "my own bond is wedged", and link is NO_REPAIR. More fundamentally, every repair path in sdk_health.sh is driven from a reachable node over ssh — so a node in this state can only be fixed by itself. A cron watchdog now does that, locally, with no dependency on the telemetry or cluster stack
• Five hex fixes come in via submodule bumps: sharedscripts in the logrotate generator ([Bug]: httpd logrotate config missing sharedscripts crash-loops httpd, blocking Horizon on the VIP-owning node #1192), hex_trim_syslog finally installed, and the redundant delaycompress dropped — plus, from review, hex_trim_syslog no longer removing the generation logrotate is about to compress, hex_trim_syslog matching the compressed generations that actually exist so the SYSLOG_DISK_PERC budget has enforcement behind it at all, and retention bounded by maxage so the promise is expressed in the days the tuning advertises

Which issue(s) this PR fixes

• Fixes [Bug]: httpd logrotate config missing sharedscripts crash-loops httpd, blocking Horizon on the VIP-owning node #1192
• Refs [Story] Keep CubeCOS telemetry working by replacing Monasca with Prometheus and exporters #672

Special notes for your reviewer

This PR now depends on a second hex PR, which has NOT merged yet.

• fix(logrotate): bound retention by age, and drop the dead cron.hourly move hex#165 — open, must merge with this one. Bounds log retention by age (maxage) so the window is expressed in the days cubesys.log.default.retention advertises rather than in generations, makes hex_trim_syslog able to match a compressed generation at all, and removes the dead cron.hourly move. The last three submodule bumps point into it.

Already merged, and no longer blocking:

• fix(logrotate): sharedscripts, install hex_trim_syslog, and drop redundant delaycompress hex#163 — the first three submodule bumps point into it. Without it the hourly timer would make things worse: eleven httpd reloads a day becomes two hundred and sixty four
• bigstack-oss/triangle#149 — gives bonded test guests a real LACP partner, so the watchdog is exercised against a realistic topology rather than the lab artifact that produced the original outage

Why the bond watchdog decides on reachability and not on any LACP state variable. Two candidates were tried and both produce false healthies, which is worth recording so they are not re-derived. Actor Churn State reads churned on perfectly healthy nodes, because a bond whose partner never answers LACPDUs runs permanently defaulted — identical on all three nodes while two served traffic and one black-holed. The Synchronization bit of the active aggregator reads set on a node that could not reach its own default gateway, because which aggregator is active matters more than whether it synchronized. Reachability is the only signal that tracked the fault in every observation. The aggregator details are logged as diagnostics; nothing decides on them.

Why cron and not the health SDK. cluster_check_repair, _essential_check_one and live_migration_gate are all driven from a reachable node. A node whose bond has stopped carrying traffic is by definition not reachable, so none of them can ever get to it. Cron rather than a telegraf exec input for the same reason — the telemetry stack's outputs are all failing precisely when this is needed.

Guards on the watchdog. It acts only when the node can reach neither its default gateway nor any peer; it excludes every locally-configured address from that test, because pinging your own IP succeeds through loopback even when the bond carries nothing; it leaves a bond with no live slave alone, since that is a link fault a bounce cannot fix; it will not arm until the node has been observed reachable at least once; it requires three consecutive failing checks rather than one; and it holds off 900s between repairs so a fault it cannot fix does not become a flap.

Why the watchdog needs proof that the signal ever worked. The last two of those guards came out of review, and they close a real hole. _network_bond_reachable() already declines to act when there is nothing to test against — but "the only target exists and refuses to answer ICMP" falls on the other side of that check, and is then indistinguishable from a black-holed bond, permanently. A gateway configured to reject ping is a normal thing to meet in the field, and must never be read as an interface failure. On a single-node cluster it is also the only target there is: CUBE_NODE_LIST_IPS holds just this node's own address, every peer is correctly excluded as local, and the check is pinned at unreachable for the life of the node — so a healthy node bounced its slaves every 15 minutes, indefinitely.

So the watchdog now requires a persistent marker, $STATE_DIR/network_bond_reachable_seen, written the first time a target actually replies. That separates the fault this exists for — was carrying traffic, then stopped — from "was never measurable". It lives under $STATE_DIR rather than /run because it has to survive a reboot and the A/B switch: a node does not become unmeasurable because it restarted. One consequence is deliberate: a node that comes up already wedged is left alone, because it is indistinguishable from the rejecting-gateway case and there is no evidence a bounce would help it. The three-consecutive-failures gate covers the neighbouring case — with a single target, one lost echo reply was otherwise enough to bounce a healthy bond. It delays a real repair by about four minutes, against the 14 hours the original fault went unnoticed.

The rotation schedule stays daily. An earlier revision of this PR shipped a logrotate.timer drop-in moving evaluation to hourly, so the maxsize 128M cap in 41 of the 62 generated configs would be checked more than once a day. @traviswu-bigstack pushed back on the cost and the retention implications, and that is reverted — the packaged OnCalendar=daily stands on its own again, and the AccuracySec=1m pin goes with it, since it only existed because the drop-in inherited AccuracySec=1h from that same packaged unit.

The consequence is worth stating rather than glossing: maxsize is evaluated only when logrotate runs, so on a daily timer it is once again a cap that bounds nothing between runs. What makes that acceptable now, and did not when this PR opened, is that the growth had a cause and the cause is fixed — the 1.4G generation was the opensearch retry amplification, not a busy service, and 7dde7496 removes it regardless of how often logrotate runs. Raising the frequency treated the symptom; the symptom no longer has a source.

maxage in hex#165 is unaffected by the schedule and stays: it states the retention promise in days, which is the unit the tuning is written in, and it is the only bound that says anything about a log that rotates rarely — with notifempty and a quiet service, generations can sit for months while staying well inside a count of 14.

Additional documentation

Verified on accept-3cc (3-node HA) and jim-1cc unless stated otherwise.

For the requirement "the rotation schedule is the packaged daily one" — the drop-in removed and daemon-reload run, on jim-1cc and all three accept-3cc nodes:

[cc1 ~]# systemctl cat logrotate.timer      # only the packaged unit remains
# /usr/lib/systemd/system/logrotate.timer
OnCalendar=daily
AccuracySec=1h
Persistent=true
[cc1 ~]# systemctl show logrotate.timer -p AccuracyUSec --value
1h                                          # back from the 1m the drop-in pinned
[cc1 ~]# systemctl show logrotate.timer -p NextElapseUSecRealtime --value
Tue 2026-09-15 00:00:00 CST                 # midnight, not the top of the hour

The timer stays active and a real logrotate pass over 188–223 considered logs exits 0 with no output on every node.

Applying the hex fixes reclaimed disk in one pass and cleared the ceph warning:

cc1  /var/log 8.5G -> 5.5G   root 85% -> 82%
cc2  /var/log 6.1G -> 5.0G   root 83% -> 82%      <- cleared "mon cc2 is low on available space"
cc3  /var/log 5.7G -> 4.9G   root 87% -> 86%

Separately, 122 orphaned *.backup files holding 5.04 GiB were found on cc1 alone — logrotate's own collision artifacts, which match no glob in any config, so no retention policy would ever have reclaimed them. Removed across all four nodes, 14.3 GiB total.

For the requirement "a wedged bond recovers itself" — induced on jim-1cc, then left entirely to cron:

04:12:01  CROND CMD   network_bond_watchdog
04:12:04  ETH00003W   bond_not_carrying_traffic
04:12:04  error: network_bond_repair: aggregatedlinks is not carrying traffic
                 [active_agg=2 eth0(agg=1,mii=up,ps=71) eth1(agg=2,mii=up,ps=79)], bouncing [eth0 eth1]
04:12:39  info:  network_bond_repair: aggregatedlinks reachable again after bouncing eth1
                 [active_agg=1 eth0(agg=1,mii=up,ps=79) eth1(agg=2,mii=up,ps=71)]
04:12:39  ETH00004I   bond_recovered
04:12:39  CROND CMDEND

38 seconds, no human involved. Note the diagnostic at fault time: ps=79 — the Synchronization bit was set while the node was black-holed, which is why the check does not use it.

For the requirement "a healthy node behind a gateway that rejects ping is never touched" — the review guards, on jim-1cc through the real hex_sdk dispatcher with the bounce stubbed out, so no NIC was ever touched:

[cc1 ~]# # a demonstrably healthy node
MII Status: up  (x3)          provider holds 10.1.0.1/16          ssh live over the bond
[cc1 ~]# # drop echo-requests to the gateway alone -- what a ping-rejecting gateway looks like
hex_sdk network_bond_check ; echo rc=$?
rc=1                                      # and it stays 1 for the life of the node
[cc1 ~]# # marker absent: 5 runs at BOND_WATCHDOG_FAILS=1 BOND_WATCHDOG_HOLDOFF=0
marker=absent  stamp=none  fails=none     # never armed, counter never even reached
[cc1 ~]# # marker present -- the fault this exists for
run 1 fails=1   run 2 fails=2   run 3 stamp=written + repair ran
[cc1 ~]# # one failing run, then a healthy one
fails=1  ->  cleared

Before these guards that same healthy node armed on the first failure, with the 900s holdoff as the only thing limiting the rate — a bounce every 15 minutes forever.

The no-op path, on all four nodes:

[cc1 ~]# hex_sdk network_bond_check ; echo rc=$?
rc=0                     # empty stdout by design: log_debug when healthy, log_error when not
[cc1 ~]# ls /run/network_bond_watchdog.last
ls: cannot access ...: No such file or directory      # never repaired a healthy bond

For the requirement "one service reload per rotation" (#1192),

[cc1 ~]# logrotate -d -f /etc/logrotate.d/httpd | grep -c 'running postrotate script'
11        # accept-3cc, 13 files matched, 11 non-empty
9         # jim-1cc, identical config
1         # both, after sharedscripts
[cc1 ~]# logrotate -f /etc/logrotate.d/httpd ; journalctl -u httpd --since ... | grep -c Reloading
4         # real reloads before
1         # after
[cc1 ~]# systemctl is-active httpd ; curl -sk -o /dev/null -w '%{http_code}' https://localhost/horizon/
active
302

For the requirement "delaycompress removal does not break compression" — isolated, rotate 14, copytruncate:

without delaycompress:  a.log.1.gz             then  a.log.1.gz  a.log.2.gz
with    delaycompress:  a.log.1 (uncompressed) then  a.log.1     a.log.2.gz

No errors either way; the newest generation is compressed instead of held. Compression errors seen on the test nodes were environmental — something outside CubeCOS had renamed rotated files to *.1-<stamp>.backup.

For the requirement "hex_trim_syslog is inert until it is needed",

[cc1 ~]# sh -x /usr/sbin/hex_trim_syslog 4592725
+ LIMIT=4592725
+ total=4
+ '[' 4 -gt 4592725 -a 14 -gt 0 ']'      # loop never entered, nothing removed

[SekiXu]

The logrotate half and the hex#163 pairing look right - b34700af is that PR's head, and it is
approved and clean, so a ff merge keeps the SHA.

One gap in the watchdog's arming condition. _network_bond_reachable() already refuses to act
when there is nothing to test against, which is the right instinct, but "gateway exists and
filters ICMP" falls on the other side of that check:

• on a single-node / all-in-one cluster CUBE_NODE_LIST_IPS holds only this node's own address,
  and locals correctly excludes it, so there are no peer targets
• the default gateway is then the only target, targets=1, and if it does not answer ICMP the
  function returns 1 permanently
• network_bond_watchdog runs every 2 min with a 900s holdoff, so a healthy node bounces its
  slaves every 15 minutes, indefinitely

Worth noting the tree already took a position on this class of signal: health_link_check is
ping-based and link is NO_REPAIR (core/modules/cli_cluster.cpp:98). Repairing on a
ping-derived signal is the right call here - as your header comment says, it is the only signal
that separates this fault from a healthy bond - so what is missing is just a guard for the case
where the signal cannot be trusted. A persistent "has been reachable at least once" stamp before
arming, or N consecutive failures instead of one, closes it without weakening the real case.

[Eandalf-Bigstack]

@SekiXu thank you — this is exactly the right catch, and it goes to the heart of the check.

A gateway that refuses to answer pings must not be deemed a network interface failure. Some environments deliberately configure their gateway to reject ping tests, and that is a perfectly normal thing to meet in the field. We discussed this several years ago and I had honestly forgotten it; thanks for picking up this possible case.

Your reading of the code is right in every detail. On a single-node / all-in-one cluster CUBE_NODE_LIST_IPS holds only this node's own address, locals correctly excludes it, so there are no peer targets; the gateway becomes the only target, targets=1, and a gateway that does not answer ICMP pins the function at 1 permanently. With cron at 2 min and the 900s holdoff, that is a healthy node bouncing its slaves every 15 minutes, indefinitely.

Reproduced on jim-1cc, which is precisely that topology, by dropping echo-requests to the gateway alone:

[cc1 ~]# # a demonstrably healthy node -- ssh is live over this very bond
MII Status: up  (x3)      provider holds 10.1.0.1/16
[cc1 ~]# hex_sdk network_bond_check ; echo rc=$?
rc=1

Fixed in eee28e0, and I took both of your suggestions rather than choosing between them, because they turn out to close different halves:

• A persistent "has been reachable at least once" stamp — $STATE_DIR/network_bond_reachable_seen, written the first time a target actually replies and never removed. This is the one that closes the case you found: it separates the fault the watchdog exists for, was carrying traffic and stopped, from was never measurable. It is under $STATE_DIR rather than /run so it survives a reboot and the A/B switch — a node does not become unmeasurable because it restarted. The targets == 0 early return deliberately does not set it: "nothing to test against" is not evidence of health.
• N consecutive failures — three, not one. This does not help with a permanently rejecting gateway, but it covers the neighbour of your case: with a single target, one lost echo reply was enough to bounce a healthy bond. On a multi-node cluster the gateway and every peer have to fail together, so a lone drop could never decide anything there; it is single-target nodes that were exposed. Four minutes of extra delay, against the 14 hours the original fault went unnoticed.

One deliberate consequence worth naming: a node that comes up already wedged now has no marker and is left alone. That is the right trade — it is indistinguishable from your rejecting-gateway case, and unlike a node that broke after months of working there is no evidence a bounce would help it.

Verified on jim-1cc through the real hex_sdk dispatcher, with the bounce stubbed out so no NIC was touched:

[cc1 ~]# # gateway rejecting ping, marker absent -- 5 runs at FAILS=1 HOLDOFF=0,
[cc1 ~]# # i.e. every other gate wide open
marker=absent  stamp=none  fails=none     # never armed; the counter was never reached
[cc1 ~]# # marker present -- the real fault
run 1 fails=1   run 2 fails=2   run 3 stamp=written + repair ran
[cc1 ~]# # one failing run, then a healthy one
fails=1  ->  cleared

Your point about health_link_check being ping-based and link being NO_REPAIR (cli_cluster.cpp:98) is well taken, and I've kept the repair-on-ping decision for the reason you give — it is still the only signal that separated this fault from a healthy bond in every observation. What was missing was exactly what you said: a guard for when that signal cannot be trusted.

Two things I noticed while testing that this PR does not change, in case you think either should be in scope: log_debug goes to logger -p user.debug and is not landing in hex_sdk.log on that node, so the new "not arming" line is invisible at default verbosity; and network_bond_check will still log bonded network UNREACHABLE every two minutes on a rejecting-gateway node. The flapping is fixed, the log noise is not.

On hex#163 — I took your cosmetic finding too, since installing the script was meant to remove "No such file or directory" from the journal and it would have come back in another form on over-budget nights. hex_trim_syslog now stops at .2 and leaves messages.1 to the compression that follows it. It costs the budget nothing, because the next rotation renames .1 to .2 and the same bytes become eligible one cycle later. Reproduced your error exactly on logrotate 3.18.0 and confirmed it is gone:

trim: removing messages.1
error: unable to open .../messages.1 for compression: No such file or directory   <- before
compressing log with: /bin/gzip                                                    <- after

That landed as hex cb13101 and comes in here via dde45bd. Thanks also for measuring the delaycompress interaction and recording that the matchable set is byte-for-byte identical either way — that saved re-litigating it.

[Eandalf-Bigstack]

@traviswu-bigstack thank you for the pushback — all three questions were the right ones to ask, and two of them changed the change. Answering each with measurements from accept-3cc and jim-1cc.

"does increasing frequency mean we lost our promise of keeping long enough logs for auditing"

Yes, it did, and worse than you put it. cubesys.log.default.retention is a published tuning whose help text reads "Set log file retention policy in days", default 14 — and it is the only retention control there is. All ~40 LogRotateConf declarations in hex and cubecos pass retention 0, and the writer only emits rotate when that is non-zero, so every generated config inherits one global value. Measured: 1 of 62 files in /etc/logrotate.d sets its own rotate.

But rotate counts generations, which equals days only while a log rotates once a day. Every config pairs daily with maxsize 128M and rotates on whichever comes first, so under an hourly timer a log that passes the cap kept 14 hours. Steady-state simulation, scaled 7x (maxage 2, rotate 48), starting empty and ageing each generation by one interval between rotations:

rotating once a day       rotate 2             kept  2 gens, oldest 1.0 days
                          rotate 48 + maxage 2 kept  3 gens, oldest 2.0 days
tripping maxsize hourly   rotate 2             kept  2 gens, oldest 0.0 days   <- 2 hours
                          rotate 48 + maxage 2 kept 48 gens, oldest 1.9 days

Fixed in bigstack-oss/hex#165, which this PR now depends on: maxage <days> carries the promise and rotate is sized so it cannot bind first. Note maxage alone would not have been enough — rotate 14 still deletes the 15th generation regardless of age, so it would have bound at 14 hours and maxage would never have been reached. The live global config now renders as rotate 336 + maxage 14.

"logrotate causes pressure to cpu & disk"

Not measurably, for the part hourly actually adds. A no-op pass over 222 considered logs across 62 configs takes 17–32 ms (three runs). Total compression work is unchanged — the same bytes per day get compressed either way, just spread over more runs instead of one.

One real new cost you're right to want named: a log that trips maxsize hourly runs its postrotate up to 24× a day instead of once. That is one service reload per rotation rather than eleven, now that sharedscripts has landed, but it is still 24 where there used to be 1.

"like why a specific logfile grow so fast"

This is the one that changed my understanding, and my first answer was wrong. I sampled the small live log and concluded Kafka consumer chatter. The actual 1.4 GB generation is 100% one message:

[INFO][logstash.outputs.opensearch][log-transformer] Retrying failed action
{status: 429, action: ["index", {...the entire document...}]}

One INFO line per retried document, each carrying the document — ~60k lines an hour, and the sampled hour ran 00:01→01:00 without a gap.

And the 429 is not a logstash problem. From OpenSearch's own log:

[2026-09-04T18:04:43][WARN][o.o.c.r.a.DiskThresholdMonitor][cc1] flood stage disk watermark [95%]
  exceeded on [cc2][/var/lib/opensearch/nodes/0] free: 4.3gb[4.9%],
  all indices on this node will be marked read-only
[2026-09-05T14:08:48] ... [cc1] free: 645.7mb[0.7%] ...
[2026-09-05T14:08:48] ... [cc3] free: 603.3mb[0.6%] ...

ClusterBlockException[index [...] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded
  flood-stage watermark, index has read-only-allow-delete block]]

/var/log and /var/lib/opensearch are the same filesystem (/dev/sda5, /). So it is a feedback loop: logs fill the disk → OpenSearch crosses the flood stage and refuses writes → logstash logs a whole document per retry → which consumes the remaining disk. It ran the cluster down to 645 MB free. The 1.4 GB generation is dated to the rotation that closed the flood window, which is the correlation.

OpenSearch refusing writes there is correct behaviour and is not being changed. The defect was the amplification, and that is fixed by raising logstash.outputs.opensearch to warn — chosen over silencing because the plugin reports the condition at ERROR (no living connections in the pool) and the recovery at WARN (Restored connection), so what goes away is only the per-document retry line, which is not individually actionable. Logstash's own logging API confirms the scope on all three nodes:

logstash.outputs.opensearch                             WARN
org.apache.kafka...ConsumerCoordinator                  WARN   (secondary: restart chatter)
org.apache.http                                         FATAL  (upstream's own precedent)
logstash.outputs.file / http / kafka                    INFO   (left alone)

Also worth noting: there is a system-partition disk-usage alert in kapacitor. The lab nodes simply have no downstream webhook or mail recipient registered, which is why this ran to 0.7% unannounced rather than because nothing watches it.

On reverting to daily

That reinstates the original defect rather than trading it: maxsize is only evaluated when logrotate runs, so a daily timer means one evaluation a day and the 128 M cap in all 62 configs is unenforceable — which is how a file reached 1.4 G under a 128 M cap. The pair is hourly plus maxage; neither half is right alone.

Housekeeping found on the way

find /var/log -name '*.backup' turned up 122 files holding 5.04 GiB on cc1 alone. They are logrotate's own collision artifacts — it renames an existing .1 aside when a rotation finds the destination occupied — and, being named *.backup, they match no glob in any config: logrotate -d considers 222 logs and zero of them. So no retention policy would ever have reclaimed them. Development leftovers from this work; removed across all four nodes, 14.3 GiB total:

cc1  6.6G -> 1.6G  (83% -> 77%)      cc2  5.9G -> 1.2G  (87% -> 81%)
cc3  5.9G -> 1.3G  (86% -> 81%)      jim-1cc  1 file

Thanks again — the audit-window question in particular would have shipped as a silent 24x regression on a published tuning.

[SekiXu]

The watchdog arming gap is closed, and closed correctly. BOND_REACHABLE_SEEN is written
only on an observed reply and deliberately not on the targets == 0 path, which is the
distinction that matters — "nothing to test against" never becomes evidence of health. The
filtered-gateway single-node case can now never arm, and $STATE_DIR is a safe home for it
(hex/make/hex_support.mk:66 creates it in every rootfs). Taking both suggestions rather
than one was the right call; FAILS=3 covers the neighbour I did not spell out.

One thing left on the bond half.

network_bond_repair bounces every bond, on a node-level signal

_network_bond_reachable pings the gateway and peers over whatever the route table picks,
so its answer is about the node, not about a bond. network_bond_repair then loops every
entry in bonding_masters and bounces each slave in turn.

More than one bond is a supported topology: BondingConfig is a
std::map<std::string, BindIfs> (hex/src/modules/include/policy_network.h:61), the CLI
offers "Select bond interface" with Create/Remove/Update over it
(hex/src/modules/include/cli_network.h:26,30), and firsttime iterates the map to display
them (firsttime_net_bonding.cpp:81).

The bound on this is that the watchdog only ever fires when the management path is already
down, so the node is unreachable regardless. But a separate storage bond can be perfectly
healthy at that moment, and the order of bonding_masters is not defined — so it can be the
one bounced first, 5s down plus 12s up per slave, ~34s off the air before the faulty bond is
even reached. That is new harm on a path that is otherwise still carrying Ceph traffic.

Scoping the repair to the bond that carries the default route would close it, and it is the
only bond the reachability signal actually says anything about. While you are in that file,
core/heavyfs/bond_watchdog.cron still says the watchdog "only pings when an aggregator
already looks wrong" and presents the lost Synchronization bit as the deciding signal — both
describe the design 0cc3a016 replaced.

One question on the timer

logrotate-hourly.conf resets OnCalendar but nothing else. If the packaged
logrotate.timer still carries AccuracySec=1h, the drop-in inherits it and each firing can
be deferred by up to an hour, putting real intervals anywhere in 0-2h. I could not confirm
the packaged unit's contents offline, and the accept-3cc measurements cover a manual run
rather than the schedule — so this may already be answered. If it is not, AccuracySec wants
resetting alongside OnCalendar, for the same reason the comment gives for OnCalendar.

Separately, hex_trim_syslog cannot remove anything as it stands — it never matches .gz,
and I=14 no longer tracks the retention now that rotate is retention * 24. Neither is
this PR's to fix; details on hex#165, which this submodule bump is waiting on anyway.

[Eandalf-Bigstack]

@SekiXu thank you — three findings, all three real, and all three now fixed. Taking them in the order you raised them.

1. network_bond_repair bounces every bond, on a node-level signal

Right, and the reasoning is right: _network_bond_reachable pings the gateway and the peers over whatever the route table picks, so its answer is about the node's management path and says nothing about any other bond. Looping bonding_masters and bouncing each slave acted on every bond from a signal that describes one.

We have rescoped the watchdog and its repair to the management interface bonding only — which is what actually went wrong in the incident this exists for. The other bondings are deliberately out of scope, and not only to avoid the harm you describe: for storage and other interfaces we have no trustable or cooperative peer guaranteed to answer our pings. Nothing promises a storage network replies to ICMP at all, so a reachability signal there would be measuring nothing — the same untrustworthy-signal case BOND_REACHABLE_SEEN exists for, reached from the other direction. A bond we cannot honestly measure is a bond we should not bounce.

Fixed in eb8b376. Resolution turned out to need OVS awareness: the bond is an OVS port of the provider bridge, which holds the management address and the default route, and OVS bridges expose no lower_* links — so the kernel walk finds nothing and ovs-vsctl has to be asked. The lower_* walk is kept for a Linux bridge or a VLAN stacked over a bond. When nothing resolves, the function returns empty and the watchdog does nothing, which is the safe direction.

Verified on jim-1cc and accept-3cc through the real dispatcher, with a second bond present and only ip link set intercepted so no NIC was touched:

bonding_masters      aggregatedlinks bond-storage
_network_mgmt_bond   aggregatedlinks
repair would bounce  set eth0, set eth1        <- aggregatedlinks' slaves
never named          dummy-s0, dummy-s1        <- bond-storage's slaves

_network_bond_list is gone with it — no callers left.

2. bond_watchdog.cron describes the design 0cc3a016 replaced

Also right, and worse than stale: it opened on "a bond whose 802.3ad aggregator has lost synchronization" and presented the lost Synchronization bit as the deciding signal — the very bit that was measured set (ps=79) on the black-holed node, and which the code logs as a diagnostic that nothing decides on. Rewritten in the same commit to state the actual decision, the new scope, and the two guards added since.

3. AccuracySec on the timer

Your question was well-founded — you could not confirm the packaged unit offline, and it does carry it:

# /usr/lib/systemd/system/logrotate.timer
OnCalendar=daily
AccuracySec=1h
Persistent=true

The drop-in reset only OnCalendar, so it inherited AccuracySec=1h and systemd was free to defer each firing by up to an hour — real intervals anywhere in 0–2h, exactly as you said. A two-hour gap is long enough for a busy log to pass 128M and keep going, so the schedule would have looked hourly in the unit file and not been hourly on disk.

Fixed in b5cf5d7 with AccuracySec=1m. It needs no empty reset line, unlike OnCalendar — it is not list-valued, so setting it wins — and the comment now says so, because the empty OnCalendar= directly above invites copying the wrong pattern. Measured AccuracyUSec=1h before and 1min after on both clusters, with OnCalendar still a single hourly entry.

4. hex_trim_syslog

Answered in full on hex#165, where it belongs — short version: you were right on both counts, the .gz half was a no-op I introduced in cb13101, and the generation list now comes from the filesystem rather than a literal that has drifted twice. Fixed in hex e2a4060, which this PR picks up in 9d6366e.

---

Four commits here: the submodule bump, AccuracySec, and the bond rescope with the cron rewrite. hex#165 still needs to merge with this one.

🤖 Generated with Claude Code

[traviswu-bigstack]

if we've identified why a logfile grow so fast, what's the benefit of using hourly rotation. It seems to cause the system jitter with cpu/mem spike, tho it might be small. And the log folder is going to be full with thousands of small log rotation files (14 x 24 x numbers of log files) and make log investiation much harder

[Eandalf-Bigstack]

@traviswu-bigstack understood — the logrotate schedule is back to daily. Reverted in e96331d, with the hex side following in 9287fd0.

What came out:

• the logrotate.timer drop-in and its install rule, so the packaged OnCalendar=daily stands on its own again
• the AccuracySec=1m pin, which only existed because that drop-in inherited AccuracySec=1h from the same packaged unit
• rotate retention * 24 in hex, which was sized for the worst case an hourly timer allowed and on a daily schedule meant 336 generations for a log that can produce at most 14

Verified on jim-1cc and all three accept-3cc nodes — systemctl cat logrotate.timer shows only the packaged unit, AccuracyUSec is back to 1h, next elapse moved to 00:00:00, the timer stays active, and a real pass over 188–223 considered logs exits 0.

The cost of reverting, stated plainly: maxsize 128M is evaluated only when logrotate runs, so on a daily timer it is once again a cap that bounds nothing between runs. 41 of the 62 generated configs carry it. A log that grows past 128M during the day is still rotated exactly once, at whatever size it reached.

What makes that acceptable now — and what I got wrong when this PR opened — is that your question "why does a specific logfile grow so fast" had a real answer, and it was not "a busy service". The 1.4G generation was 100% one message: the opensearch output logging one INFO line per retried document, with the whole document inline, at roughly 60k lines an hour, while OpenSearch refused writes at its flood-stage watermark. 7dde7496 pins that logger to warn, so the amplifier is gone regardless of how often logrotate runs. Raising the frequency was treating the symptom; the symptom no longer has a source.

One thing I'd keep on the record rather than close silently: the underlying loop is still there in the shape of the system. /var/log and /var/lib/opensearch are the same filesystem, so anything that fills one starves the other, and OpenSearch's response to that is to refuse writes — correctly. We have removed the amplifier that turned a slow fill into a fast one, not the coupling. If a different component ever starts logging per-item under backpressure, the daily timer will not bound it either.

Two things from this work that are unaffected by the revert and stay:

• maxage in hex#165. It states the retention promise in the days cubesys.log.default.retention advertises rather than in generations, and it is the only bound that covers a log which rotates rarely — with notifempty and a quiet service, generations sit for months while staying well inside a count of 14. On the daily schedule it and rotate agree exactly, so it changes nothing visible; measured both ways on accept-3cc.
• hex_trim_syslog actually working. @SekiXu found it had never been able to remove a compressed generation, so SYSLOG_DISK_PERC had no enforcement behind it at all. That is the budget that bounds the /var/log/messages* set, and it is independent of the rotation schedule.

Thanks for holding the line on this one — the frequency change would have shipped as a permanent cost for a problem that turned out to have a specific, fixable cause.

🤖 Generated with Claude Code