fix(logrotate): sharedscripts, install hex_trim_syslog, and drop redundant delaycompress

[Eandalf-Bigstack]

What type of PR is this?

• bug

What this PR does / why we need it

Three defects in log rotation, all in the generator rather than in any one generated file — which is why they reproduced identically across httpd, syslog and prometheus.

• sharedscripts was never emitted. WriteLogRotateConf wrote prerotate/postrotate blocks without it, so logrotate ran them once per matched file — its documented behaviour. Every script CubeCOS generates is a "signal the service once" operation, so a glob matching N files fired N signals within the same second. For /var/log/httpd/*.log that is up to eleven overlapping systemctl reload httpd.service calls racing in the master's worker lifecycle, which can crash httpd outright and 503 Horizon and Keystone on whichever control node holds the VIP
• hex_trim_syslog was referenced but never installed. config_syslogd writes /usr/sbin/hex_trim_syslog into the syslog logrotate postrotate, but no install rule ever shipped it. The script has been in data/hex_syslogd since the initial commit, and its sibling hex_log_event.sh is installed a few lines above — this one was simply missed. Every nightly rotation logged No such file or directory, and the /var/log/messages size budget it enforces never applied
• delaycompress was emitted unconditionally and is redundant here. It exists for the case where the writing process keeps its old file descriptor and must be signalled to reopen, so the newest rotated generation has to stay uncompressed. All 30 LogRotateConf declarations across hex and cubecos set copytruncate, so the rotated file is already a finished copy that nothing is still writing to. Delaying only kept the single largest generation uncompressed

Which issue(s) this PR fixes

• Part of [Bug]: httpd logrotate config missing sharedscripts crash-loops httpd, blocking Horizon on the VIP-owning node cubecos#1192 (the sharedscripts half)

Special notes for your reviewer

Paired with bigstack-oss/cubecos PR for the same work — that PR carries the submodule bump plus the hourly logrotate.timer drop-in, and the two need to land together. The hourly timer is what makes the per-config maxsize 128M cap actually enforceable (maxsize is only evaluated when logrotate runs, so a daily timer let a logstash.log.1 reach 1.4G under a 128M cap). It is also why the sharedscripts fix could not wait: hourly rotation would have turned eleven httpd reloads a day into two hundred and sixty four.

Why sharedscripts is conditional. It is emitted only when a config actually carries prerotate or postrotate commands. On a config with no scripts the directive is meaningless, and three configs is the whole affected set: httpd (systemctl reload httpd.service), syslog (hex_trim_syslog + rsyslog HUP) and prometheus (killall -HUP prometheus).

Why delaycompress is removed globally rather than per config. extraArgs is written before the common block, so a nodelaycompress override from cubecos would be cancelled again by the very directive it is trying to override. And since every declaration in both repos sets copytruncate, there is no config that wants the old behaviour.

One caveat on hex_trim_syslog. It has never run in production, so its behaviour on a real oversized /var/log/messages set is unexercised. It is inert below its limit — traced under sh -x with the limit it is actually passed (4592725 KB) against a 4 KB total, the loop does not execute — but the trimming path itself only starts being reachable with this PR.

Additional documentation

Verified on accept-3cc (3-node) and jim-1cc.

sharedscripts -- forced rotation of the httpd config, counting postrotate runs:

  [cc1 ~]# logrotate -d -f /etc/logrotate.d/httpd | grep -c 'running postrotate script'
  11                      # accept-3cc, 13 files matched, 11 non-empty
  9                       # jim-1cc, same config
  1                       # both, after sharedscripts

Measured against the live service rather than only the dry run:

  [cc1 ~]# logrotate -f /etc/logrotate.d/httpd ; journalctl -u httpd --since ... | grep -c Reloading
  4                       # before
  1                       # after
  [cc1 ~]# systemctl is-active httpd ; curl -sk -o /dev/null -w '%{http_code}' https://localhost/horizon/
  active
  302

The same defect in the other two script-bearing configs, before the fix:

  syslog: postrotate=3  sharedscripts=0
  prometheus: postrotate=0 sharedscripts=0

syslog's count of 3 is the direct cause of a symptom seen independently: exactly
three "hex_trim_syslog: No such file or directory" errors in the journal every
night on every node -- three of syslog's five files rotate, and the script ran
once per file.

hex_trim_syslog -- before, every nightly rotation logged the missing file; after
installing, a forced rotation of the syslog config exits 0 with no error. Traced
under sh -x with the real limit against a 4 KB total: the loop does not execute
and no retention is removed.

delaycompress -- isolated single-file comparison, rotate 14, copytruncate:

  without delaycompress:  a.log.1.gz            then  a.log.1.gz  a.log.2.gz
  with    delaycompress:  a.log.1 (uncompressed) then a.log.1  a.log.2.gz

No errors either way; the newest generation is simply compressed instead of held.
Applied across a live cluster this reclaimed 3.0G on cc1, 1.1G on cc2 and 0.8G on
cc3 in a single pass, and cleared ceph's "mon cc2 is low on available space".

[SekiXu]

Ran a scoped experiment on a 1cc (logrotate 3.18.0) to check one interaction, and it came out in your favour — recording it here so it's not re-litigated later.

postrotate runs before compression, so what hex_trim_syslog observes is byte-for-byte identical with and without delaycompress — messages + plain messages.1 + .2.gz….4.gz in both. Dropping delaycompress does not shrink the set the script can match.

One cosmetic side effect when the trim actually fires: it removes messages.1 during postrotate, and logrotate then tries to compress the file that just went away —

trim: removing messages.1
error: unable to open /var/log/messages.1 for compression: No such file or directory

Deterministic, 6/6 forced cycles, and absent with delaycompress. No data loss and the trim still does its job, so not a blocker — but it does mean the "No such file or directory" line the install rule removes comes back in a different form on over-budget nights. Cheap to avoid if you want the journal fully clean.

[Eandalf-Bigstack]

Thank you for catching that. Fixed in cb13101.