From fea23e1cece2805b97574018f3bab8e49549153b Mon Sep 17 00:00:00 2001 From: Jim Lin Date: Sun, 6 Sep 2026 15:00:29 +0800 Subject: [PATCH 1/4] build(syslogd): install hex_trim_syslog, referenced but never shipped config_syslogd writes /usr/sbin/hex_trim_syslog into the syslog logrotate postrotate, but nothing ever installed it. The script has been in data/hex_syslogd since the initial commit; its sibling hex_log_event.sh is installed a few lines above and this one was simply missed. Effect on a running node: every nightly rotation logged three 'No such file or directory' errors, and the /var/log/messages size budget the script enforces was never applied. Because it is postrotate guarded by '|| true' the rotation itself still happened, which is why this stayed invisible. Verified on jim-1cc: before, 'logrotate -f /etc/logrotate.d/syslog' logged the missing-file error; after installing, it exits 0 with no error. Traced the script under sh -x with the real limit it is passed (4592725 KB) against a 4 KB total -- the loop does not execute and no retention is removed, so it is inert until /var/log/messages* actually exceeds the budget. Signed-off-by: Jim Lin Co-authored-by: Eandalf Co-authored-by: Claude Opus 5 (1M context) --- make/hex_support.mk | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/make/hex_support.mk b/make/hex_support.mk index fca01ff..da6e3fe 100755 --- a/make/hex_support.mk +++ b/make/hex_support.mk @@ -22,6 +22,12 @@ $(call PROJ_INSTALL_SCRIPT,-f,$(HEX_DATADIR)/hex_fixpack/hex_fixpack_install.sh, # hex_log_event: event logging helper invoked by hex modules/cluster scripts $(call PROJ_INSTALL_SCRIPT,-f,$(HEX_DATADIR)/hex_syslogd/hex_log_event.sh,./usr/sbin/hex_log_event) +# hex_trim_syslog: drops /var/log/messages.N retentions when the set exceeds the +# size budget. config_syslogd writes it into the syslog logrotate postrotate, so +# without this line every nightly rotation logs "No such file or directory" and +# the size budget it is supposed to enforce is never applied. +$(call PROJ_INSTALL_SCRIPT,-f,$(HEX_DATADIR)/hex_syslogd/hex_trim_syslog.sh,./usr/sbin/hex_trim_syslog) + # Add a utility script that does not seem to really fit anywhere else $(call PROJ_INSTALL_SCRIPT,-f,$(HEX_DATADIR)/hex_uptime,./usr/sbin/hex_uptime) From 56fdc4788ec82e55daac8008de46be44a6897273 Mon Sep 17 00:00:00 2001 From: Jim Lin Date: Sun, 6 Sep 2026 15:02:50 +0800 Subject: [PATCH 2/4] fix(logrotate): drop delaycompress, redundant under copytruncate WriteLogRotateConf emitted delaycompress for every generated config. That directive exists for the case where the writing process keeps its old file descriptor and must be signalled to reopen, so the newest rotated generation has to stay uncompressed until the next cycle. Every LogRotateConf in hex and cubecos -- all 30 of them -- sets copytruncate, so the rotated file is already a finished copy that nothing is still writing to. Delaying bought nothing and kept the single largest generation uncompressed. Measured on accept-3cc: logstash.log.1 was 1.4G uncompressed against logstash.log.2.gz at 25M, a ~56x difference on the generation that dominates the footprint. Removing it and re-running logrotate reclaimed 3.0G on cc1, 1.1G on cc2 and 0.8G on cc3 in a single pass, and cleared ceph's 'mon cc2 is low on available space'. Done here rather than per-config in cubecos because extraArgs is written before the common block, so a nodelaycompress override would be overridden again by the directive it is trying to cancel. Signed-off-by: Jim Lin Co-authored-by: Eandalf Co-authored-by: Claude Opus 5 (1M context) --- src/hex_sdk_library/logrotate/logrotate.cpp | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/hex_sdk_library/logrotate/logrotate.cpp b/src/hex_sdk_library/logrotate/logrotate.cpp index bdb1d45..d19c5a4 100644 --- a/src/hex_sdk_library/logrotate/logrotate.cpp +++ b/src/hex_sdk_library/logrotate/logrotate.cpp @@ -99,9 +99,17 @@ WriteLogRotateConf(LogRotateConf conf) fprintf(fout, " %s\n", conf.extraArgs.c_str()); // common configs + // + // No delaycompress: it exists for the case where the writing process keeps + // its old file descriptor and must be signalled to reopen, so the newest + // rotated generation has to stay uncompressed. Every LogRotateConf in hex + // and cubecos sets copytruncate, which means the rotated file is already a + // finished copy that nothing is still writing to -- so delaying only kept + // the largest generation uncompressed for no benefit. Measured on + // accept-3cc: logstash.log.1 1.4G uncompressed against logstash.log.2.gz + // at 25M, a ~56x difference on the generation that dominates the footprint. fprintf(fout, " missingok\n"); fprintf(fout, " compress\n"); - fprintf(fout, " delaycompress\n"); fprintf(fout, " notifempty\n"); fprintf(fout, "}\n"); From b34700afc60188148a18c70b336c2e7789462ed1 Mon Sep 17 00:00:00 2001 From: Jim Lin Date: Sun, 6 Sep 2026 16:56:58 +0800 Subject: [PATCH 3/4] fix(logrotate): run rotation scripts once per cycle, not once per file WriteLogRotateConf emitted prerotate/postrotate without sharedscripts, so logrotate ran them once per matched file -- its documented behaviour. Every script we generate is a 'signal the service once' operation, so a glob matching N files fired N signals in the same second. For httpd that is up to eleven overlapping 'systemctl reload httpd.service' calls racing each other in the master's worker lifecycle, which can take httpd down outright and 503 Horizon and Keystone on whichever control node holds the VIP (cubecos#1192). The same shape applied to syslog, where three of five files rotate and hex_trim_syslog therefore ran three times per night, and to prometheus's killall -HUP. It also had to be fixed before rotation moves to hourly: eleven reloads a day becomes two hundred and sixty four. Verified on accept-3cc and jim-1cc: forced rotation of the httpd config ran the postrotate script 11 and 9 times respectively before, exactly 1 after. Measured on the live service, a forced rotation produced 4 real httpd reloads before and 1 after, with httpd staying active and Horizon answering 302. Signed-off-by: Jim Lin Co-authored-by: Eandalf Co-authored-by: Claude Opus 5 (1M context) --- src/hex_sdk_library/logrotate/logrotate.cpp | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/src/hex_sdk_library/logrotate/logrotate.cpp b/src/hex_sdk_library/logrotate/logrotate.cpp index d19c5a4..19dee7e 100644 --- a/src/hex_sdk_library/logrotate/logrotate.cpp +++ b/src/hex_sdk_library/logrotate/logrotate.cpp @@ -83,6 +83,19 @@ WriteLogRotateConf(LogRotateConf conf) if (conf.retention > 0) fprintf(fout, " rotate %u\n", conf.retention); + // sharedscripts: run the pre/postrotate scripts once per rotation cycle rather + // than once per matched file. Without it logrotate's documented behaviour is + // per-file, and every script we generate is a "signal the service once" + // operation -- so a glob matching N files fired N reloads in the same second. + // + // /var/log/httpd/*.log matches 13 files on a control node, 11 of them non-empty: + // eleven overlapping `systemctl reload httpd.service` calls raced each other in + // the master's worker lifecycle and could take httpd down outright, 503ing + // Horizon and Keystone on whichever node held the VIP (cubecos#1192). The same + // shape applied to syslog (5 files) and prometheus. + if (!conf.preRotateCmds.empty() || !conf.postRotateCmds.empty()) + fprintf(fout, " sharedscripts\n"); + if (!conf.preRotateCmds.empty()) { fprintf(fout, " prerotate\n"); fprintf(fout, " %s\n", conf.preRotateCmds.c_str()); From cb13101559182b69e5e6aedf4a1bed8747eaf4fc Mon Sep 17 00:00:00 2001 From: Jim Lin Date: Tue, 8 Sep 2026 23:19:42 +0800 Subject: [PATCH 4/4] fix(logrotate): leave messages.1 for the compression that follows the trim hex_trim_syslog runs from the syslog postrotate, and logrotate runs postrotate *before* compression. Dropping delaycompress in 56fdc47 made messages.1 the generation logrotate compresses in the same cycle, so a trim that reaches .1 removes the file logrotate is about to open: trim: removing messages.1 error: unable to open /var/log/messages.1 for compression: No such file or directory That is the same class of error fea23e1 installed the script to stop -- it would have come back in a different form, on exactly the over-budget nights the trim exists for. Stopping the loop at .2 avoids it, and costs the budget nothing: the next rotation renames .1 to .2, so the same bytes become eligible one cycle later. Reported by SekiXu in review on #163, who measured the interaction and confirmed the set the script can match is byte-for-byte identical with and without delaycompress -- so this is the only thing that needed changing. Verified on jim-1cc (logrotate 3.18.0), in an isolated tree mirroring the real syslog config -- compress, no delaycompress, copytruncate, the trim in postrotate -- forced with a 1KB limit so the loop reaches the newest generation. With the old bound the "removing messages.1" line is followed by the compression error; with the new one gzip runs normally and no error is logged. The installed script stays inert at the real limit: total 179712 KB against LIMIT 4592725. Refs #163 Signed-off-by: Jim Lin Co-authored-by: Eandalf Co-authored-by: Claude Opus 5 (1M context) --- data/hex_syslogd/hex_trim_syslog.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/data/hex_syslogd/hex_trim_syslog.sh b/data/hex_syslogd/hex_trim_syslog.sh index 387d27d..fa98611 100644 --- a/data/hex_syslogd/hex_trim_syslog.sh +++ b/data/hex_syslogd/hex_trim_syslog.sh @@ -1,9 +1,19 @@ #!/bin/sh +# Drop /var/log/messages.N retentions, oldest first, until the whole messages set +# fits the byte budget config_syslogd computes (SYSLOG_DISK_PERC of the partition). +# +# Stops at .2 and never removes messages.1. This runs from logrotate's postrotate, +# which fires *before* compression, so messages.1 is the file logrotate is about to +# compress in this same cycle -- removing it under logrotate's feet makes the cycle +# log "unable to open /var/log/messages.1 for compression: No such file or +# directory", the same class of error installing this script was meant to stop. +# Nothing is lost from the budget by waiting: the next rotation renames .1 to .2, +# and this becomes eligible to remove it one cycle later. LIMIT=$1 I=14 total=`du -c /var/log/messages* |grep total |awk '{print $1}'` -while [ $total -gt $LIMIT -a $I -gt 0 ]; do +while [ $total -gt $LIMIT -a $I -gt 1 ]; do NAME=/var/log/messages.$I if [ -e $NAME ]; then logger "syslog message total size exceeded the limit. Removing retentions."