From b5ba5d8c3739a8882680eca4c4aebe63514842b1 Mon Sep 17 00:00:00 2001 From: Jim Lin Date: Wed, 9 Sep 2026 12:36:26 +0800 Subject: [PATCH 1/4] fix(logrotate): bound retention by age, so the days promise survives hourly runs cubesys.log.default.retention is a published tuning whose help text reads "Set log file retention policy in days", default 14, and it is the only retention control there is: all ~40 LogRotateConf declarations in hex and cubecos pass retention 0, WriteLogRotateConf only emits `rotate` when that is non-zero, so every generated config inherits the global this function writes. Measured on accept-3cc: 1 of 62 files in /etc/logrotate.d sets its own rotate. logrotate's `rotate` is a count of GENERATIONS, not days. The two coincide only while every log rotates exactly once a day, which stopped being true when the logrotate timer moved to hourly: each generated config pairs `daily` with `maxsize 128M` and rotates on whichever comes first, so a log that passes the cap every hour kept `rotate` hours of history instead of `rotate` days. That is the audit window, and it silently shrank by 24x for exactly the busiest logs. So bound the window by age and leave the count as a safety net rather than the policy: maxage carries the days promise, and rotate is sized for the worst case the hourly timer allows -- 24 rotations a day -- so it cannot bind first. A log that rotates once a day is unaffected; maxage removes its 15th day exactly as rotate 14 used to. Verified on accept-3cc against logrotate 3.18.0, as a steady-state simulation rather than a pre-seeded directory -- logrotate never revisits generations beyond `rotate`, so seeding more than that measures nothing. Scaled 7x (maxage 2 days, rotate 48), starting empty and ageing every generation by one interval between rotations: rotating once a day rotate 2 kept 2 gens, oldest 1.0 days rotate 48+maxage2 kept 3 gens, oldest 2.0 days tripping maxsize hourly rotate 2 kept 2 gens, oldest 0.0 days <-- 2 hours rotate 48+maxage2 kept 48 gens, oldest 1.9 days Applied to the live global config on jim-1cc it renders as `rotate 336` + `maxage 14`, logrotate parses it, and a real pass over 184 considered files exits 0 with no output. Clean CXX logrotate.o and AR libhex_sdk.a in the build jail, with the emitted directive list in the object now daily / su root syslog / rotate %d / maxage %d / create / include. Reported by traviswu-bigstack in review on bigstack-oss/cubecos#1433. Refs https://github.com/bigstack-oss/cubecos/pull/1433 Signed-off-by: Jim Lin Co-authored-by: Eandalf Co-authored-by: Claude Opus 5 (1M context) --- src/hex_sdk_library/logrotate/logrotate.cpp | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/src/hex_sdk_library/logrotate/logrotate.cpp b/src/hex_sdk_library/logrotate/logrotate.cpp index 19dee7e..6037661 100644 --- a/src/hex_sdk_library/logrotate/logrotate.cpp +++ b/src/hex_sdk_library/logrotate/logrotate.cpp @@ -18,9 +18,24 @@ WriteDefLogRotateConf(int retention = 4) return false; } + // retention is a number of DAYS -- that is what cubesys.log.default.retention + // promises operators ("Set log file retention policy in days") and what the + // audit expectation is built on. logrotate's `rotate` is a number of + // GENERATIONS, so the two only coincide while every log rotates exactly once a + // day. They stopped coinciding when the logrotate timer moved to hourly: every + // generated config pairs `daily` with `maxsize 128M`, which rotates on + // whichever comes first, so a log that passes the cap every hour used to keep + // `rotate` hours rather than `rotate` days. + // + // So bound the window by age and let the count be the safety net, not the + // policy: maxage is the days promise, and rotate is sized for the worst case + // the hourly timer allows (24 rotations a day) so it cannot bind first. A log + // that rotates once a day is unaffected -- maxage removes its 15th day exactly + // as `rotate 14` used to. fprintf(fout, "daily\n"); fprintf(fout, "su root syslog\n"); - fprintf(fout, "rotate %d\n", retention); + fprintf(fout, "rotate %d\n", retention * 24); + fprintf(fout, "maxage %d\n", retention); fprintf(fout, "create\n"); fprintf(fout, "include /etc/logrotate.d\n"); From 5cfb9978631176023f7fc8589cd9d90d05b5b18f Mon Sep 17 00:00:00 2001 From: Jim Lin Date: Wed, 9 Sep 2026 12:37:28 +0800 Subject: [PATCH 2/4] build(rootfs): drop the cron.hourly logrotate move, dead since the systemd timer full_rootfs_install ran chroot $(ROOTDIR) mv /etc/cron.daily/logrotate /etc/cron.hourly/ || true to make logrotate run hourly. RHEL and CentOS stopped shipping that cron job when logrotate moved to a systemd timer, so on a stream9 rootfs the source path does not exist, the mv fails, and `|| true` swallows it. Verified on a running node: the logrotate rpm owns only /usr/lib/systemd/system/logrotate.service /usr/lib/systemd/system/logrotate.timer /etc/cron.daily/logrotate is absent and /etc/cron.hourly holds nothing but 0anacron. The line above it disables crond as well, so even a successful move would have needed something to re-enable cron before it could fire. The intent was right and has simply been unreachable: the effective schedule on every image built since the switch has been the packaged timer's OnCalendar=daily, which is what made the maxsize 128M cap in every generated config unenforceable. cubecos ships a logrotate.timer drop-in to set the hourly schedule where it now belongs; this removes the line that looked like it was already doing that job, so the next person reading either repo is not told two different things. No behaviour change on any built image -- the command has never had an effect. Refs https://github.com/bigstack-oss/cubecos/pull/1433 Signed-off-by: Jim Lin Co-authored-by: Eandalf Co-authored-by: Claude Opus 5 (1M context) --- rootfs/Makefile | 1 - 1 file changed, 1 deletion(-) diff --git a/rootfs/Makefile b/rootfs/Makefile index 0a13994..e5e5021 100755 --- a/rootfs/Makefile +++ b/rootfs/Makefile @@ -164,7 +164,6 @@ full_rootfs_install:: $(Q)sed -i -e "s/.*PermitRootLogin .*/PermitRootLogin yes/" -e "s/.*PasswordAuthentication .*/PasswordAuthentication yes/" $(ROOTDIR)/etc/ssh/sshd_config $(Q)ln -sf /lib/systemd/system/systemd-hexctl-user.service $(ROOTDIR)/etc/systemd/system/multi-user.target.wants/systemd-hexctl-user.service $(Q)chroot $(ROOTDIR) systemctl disable crond sshd - $(Q)chroot $(ROOTDIR) mv /etc/cron.daily/logrotate /etc/cron.hourly/ || true $(Q)chroot $(ROOTDIR) mv /etc/logrotate.d/rsyslog /etc/logrotate.d/rsyslog.disabled $(Q)chroot $(ROOTDIR) bash -c "rm -rf /usr/local/share/* /usr/share/{man,doc,licenses} /usr/src /usr/local/src /var/{log,cache}/* /tmp/* /lib/.build-id" From e2a406047280073c1bd3bf668ea1502a193c4a42 Mon Sep 17 00:00:00 2001 From: Jim Lin Date: Sat, 12 Sep 2026 10:48:49 +0800 Subject: [PATCH 3/4] fix(logrotate): match the generations that exist, so the trim can remove something hex_trim_syslog has never been able to remove a compressed generation, and since cb13101 it has not been able to remove anything at all. Two mismatches with the directory it runs against, one of them mine: - It tested `-e /var/log/messages.$I` only. WriteLogRotateConf always emits `compress` and the global conf sets no `dateext`, so at postrotate time the directory holds `messages`, a still-uncompressed `messages.1`, and `messages.2.gz` onward. The only unsuffixed generation that ever exists is .1 -- which cb13101 correctly stopped it from touching, leaving a loop with nothing it could match. - `I=14` mirrored the old global `rotate 14`. b5ba5d8 raises that to `retention * 24`, rendering as 336, so even with .gz matching the scan would have covered generations 2..14 of up to 336 -- and a log that reaches 336 generations is exactly the busy log the budget exists for. Take the generation list from the filesystem instead of counting down from a literal. That removes the coupling that has now drifted twice, and it is immune to whatever `rotate` becomes next. messages.1 stays excluded for cb13101's reason: postrotate runs before compression, so .1 is the file logrotate is about to open. Reported by SekiXu in review on #165, who also traced it back to his own #163 measurement -- "the matchable set is byte-for-byte identical with and without delaycompress" was already this fact, one step short of the conclusion. Verified on jim-1cc, both revisions run out of git with only /var/log rewritten to a scratch dir, seeded to the real postrotate shape observed on that node (messages, plain messages.1, messages.2.gz..messages.14.gz) and LIMIT set to a sixth of the total: shipped total 3236 -> 3236 removed nothing fixed total 3236 -> 688 messages.1 survived Inert at the real budget: with the messages set at 168816K against a 5% budget of 4592725K on accept-3cc, a run changes nothing. And seeded to 336 generations with a large one at .300, it trims oldest-first down to .299 -- generations the old bound could never reach. Refs https://github.com/bigstack-oss/cubecos/pull/1433 Signed-off-by: Jim Lin Co-authored-by: Eandalf Co-authored-by: Claude Opus 5 (1M context) --- data/hex_syslogd/hex_trim_syslog.sh | 49 ++++++++++++++++++++--------- 1 file changed, 34 insertions(+), 15 deletions(-) diff --git a/data/hex_syslogd/hex_trim_syslog.sh b/data/hex_syslogd/hex_trim_syslog.sh index fa98611..fd569c8 100644 --- a/data/hex_syslogd/hex_trim_syslog.sh +++ b/data/hex_syslogd/hex_trim_syslog.sh @@ -3,22 +3,41 @@ # Drop /var/log/messages.N retentions, oldest first, until the whole messages set # fits the byte budget config_syslogd computes (SYSLOG_DISK_PERC of the partition). # -# Stops at .2 and never removes messages.1. This runs from logrotate's postrotate, -# which fires *before* compression, so messages.1 is the file logrotate is about to -# compress in this same cycle -- removing it under logrotate's feet makes the cycle -# log "unable to open /var/log/messages.1 for compression: No such file or -# directory", the same class of error installing this script was meant to stop. +# Generations are discovered from what is on disk rather than counted down from a +# literal, because both halves of the old loop had drifted out of step with the +# files it was supposed to match: +# +# - it tested `-e /var/log/messages.$I` only, and every rotated generation is +# compressed. WriteLogRotateConf always emits `compress` and the global conf sets +# no `dateext`, so at postrotate time the directory holds `messages`, a still +# uncompressed `messages.1`, and `messages.2.gz` onward. The one unsuffixed +# generation that ever exists is `.1` -- which this must not touch, see below -- +# so the loop could never match anything at all. +# - it started at I=14, mirroring the old global `rotate 14`. That is now +# `retention * 24` (336 at the default), so even with .gz matching it would have +# scanned 2..14 of up to 336 -- and a log that reaches 336 generations is exactly +# the busy log the budget exists for. +# +# Taking the list from the filesystem removes the coupling that broke twice. +# +# messages.1 is deliberately skipped. This runs from logrotate's postrotate, which +# fires *before* compression, so .1 is the file logrotate is about to compress in +# this same cycle -- removing it under logrotate's feet makes the cycle log +# "unable to open /var/log/messages.1 for compression: No such file or directory". # Nothing is lost from the budget by waiting: the next rotation renames .1 to .2, -# and this becomes eligible to remove it one cycle later. +# and it becomes eligible one cycle later. LIMIT=$1 -I=14 total=`du -c /var/log/messages* |grep total |awk '{print $1}'` -while [ $total -gt $LIMIT -a $I -gt 1 ]; do - NAME=/var/log/messages.$I - if [ -e $NAME ]; then - logger "syslog message total size exceeded the limit. Removing retentions." - rm -f $NAME - total=`du -c /var/log/messages* |grep total |awk '{print $1}'` - fi - I=$((I-1)) + +# oldest first: sort by generation number, highest first, .gz or not +for NAME in `ls /var/log/messages.* 2>/dev/null | + sed -n 's|^/var/log/messages\.\([0-9][0-9]*\)\(\.gz\)\{0,1\}$|\1 &|p' | + sort -rn | awk '{print $2}'`; do + [ $total -gt $LIMIT ] || break + case "$NAME" in + /var/log/messages.1|/var/log/messages.1.gz) continue ;; + esac + logger "syslog message total size exceeded the limit. Removing retentions." + rm -f $NAME + total=`du -c /var/log/messages* |grep total |awk '{print $1}'` done From 1d8bd9b06ca741467e007d5cf304ca181872d8d0 Mon Sep 17 00:00:00 2001 From: Jim Lin Date: Mon, 14 Sep 2026 15:56:40 +0800 Subject: [PATCH 4/4] fix(logrotate): size rotate for the daily schedule, and say what maxage is for Two changes, both following cubecos reverting the logrotate timer to daily on the product owner's call. `rotate` goes back to the retention value. It was raised to retention * 24 for the worst case an hourly timer allowed -- 24 rotations a day -- and on the daily schedule that is 336 generations for a log that can produce at most 14. Nothing kept the extra files, but the number described a schedule we no longer ship. The comment went with it. It had justified maxage entirely by the hourly timer ("they stopped coinciding when the logrotate timer moved to hourly"), which would have left the code explained by something that no longer exists -- and worse, it claimed rotate was sized so it could not bind first, which stops being true the moment rotate is the day count again. So state what the two bounds actually do, since they are not redundant: rotate caps how many files a busy log leaves behind; on a daily schedule that is also its age, one generation per day. maxage caps their age directly, and is the only one of the two that says anything about a log that rotates *rarely* -- with notifempty and a quiet service, generations sit for months while staying well inside a count of 14, and without maxage they are kept indefinitely. Measured on accept-3cc, scaled 7x (retention 2), logrotate 3.18.0: daily rotation, 20 cycles rotate 2 2 gens, oldest 1.0 days rotate 2+maxage 2 2 gens, oldest 1.0 days a log that rotates rarely rotate 2 2 gens, oldest 40 days (generations already 40d old) rotate 2+maxage 2 1 gen, oldest 0 days Identical on the shipped schedule, which is why the change is safe; the second row is the case maxage exists for. The residual is recorded in the comment rather than papered over: if something rotates a log more than once in a day -- an operator's `logrotate -f`, a fixpack driving a rotation -- rotate spends a generation without spending a day and the window shrinks below the advertised days. That is long-standing behaviour, and sizing rotate past the daily count is what this commit is undoing. Clean CXX logrotate.o and AR libhex_sdk.a, emitted directive list unchanged. Refs bigstack-oss/cubecos#1192 Signed-off-by: Jim Lin Co-authored-by: Eandalf Co-authored-by: Claude Opus 5 (1M context) --- src/hex_sdk_library/logrotate/logrotate.cpp | 34 +++++++++++++-------- 1 file changed, 22 insertions(+), 12 deletions(-) diff --git a/src/hex_sdk_library/logrotate/logrotate.cpp b/src/hex_sdk_library/logrotate/logrotate.cpp index 6037661..e3a6616 100644 --- a/src/hex_sdk_library/logrotate/logrotate.cpp +++ b/src/hex_sdk_library/logrotate/logrotate.cpp @@ -20,21 +20,31 @@ WriteDefLogRotateConf(int retention = 4) // retention is a number of DAYS -- that is what cubesys.log.default.retention // promises operators ("Set log file retention policy in days") and what the - // audit expectation is built on. logrotate's `rotate` is a number of - // GENERATIONS, so the two only coincide while every log rotates exactly once a - // day. They stopped coinciding when the logrotate timer moved to hourly: every - // generated config pairs `daily` with `maxsize 128M`, which rotates on - // whichever comes first, so a log that passes the cap every hour used to keep - // `rotate` hours rather than `rotate` days. + // audit expectation is built on. logrotate's `rotate` counts GENERATIONS, so + // on its own it only ever approximates that promise, and the approximation + // holds exactly while a log rotates once per day and no more. // - // So bound the window by age and let the count be the safety net, not the - // policy: maxage is the days promise, and rotate is sized for the worst case - // the hourly timer allows (24 rotations a day) so it cannot bind first. A log - // that rotates once a day is unaffected -- maxage removes its 15th day exactly - // as `rotate 14` used to. + // The two bounds do different work, which is why both are emitted: + // + // rotate caps the number of files a busy log can leave behind. On the + // shipped daily schedule that is also its age, one generation per + // day. + // maxage caps their age directly, and is the only one of the two that says + // anything about a log that rotates *rarely*. `notifempty` and a + // quiet service mean generations can sit for months and still be + // well inside a count of 14 -- without maxage they are kept + // indefinitely, long past the window the tuning advertises. + // + // What neither covers, and is worth knowing rather than papering over: if + // something rotates a log more than once in a day -- an operator's + // `logrotate -f`, a fixpack or enabler driving a rotation -- `rotate` spends a + // generation without spending a day and the window shrinks below the + // advertised days. That is long-standing behaviour and this does not change + // it; sizing `rotate` past the count needed for a daily schedule would, at the + // cost of keeping far more files than the schedule can ever produce. fprintf(fout, "daily\n"); fprintf(fout, "su root syslog\n"); - fprintf(fout, "rotate %d\n", retention * 24); + fprintf(fout, "rotate %d\n", retention); fprintf(fout, "maxage %d\n", retention); fprintf(fout, "create\n"); fprintf(fout, "include /etc/logrotate.d\n");