fix(logrotate): bound retention by age, and drop the dead cron.hourly move

[Eandalf-Bigstack]

What type of PR is this?

• bug

What this PR does / why we need it

Both halves came out of review on bigstack-oss/cubecos#1433, which moves the logrotate timer to hourly so the maxsize 128M cap in every generated config is actually evaluated.

• Retention is bounded by age, not by generation count. cubesys.log.default.retention is a published tuning whose help text reads "Set log file retention policy in days", default 14 — and it is the only retention control there is. All ~40 LogRotateConf declarations in hex and cubecos pass retention 0, and WriteLogRotateConf only emits rotate when that is non-zero, so every generated config inherits the global WriteDefLogRotateConf writes. Measured on a live 3-node cluster: 1 of 62 files in /etc/logrotate.d sets its own rotate. But logrotate's rotate counts generations, which equals days only while a log rotates exactly once a day. With the timer hourly, each config pairs daily with maxsize 128M and rotates on whichever comes first — so a log passing the cap every hour kept rotate hours of history. maxage now carries the days promise and rotate is sized for the worst case the hourly timer allows (24/day) so it cannot bind first.
• The cron.hourly move is removed, having never once taken effect. full_rootfs_install moved /etc/cron.daily/logrotate into /etc/cron.hourly/, guarded with || true. RHEL/CentOS stopped shipping that cron job when logrotate moved to a systemd timer, so the source path does not exist on a stream9 rootfs, the mv fails and the guard swallows it.

Which issue(s) this PR fixes

Refs bigstack-oss/cubecos#672

Special notes for your reviewer

This is the hex half of a pair. bigstack-oss/cubecos#1433 carries the logrotate.timer drop-in that makes the schedule hourly, plus a submodule bump onto this branch's head. Neither half is right alone: hourly without maxage shrinks the audit window 24x for the busiest logs, and maxage without hourly leaves the size cap unenforceable.

Why rotate had to change too, and not just maxage. maxage prunes by age but does not extend anything — rotate 14 still deletes the 15th generation regardless of its age, so on an hourly-rotating log it would have bound first at 14 hours and maxage 14 would never have been reached. rotate retention * 24 is the count that cannot bind before 14 days at the fastest rate the timer permits. A log rotating once a day is unaffected either way: maxage removes its 15th day exactly as rotate 14 used to.

On the removed line, "no behaviour change" is meant literally. The command has never had an effect on any image built since the systemd-timer switch. Verified on a running node — the logrotate rpm owns only /usr/lib/systemd/system/logrotate.{service,timer}, /etc/cron.daily/logrotate is absent, and /etc/cron.hourly holds nothing but 0anacron. The line immediately above it disables crond, so even a successful move would have needed cron re-enabled before it could fire. It is removed so the two repos stop describing two different schedules.

Additional documentation

Verified on accept-3cc and jim-1cc, logrotate 3.18.0.

For "the days promise holds at any rotation frequency" -- a steady-state
simulation, not a pre-seeded directory: logrotate never revisits generations
beyond `rotate`, so seeding more than that measures nothing. Scaled 7x
(maxage 2 days, rotate 48 = 24*2), starting empty, ageing every existing
generation by one interval between forced rotations:

  rotating once a day       rotate 2            kept  2 gens, oldest 1.0 days
                            rotate 48 + maxage 2 kept  3 gens, oldest 2.0 days
  tripping maxsize hourly   rotate 2            kept  2 gens, oldest 0.0 days   <- 2 hours
                            rotate 48 + maxage 2 kept 48 gens, oldest 1.9 days

Scaled to the shipped default the second row is the regression this fixes:
14 generations of an hourly-rotating log is 14 hours, not 14 days.

Applied to the live global config on jim-1cc, the writer renders exactly:

  daily
  su root syslog
  rotate 336
  maxage 14
  create
  include /etc/logrotate.d

logrotate parses it, and a real pass over 184 considered files exits 0 with no
output. A no-op pass costs 17-32 ms over 222 files across 62 configs on
accept-3cc, so the hourly evaluation itself is not a measurable cost.

Build: clean CXX logrotate.o and AR libhex_sdk.a; the emitted directive list in
the object is now daily / su root syslog / rotate %d / maxage %d / create /
include /etc/logrotate.d.

[SekiXu]

rotate retention * 24 + maxage retention is the right shape, and the steady-state
simulation is the right way to have measured it — seeding beyond rotate would have proved
nothing. One thing to fold in while this is open, because it is in the lines this PR moves.

hex_trim_syslog cannot remove anything

Two separate mismatches, one of them from this PR.

.gz is never matched. The loop tests [ -e /var/log/messages.$I ]. At postrotate time
— before compression, as the comment added in cb13101 says — the directory holds
messages, an uncompressed messages.1, and messages.2.gz onward: the global conf has no
dateext, and WriteLogRotateConf always emits compress. So the only unsuffixed
generation that ever exists is messages.1, which cb13101 correctly stops it from
touching. That fix is right; what it leaves is a loop with nothing it can match.

Ran both revisions straight out of git with only /var/log rewritten to a scratch dir, so
after the substitution the sole difference between the files is the loop bound. Directory
seeded to the postrotate state above, LIMIT set six times under total:

                 removed          total (512B blocks)
d500f450         messages.1       6000 -> 5600     still 5.6x over budget
5cfb9978         nothing          6000 -> 6000     rc=0

fea23e1 made this reachable by shipping the script for the first time, and
config_syslogd.cpp:117 makes it the only enforcement SYSLOG_DISK_PERC has.

This is where my own #163 measurement landed and I did not follow it through: "the matchable
set is byte-for-byte identical with and without delaycompress" was already this fact — both
cases leave exactly messages.1 matchable — one step short of the conclusion. Mine to have
caught before approving #163.

I=14 no longer matches the retention. It mirrors the old global rotate 14; b5ba5d8
raises that to retention * 24, rendering as rotate 336. Even with .gz matching fixed
the scan would cover generations 2..14 of up to 336 — and a log that reaches 336 generations
is exactly the busy log the budget exists for.

Both want the same edit: remove messages.$I and messages.$I.gz, and take the upper bound
from the retention value WriteDefLogRotateConf already has rather than a literal.

[Eandalf-Bigstack]

@SekiXu thank you for checking this — and for not stopping at the shape of the change. You found that the script this PR touches could not remove anything at all, and you were right on both counts. Fixed in e2a4060.

.gz was never matched. Exactly as you describe: WriteLogRotateConf always emits compress, the global conf sets no dateext, so at postrotate time the directory is messages, an uncompressed messages.1, and messages.2.gz onward. The one unsuffixed generation that ever exists is .1 — which cb13101 had just stopped it touching. That combination is mine, and it turned a script that could remove one generation into one that can remove none. The real directory on jim-1cc is precisely the shape you predicted:

134605633  /var/log/messages
 55332983  /var/log/messages.1
   932982  /var/log/messages.2.gz   ...   238866  /var/log/messages.14.gz

I=14 no longer tracks the retention. Also right, and it is the second time that literal has drifted out of step with the config — first against compress, now against rotate retention * 24.

Because of that I went one step past your suggestion. Rather than taking the bound from the retention value, the generation list now comes from the filesystem: sort whatever messages.<N> and messages.<N>.gz actually exist, highest first, and remove until the set is under budget. It needs no plumbing from cubecos's retention tuning into hex's config_syslogd, and it cannot drift again whatever rotate becomes. messages.1 stays excluded for cb13101's reason.

Reproduced your A/B before and after, both revisions run out of git with only /var/log rewritten to a scratch dir, seeded to the real postrotate shape above, LIMIT at a sixth of the total:

shipped   total 3236 -> 3236   removed nothing
fixed     total 3236 ->  688   messages.1 survived

Still inert when it should be — on accept-3cc the messages set is 168816K against a 5% budget of 4592725K, and a run changes nothing. And seeded to 336 generations with a large one at .300, it trims oldest-first down to .299: generations the old bound could never have reached.

On your note that this is where your own #163 measurement landed — "the matchable set is byte-for-byte identical with and without delaycompress" was indeed already this fact. For what it is worth, I read that line, agreed with it, and drew the same wrong conclusion from it while writing cb13101. It took you re-reading your own measurement to see what it actually said, which is the harder direction.

The rest of this PR is unchanged: maxage retention with rotate retention * 24 so the count cannot bind first, and the dead cron.hourly move removed. cubecos#1433 bumps the submodule onto this head.

🤖 Generated with Claude Code

[SekiXu]

Approving. Taking the generation list from the filesystem is better than the bound I suggested — it needs no retention value plumbed from cubecos's config_syslogd into hex, and it cannot drift again whatever rotate becomes. That is the more durable answer to a literal that has now drifted twice.

Reproduced the A/B independently rather than taking the numbers, both revisions run straight out of git with only /var/log rewritten to a scratch dir. Seeded to the postrotate shape — messages 2000K, an uncompressed messages.1 800K, .2–.14.gz at 100K each — with LIMIT at a sixth of the total:

                 total        survivors
shipped   8200 -> 8200        everything      removed nothing
this head 8200 -> 5600        messages, messages.1

And seeded to 336 generations with a large one at .300, LIMIT just under the total: it trims oldest-first and stops at .299, having removed 37 files, with messages.1 intact. Generations the old bound could never have reached, and the exclusion holding.

One nit, in the lines this rewrites. logger fires once per removed file, so the 336-generation case writes ~335 lines into the log it is trimming; the old bound capped that at 13. It is small against a log that has busted a 5% partition budget, but a single line before the loop says the same thing and does not scale with the damage.

The other half is unchanged and still right: maxage retention with rotate retention * 24 so the count cannot bind first, and the cron.hourly move removed as dead since the systemd timer.