henyey mainnet daily — 2026-05-26

[tomerweller]

The quietest day yet — no commits on origin/main, no deploys, no crashes, no INV-H2 WARNs, zero state transitions in 24 h, no issues filed or closed. Validator has now been continuously Validating on 1a3b07106c3f for 44 h 24 m, all 46 monitor ticks OK.

Validator

• Build: 1a3b07106c3f (uptime 44h 24m; 0 deploys in last 24h)
• State: 🟢 Validating (age=5s, agree=21/missing=0, lag_ms=448)
• SCP propagation (first_to_self_externalize_seconds): 141 ms (current scalar)
• SCP slot-cycle (externalized_seconds): 2.96 s (down from 5.02s yesterday — network running faster)
• Memory: RSS 17.3 G, frag 22.5 % (heap_components_mb 5494 → 5490 → 5497 over last three reports — flat; RSS growth has plateaued ~17.3 G after the post-deploy settling)
• Disk: data 84 % (1.4 T of 1.7 T), session 3.1 G, mainnet 51 G (mainnet grew ~4 G in 24h — normal close churn)

Deploys (0)

(none) — no commits landed on origin/main in the window.

Incidents

(none) — no crashed-* / frozen-* rotations, no urgent issues filed, no MONITOR ACTION ticks, no log rotations.

Issues activity

Filed today (0):

• (none) in the 24 h window.

Closed today (0):

• (none) in the 24 h window.

Still open (selected, sorted by last activity):

• ⚪ HERDER spec adherence: Finish remaining persistence and recovery partial audit rows #2820 HERDER spec adherence: Finish remaining persistence and recovery partial audit rows — updated 2026-05-26 (active in /plan)
• ⚪ Force-converge residuals from PR #2884 (issue #2873) #2893 Force-converge residuals from PR Refresh OVERLAY and LEDGER spec-adherence reports #2884 (issue Refresh OVERLAY + LEDGER spec-adherence audits against current main #2873) — updated 2026-05-24
• ⚪ Force-converge residuals from PR #2875 (issue #2869) #2892 Force-converge residuals from PR Implement observer txset build + cache parity (HERDER §5.2) #2875 (issue HERDER §5.2: Non-validator txset build + cache parity #2869) — updated 2026-05-24
• ⚪ Force-converge residuals from PR #2835 (issue #2816) #2890 Force-converge residuals from PR Implement ctValidityOffset trigger-time and far-ahead abort semantics #2835 (issue HERDER spec adherence: Align ctValidityOffset trigger-time and abort semantics #2816) — updated 2026-05-24
• ⚪ Force-converge residuals from plan #2820 (HERDER §5.3 + §15.3 round-2 REVISE-MAJOR) #2895 Force-converge residuals from plan HERDER spec adherence: Finish remaining persistence and recovery partial audit rows #2820 — updated 2026-05-24
• ⚪ Force-converge residuals from PR #2885 (issue #2877) #2894 Force-converge residuals from PR Handle deferred auto-merge in /review-pr skill #2885 (issue Pipeline merge permission gap for PR #2875 #2877) — updated 2026-05-24
• ⚪ Enforce $HOME/data validation for reviewer/critic workspace overrides #2837 Enforce $HOME/data validation for reviewer/critic workspace overrides — updated 2026-05-24
• ⚪ HERDER spec adherence: Align ctValidityOffset trigger-time and abort semantics #2816 HERDER spec adherence: Align ctValidityOffset trigger-time and abort semantics — updated 2026-05-24
• ⚪ /review-pr sub-agents create huge worktrees in /tmp (11GB seen, near-OOM root FS) #2843 /review-pr sub-agents create huge worktrees in /tmp (11GB seen, near-OOM root FS) — updated 2026-05-21
• ⚪ Spec adherence: LEDGER — implement INV-L4 ConservationOfLumens #2804 Spec adherence: LEDGER — 18 findings (79%) — updated 2026-05-20
• ⚪ HERDER: Wire ScpPersistenceManager::restore_scp_state into startup #2769 HERDER: Wire ScpPersistenceManager::restore_scp_state into startup — updated 2026-05-20
• ⚪ Spec adherence: HERDER — 10 findings (86%) #2803 Spec adherence: HERDER — 10 findings (86%) — updated 2026-05-20

The project pipeline appears idle today — no triage, plan, do, or review-pr cycles ran. Last activity was the late afternoon of 2026-05-24 (just after the deploy went out).

Watch items

• State transitions: 0 in the last 24 h — first such day in the post-recovery window. The flip-cluster pattern (which appeared 3× yesterday: 15:21, 16:11, 19:27 UTC on the new binary) did not recur. Whether that's a temporary lull or the issue resolving itself is too early to tell.
• INV-H2 WARN count: 0 in last 24 h. Session total still 1 (from the 07:44 UTC 2026-05-25 self-correct on the new binary).
• RSS plateau: 16.2 G → 17.3 G over 24 h. Growth rate has slowed sharply — yesterday saw ~340 MB/h post-deploy settling, today only ~45 MB/h. Heap-components flat at ~5.5 G; growth in mmap_mb (14.96 → 14.97) is negligible. unaccounted_mb 4.6 → 4.7 G — slightly less than yesterday's 5.6 G level (post-recovery process has different RSS distribution from the previous incarnation).
• Disk usage: held perfectly steady at 84 % / 274–275 G free for the entire 24 h window — the agent worktree growth+cleanup cycles have balanced.
• Quorum.validated=false: noticed in the 13:41 snapshot. last_check_ledger=62717281 is ~27,770 ledgers (~38 h) behind current — the transitive intersection re-check hasn't fired since the deploy. Heartbeats continue to show heard_from_quorum=true, so the qset itself is healthy; just a stale validation timestamp. Worth checking whether the transitive-check timer resets correctly across restarts.
• Daily-summary cron: missed firing today at 13:07 (34 min past at post time). 5th miss in 8 days. The cron's reliability is consistently degrading.

Tick aggregates (last 24h)

• Cron: 6c74937d (daily-summary, recurring 13:07 UTC) — missed today; 5th miss in 8 days. Monitor-tick remains on a ScheduleWakeup dynamic loop (~30 m cadence).
• Ticks observed: 46 (close to nominal 48)
• Outcomes: 46 OK / 0 ACTION / 0 WARNING / 0 OFFLINE / 0 DEPLOY — first 24 h window with 100 % OK rate since the post-recovery period
• Alarm-firings: 0
• Self-reflection events: 0
• Skill commits today: 0

Open questions

• The 24 h state_transitions=0 is a real signal — the recurring 4-flip clusters that have been a daily feature for the last 4 days simply didn't happen today. Is this (a) the network upstream calming down, (b) coincidence (we'll see clusters again tomorrow), or (c) a daylight/timezone-correlated pattern we hadn't noticed? Worth tracking the clusters-per-day count over the next week to characterize it.
• Quorum.validated=false with stale last_check_ledger=62717281 (38 h old) — should the transitive-check timer be running more often, or is once per restart by design? If by design, the validated field is misleading in steady-state because it lags the actual quorum state by however long since boot.
• Daily-summary cron has now missed 5 of the last 8 days, with no successful fires since 2026-05-23 (and that one was 18 min late). The cron mechanism is effectively dead — time to move the post to a host-side scheduler or fold it into monitor-tick.