henyey mainnet daily — 2026-04-30

[tomerweller]

Validator

• Build: 96998244 (uptime 17m on this binary; ~15 deploys in last 24h, ~96m avg between deploys under the new 60m cool-down)
• State: 🟢 Validating, ledger=62355645, age≈4s, agree=21/missing=0
• SCP propagation / slot-cycle metrics: warmup (post-restart, histogram count=0)
• Memory: RSS 5.7 GB, frag 24.1% (settling post-warmup); heap_components 4890 MB stable; unaccounted -554 MB ✓
• Disk: data 5% (74 G of 1.7 T), root 97% (94 G of 97 G — CRITICAL), session 3.2 G, mainnet 24 G

Deploys (~15)

Major themes today — heavy security-audit follow-ups (AUDIT-200..217, ~12 fixes), correctness refactors (FeeMode/FeeStrategy enums, InvariantManager subsystem, PreparedTransactionSet newtype, hostname/peer-address validation), parity hardening (XDR-byte sort, sealed-box without crypto_box, low-order Curve25519 reject), and 2 monitor-side skill commits.

sha	subject	watch sentinel
bb1d8bb6	Serialize SCP envelope processing with ledger close	silent
2ee49eb6	Move SCP queue drain from advance_tracking_slot to ledger_closed	silent
604f5894	Treat HTTP_PORT=0 / HTTP_QUERY_PORT=0 as disabled in compat	silent
52219185	Deduplicate keys in getLedgerEntries to prevent amplification	silent
06010c8c	Add status index and query budget for getTransactions	silent
3d903a97	Tighten get_string_array to reject non-string elements	silent
002c4e7f	Update partial-fee test to use FeeMode::Skip enum	silent
260f84d7	Extract record_delete_meta helper to enforce delete ordering	silent
724abcd2	Clean up stale 'effective_min' references	silent
d0ea81ca	Introduce InclusionFee / TotalFee / ResourceFee newtypes	silent
e08cd6e2	Add regression tests for outbound timeout domain separation	silent
e7d57f4d	Extract shared SQLite progress-handler helper	silent (deployed via fresh-sync recovery)
96998244	Validate numeric IPv4 hosts and route HTTP connect through shared parser	silent
2abfffec	monitor-{tick,loop}: align cadence wording with 20m cron	n/a (skill)
3a0b9834	monitor-tick: raise deploy cool-down from 30m to 60m	n/a (skill)

Plus ~30 commits landed but rolled into bundles by the cool-down (only HEAD-of-bundle deploys actually happen).

Incidents (1)

• 🟢 Session directory ~/data/74535976/ wiped externally between 11:27Z and 11:47Z UTC
  • Window: 11:27Z → ~12:07Z (~40 min total downtime + recovery)
  • Symptom: validator process gone; ~/data/74535976/ (logs, binary, last_ledger, cargo-target) all deleted; ~/data/mainnet/mainnet.db reduced from ~25 GB to a 1.7 GB partial — clearly external action, not validator self-shutdown
  • Root cause: external (likely a parallel agent or maintenance process performing data cleanup; system uptime 51 days, no reboot)
  • Resolved by: monitor-tick recreated session dir, rebuilt binary on e7d57f4d (5m 55s), wiped the 1.7 GB partial state, fresh-sync from network archive (catchup completed in ~12 min — much faster than the 4 h fresh-start budget thanks to recent network position)

Issues activity

Filed today (~58 — heavy audit batch + plan-do-review follow-ups):

• 🔴 🚨 Hash mismatch FATAL at L62342648 on 9f3ac628 — catchup-replay path divergence (different from #2074 gap-fill close) #2098 [urgent] (closed) — catchup-replay hash mismatch (different from gap-fill 🚨 Hash mismatch FATAL at L62338433 — gap-fill close after buffered sequence gap (sequence_gap=1) closed L62338432 with divergent hash; immediately followed #2073 SYNC FAILURE #2074)
• 🔴 Fix XDR-byte sort of hot_archive_restored_keys in apply.rs to match stellar-core #2110 [urgent,follow-up] (closed) — XDR-byte sort hot_archive_restored_keys parity
• Audit batch ([AUDIT-200] through [AUDIT-217]): [AUDIT-200] Locally Emitted SCP Envelopes Do Not Update Slot v-blocking State #2083, [AUDIT-201] Pull-Control Messages Pollute the Global FloodGate #2084, [AUDIT-202] Arbitrage flood damping skips upstream ban-and-drop cleanup #2085, Add partial-fee regression tests for parallel and tx-crate paths #2087, [AUDIT-204] Local ballot self-emission bypasses upstream validation gate #2089, [AUDIT-205] Compat PUBLIC_HTTP_PORT ignored for HTTP_QUERY_PORT, making public query servers localhost-only #2090, [AUDIT-206] getLedgerEntries duplicate keys can amplify one large bucket entry into a large response #2093, [AUDIT-207] getTransactions status filters can force retained transaction scans #2094, [AUDIT-208] Unknown FloodDemand Hashes Trigger Per-Hash DontHave Amplification #2095, [AUDIT-209] Apply-Lag Deferred Validation Can Permanently Suppress Local SCP Emission #2096, [AUDIT-210] Predictable world-temp publish directory lets local users pin archive publishing #2097, [AUDIT-211] Recent post-LCL envelopes are pre-filtered before SCP can replay final messages #2100, [AUDIT-212] Outbound handshakes use TCP connect timeout instead of auth timeout #2101, [AUDIT-213] Compat invariant checks are accepted but never enabled #2102, [AUDIT-214] Resource-fee-heavy Soroban transactions can make Henyey sign rejected tx set #2103, [AUDIT-215] Live recovery catchup ignores configured complete/recent replay depth #2104, [AUDIT-216] XDR-Length-Sorted Nomination Values Diverge From Stellar-Core Ordering #2105, [AUDIT-217] Watcher mode skips run-command catchup but performs App::run fallback catchup #2106 — most closed same-day
• Refactor / centralization follow-ups: Replace deduct_fee bool with typed execution mode enum #2086 (FeeMode), Add apply-lag restoration for MaybeValidDeferred future-slot path #2091 (apply-lag MaybeValidDeferred), Centralize local self-envelope bookkeeping in Slot #2092 (Slot bookkeeping), Add regression test for handle_flood_demand demand-miss behavior #2107..Convert overlay peer config from Vec<String> to Vec<PeerAddress> #2140 (validation matrix, runtime state, parser tightening, port/peer-address validation)

Closed today (~50): #2098 + #2110 (urgent), #2083..#2106 audit batch (most), and ~30 follow-ups #2087..#2140

Still open (sample, ~10): #2106 [audit-info] AUDIT-217 watcher-mode catchup, #2134 (FeeRate newtype), #2135 (closing-gate replay variant test), #2136 (peer_port != 0 validation), #2137 (RPC port collision), #2138 (top-level VALIDATORS/HISTORY parsing), #2139 (RPC getEvents filter), #2140 (overlay PeerAddress conversion). Plus #2041 [not-ready] Quickstart build/complete exit 1 flake (CI hygiene).

Watch items

• 🚨 Root partition at 97% (94 G of 97 G; only 3.3 GB free). Cargo target dirs from parallel agents (pdr-2126, etc.) appear to be eating into root. The /home/tomer/data partition has plenty of room (5%). Worth running a cleanup of pdr-*/debug/ build caches.
• validator_status: 🔴 OFFLINE (~40 min) → 🟢 Validating after fresh-sync recovery. Resolved.
• cool-down floor: raised from 30m to 60m on 2026-04-30. First full day under new rule produced ~15 deploys vs ~23-25 yesterday under 30m — closer to the desired settling cadence.

Tick aggregates (last 24h)

• Cron: 75315f0c (every 20m, schedule 3,23,43 * * * *). Pre-wipe ticks lost (tick-history.jsonl was inside the wiped session dir).
• Outcomes (post-wipe only, ~6 ticks): 4 OK / 2 ACTION / 0 WARNING / 0 OFFLINE
• Self-reflection events: 0
• Skill commits today: 3
  • b8d88be0 "monitor-tick: auto-wipe corrupt persisted state after repeated FATAL" (yesterday's evening, included)
  • 3a0b9834 "monitor-tick: raise deploy cool-down from 30m to 60m"
  • 2abfffec "monitor-{tick,loop}: align cadence wording with 20m cron"

Open questions for the operator

• Root partition at 97%. Multiple parallel cargo build trees under /home/tomer/data/pdr-*/debug/ look like they're driving the pressure. Want me to clean stale pdr-*/debug/ directories that no longer have a running cargo process? It's destructive (~tens of GB recoverable) but harmless to those agents — they'll rebuild on next run.
• Session dir wipe protection. The session dir ~/data/74535976/ was deleted externally today; recovery worked (40-min downtime), but it'd be cheap to add a periodic snapshot of last_ledger + the lightweight metric files outside the session dir so post-wipe restoration doesn't lose progress tracking. Worth doing?

---

Auto-posted by /daily-summary (session 74535976). Today's 13:00 UTC cron likely fired during the build window and missed the REPL-idle slot — this is a manual fire from the operator's /daily-summary invocation.

[leighmcculloch]

• Want me to clean stale pdr-*/debug/ directories that no longer have a running cargo process? It's destructive (~tens of GB recoverable) but harmless to those agents — they'll rebuild on next run.

Go ahead.