henyey mainnet daily — 2026-04-27 (slack-integration test)

[tomerweller]

Validator

• Build: henyey 26.0.0-alpha.1 running on 40bd21fc (uptime 52m; 3 deploys in last ~3.5h, ~70m avg/build)
• State: 🟢 Validating (age=2s, agree=21/missing=0, lag_ms=226359 — note: lag_ms here is the qset's tracking-slot field, known to be noisy/stale — see Add lag_ms field to /info quorum reporting #1945)
• SCP propagation (stellar_scp_timing_first_to_self_externalize_seconds): 79 ms ✓ (matches stellar-core's first-to-self-externalize-lag healthy band 0.3–0.5s; well below)
• SCP slot-cycle (stellar_scp_timing_externalized_seconds): 4.75 s ✓
• Memory: RSS 6.2 GB, jemalloc frag 14%, heap_components_mb stable (4837 → 4845, +8 MB across last 2 reports)
• Disk: data 61% (1004 GB of 1.7 TB), session 8.3 GB, mainnet 117 GB

Deploys (3 today)

Three deploys in the last ~3.5h via the redeploy sentinel (preredeploy log rotations at 12:42:14Z, 13:41:37Z, 14:19:20Z).

• 14d8100a Fix pending envelope release to use LIFO ordering within slots (PR via main)
  • Files touched: crates/herder/
  • Watch sentinel: silent
• 40bd21fc Propagate SCP timing close-time upgrade to Herder and tx queue
  • Files touched: crates/herder/, crates/scp/
  • Watch sentinel: silent (currently running binary)
• (Earlier 12:42Z deploy — likely a metric/dashboard touch; details TBD when tick-history captures kick in.)

Incidents (1 today)

• 🟢 Mainnet hash mismatch at L62307505 (resolved)
  • Window: ~07:11Z (1ba93577 deployed) → ~10:30Z restart on fix (~3h 20m total downtime + investigation)
  • Symptom: pre-close header-hash divergence shutting validator down
  • Root cause: bucket-loader hash-verification gap on merge restart — corrupted/incorrect bucket file was loaded without verifying its content hash matched filename, persisting a bad ledger header. (Initial hypothesis on URGENT: Mainnet hash mismatch at L62307505 on eb3649c2 — pre-close divergence (likely 72d6f901 sponsorship parity) #1982 was 72d6f90 sponsorship parity; downstream investigation pivoted to the bucket loader.)
  • Resolved by: 11199f57 "Verify bucket file integrity during merge restart" + 26b4ed78 "Address review: hot-archive state-1 downgrade and merge-map hash check" (URGENT: Mainnet hash mismatch at L62307505 on eb3649c2 — pre-close divergence (likely 72d6f901 sponsorship parity) #1982 closed)

Issues activity

Filed today (12):

• 🔴 Abort startup on bucket directory corruption instead of falling through to catchup #1988 [urgent] Abort startup on bucket directory corruption instead of falling through to catchup
• 🔴 Add hash assertions to restore_from_has bucket loads #1985 [urgent] Add hash assertions to restore_from_has bucket loads
• 🔴 URGENT: Mainnet hash mismatch at L62307505 on eb3649c2 — pre-close divergence (likely 72d6f901 sponsorship parity) #1982 [urgent] (closed) URGENT: Mainnet hash mismatch at L62307505 on eb3649c
• ⚪ Non-critical: RPC pruning gap grows unbounded post header/tx-history retention fix #1989 (no-label) Non-critical: RPC pruning gap grows unbounded post header/tx-history retention fix
• Expose SCP timing fields in /sorobaninfo HTTP endpoint #1992 [low/follow-up] Expose SCP timing fields in /sorobaninfo HTTP endpoint
• Introduce Duration-typed close-time API to eliminate seconds/ms confusion #1991 [low/follow-up] Introduce Duration-typed close-time API to eliminate seconds/ms confusion
• Fix premature seconds truncation in MAX_SLOTS_TO_REMEMBER calculation #1990 [medium/follow-up] Fix premature seconds truncation in MAX_SLOTS_TO_REMEMBER calculation
• Add arbitrage flood damping (allowTxBroadcast) #1987 [medium/follow-up] Add arbitrage flood damping (allowTxBroadcast)
• Add DEX-lane flood budget limits to broadcastSome #1986 [medium/follow-up] Add DEX-lane flood budget limits to broadcastSome
• Audit persisted-index trust model for hash verification #1984 [high/follow-up] Audit persisted-index trust model for hash verification
• Move outbound preferred-peer eviction to authentication promotion time #1983 [low/follow-up] Move outbound preferred-peer eviction to authentication promotion time
• Apply byte budget to getEvents RPC endpoint #1981 / Apply byte budget to getTransactions RPC endpoint #1980 [high/follow-up] Apply byte budget to getEvents / getTransactions RPC endpoints
• Add end-to-end test for custom bucket directory #1979 [low/follow-up] Add end-to-end test for custom-sponsorship signer removal

Closed today (many — sample):

• URGENT: Mainnet hash mismatch at L62307505 on eb3649c2 — pre-close divergence (likely 72d6f901 sponsorship parity) #1982 URGENT mainnet hash mismatch (resolved by 11199f57)
• [AUDIT-179] Nomination leader election bypasses application-specific node weights #1957 [AUDIT-179] Nomination leader election bypasses application-specific node weights
• [AUDIT-178] Pre-auth signer removal leaves stale unsponsored signer sponsorship descriptors #1955 [AUDIT-178] Pre-auth signer removal leaves stale unsponsored signer sponsorship descriptors
• [AUDIT-177] Unnormalized Local Quorum Set Hash Diverges From Stellar-Core #1953 [AUDIT-177] Unnormalized Local Quorum Set Hash Diverges From Stellar-Core
• [AUDIT-176] Runtime config-upgrade proposals bypass upstream upgrade-needed validation #1952 [AUDIT-176] Runtime config-upgrade proposals bypass upstream upgrade-needed validation
• Wedge: event loop frozen at phase=32 (scp_verified) on 6cc52d22 — possible regression from quorum intersection commits #1949 Wedge: event loop frozen at phase=32 (scp_verified) — closed via re-fetch recovery (e6c2e467, 41ddefde)
• CI flaky: test_core3_restart_rejoin both variants timeout — recurs from closed #1939, #1838 #1942 CI flaky: test_core3_restart_rejoin both variants timeout

Watch items

• pruning_gap: latestLedger - oldestLedger = 2682 (was 1055 at 12:35Z, 1846 at 15:00Z — linear growth, no plateau yet) — tracked by Non-critical: RPC pruning gap grows unbounded post header/tx-history retention fix #1989. Monitor threshold last retuned to 3*retention+500 = 1580; currently exceeds it. Awaiting downstream judgment on whether equilibrium is just higher than predicted (no-fix) or release path is missing a hook (code fix).
• qset.lag_ms: 226359 ms reported in /info — known-noisy tracking-slot artifact (see Add lag_ms field to /info quorum reporting #1945); not a real consensus-lag signal. Not currently actionable.

Tick aggregates (last 24h)

• Cron: 20m cadence (3,23,43 * * * * UTC), session-scoped — ~72 ticks expected, exact fired count to be reported once the new tick-history.jsonl capture (committed in 5e2faa27) starts populating. First daily run — bootstrapping.
• Outcomes (last 24h, as best reconstructable from log rotations + GH issue filings): ~3 ACTION (deploys), ~6 OFFLINE (during the 07:11Z–10:30Z hash-mismatch incident), 0 unhandled WARNINGS, balance OK.
• Self-reflection events: multiple inline-fixes today — most notably 46710d6d (pruning-gap threshold retune), 68c9efb2 (issue body formatting via --body-file), 490521a5 (SCP timing threshold fix).
• Skill commits today (10): 5e2faa27 (daily-summary skill + tick-history capture), 46710d6d, 68c9efb2, 490521a5, dd859523, d5c35bea, b35b0bc0, 7c29e4c3, 2a222b9d, 0bb33c4c (dashboard fix).

Open questions for the operator

• Non-critical: RPC pruning gap grows unbounded post header/tx-history retention fix #1989 pruning-gap drift: gap is now ~2.5x predicted equilibrium. Want me to file a follow-up "investigate maintainer release path" issue (route through plan-do-review), or wait for downstream to land instrumentation per the existing Non-critical: RPC pruning gap grows unbounded post header/tx-history retention fix #1989 suggestion?
• Abort startup on bucket directory corruption instead of falling through to catchup #1988 + Add hash assertions to restore_from_has bucket loads #1985 are urgent / hash-correctness-adjacent — should I prioritize those over the SCP-timing follow-ups (Fix premature seconds truncation in MAX_SLOTS_TO_REMEMBER calculation #1990, Introduce Duration-typed close-time API to eliminate seconds/ms confusion #1991, Expose SCP timing fields in /sorobaninfo HTTP endpoint #1992)?

---

Auto-posted by /daily-summary (commit 5e2faa27). Next post: tomorrow at 13:00 UTC.