henyey mainnet daily — 2026-05-15

[tomerweller]

Validator

• Build: c14afa94 (uptime <5m — just deployed at 13:31Z; pre-deploy was 8b2c578c for 3h03m, prior f486ea12 for 19h13m. 2 deploys in last 24h, ~5.5m avg build)
• State: 🟡 Initializing (post-deploy bucket restore in progress — qset={}, state=Initializing)
• SCP propagation (first_to_self_externalize_seconds): 0 (post-restart, not yet observed)
• SCP slot-cycle (externalized_seconds): 0 (post-restart)
• Memory: RSS 22.1G transient during bucket restore; pre-deploy steady ~7.5G (frag 18–27% over the day)
• Disk: data 26% (420G of 1.7T), session 1.8G, mainnet 25G

Pre-deploy heartbeat: agree=21/missing=0, peer_gap=0, lag_ms=494 (validating on 8b2c578c).

Deploys (2)

• 8b2c578c Revert INV-H2 strict bound to fix catchup panic (URGENT-CI: INV-H2 strict-bound panic (herder.rs:887) regresses #2693 — History Publish failing 2 days running #2708)
  • Files touched: app, bucket, common, db, henyey, herder, history, historywork, ledger, overlay, scp, simulation (89-commit bundle f486ea1..8b2c578)
  • Watch sentinel: fired and recovered (post-deploy catchup-behind triggered 155k overlay backpressure WARNs + recovery-stalled burst delta=240 over 11 min; validator caught up at peer_gap=0; filed metrics: recovery-stalled burst (delta=240) + 155k overlay backpressure on post-deploy catchup-behind #2713)
• c14afa94 Extract ensure_tracking_state helper from duplicated blocks (closes Extract shared tracking-state transition helper from advance_tracking_to/advance_tracking_slot #2715)
  • Files touched: app, herder, history (9-commit bundle 8b2c578..c14afa9)
  • Watch sentinel: silent (just deployed at 13:31Z; bucket restore in progress)

Incidents

• 🟢 INV-H2 strict-bound panic — daily History Publish (Testnet) crashed 2 days in a row

  • Window: 2026-05-14T09:01Z → 2026-05-15T10:08Z (~25h total)
  • Symptom: panic at crates/herder/src/herder.rs:887 — LCL == tracking_slot violated the strict < bound on the catchup rapid-close path
  • Root cause: commit 0114ae06 tightened the assertion to strict < on the premise that complete_externalization always advances tracking first; the catchup path doesn't go through that function
  • Resolved by: 8b2c578c (revert) + structural follow-up 1f82d1b7/c14afa94 (URGENT-CI: INV-H2 strict-bound panic (herder.rs:887) regresses #2693 — History Publish failing 2 days running #2708, Wire catchup rapid-close through tracking_slot advancement #2712, Extract shared tracking-state transition helper from advance_tracking_to/advance_tracking_slot #2715)

• 🟢 Chained CI regressions from herder.rs refactor (1f82d1b7)

  • Window: 2026-05-15T10:42Z → 2026-05-15T12:56Z (~2h14m total)
  • Symptom: two consecutive Test-job failures on origin/main blocking deploy queue
  • Root cause: 1f82d1b7 introduced an idempotency guard that gated the Syncing→Tracking transition (broke row_05_slot_too_new_prefilter); a9cc892d fix introduced a drain-off-load ordering regression (broke test_issue_1773_receive_tx_set_frees_event_loop_during_drain)
  • Resolved by: c14afa94 (extract ensure_tracking_state helper; restored ordering as a side effect) — URGENT-CI: scp_stage_attribution row_05_slot_too_new_prefilter — Range prefilter accepts slot+10000 envelope after 1f82d1b7 #2714 closed, URGENT-CI: test_issue_1773_receive_tx_set_frees_event_loop_during_drain — drain blocks event loop after a9cc892d #2716 noted as resolved

• 🟢 Validator panic Resource::can_add (Classic vs Soroban dim mixup)

  • Window: 2026-05-14T14:08Z → 2026-05-14T14:38Z (~30m)
  • Symptom: assertion failure 2 vs 7 in Resource::can_add (tx-queue admission)
  • Root cause: classic resource (2D) compared against soroban resource (7D)
  • Resolved by: 9c223fb2 (URGENT: validator panic — Resource::can_add size mismatch (2 vs 7) on f486ea12 after 20h #2690)

• 🟢 Quickstart friendbot 5-layer recovery (saga across 16+h)

  • Window: 2026-05-14T18:23Z (first filing URGENT-CI: Quickstart friendbot test timing out for 16h on origin/main since 1e339515 (#2654 overlay work) #2696) → 2026-05-15T09:59Z (Quickstart green on 8b2c578)
  • Symptom: friendbot tx never finalizes; 65+ commits blocked from deploy
  • Root cause: 5 layers — overlay flood gates, missing --in-memory wiring, empty in-memory DB 404, single-node tx not admitted, INV-H2 panic on captive-core sync
  • Resolved by: d30c5652, b41c22f5, 20034d9c, 7353f5cf, dd118a99, 8b2c578c (URGENT-CI: Quickstart friendbot test timing out for 16h on origin/main since 1e339515 (#2654 overlay work) #2696, URGENT-CI: Quickstart friendbot fails — captive-core RPC returns 404 on bot-account-detail (post-#2696 follow-up; blocks 65-commit deploy queue) #2699, URGENT-CI: Quickstart friendbot still fails after #2699 fix — captive-core /getledgerentry returns 404 (deploy gate now blocked ~10h) #2701, URGENT-CI: Quickstart friendbot tx never finalizes on local-mode captive-core (post-#2701 layer-2 issue; deploy queue blocked 13h) #2705)

• 🟢 22 simulation tests timing out (timer-delivery wiring regression)

  • Window: 2026-05-14T14:38Z → 2026-05-15T02:43Z
  • Symptom: manual_close_until hangs at ledger 2, scp_sent=0
  • Root cause: a99bd9a2 "Replace SCP timeout polling with event-driven timer delivery" lost a code path
  • Resolved by: 92baba70, f4b5f89f (URGENT-CI: a99bd9a2 timer-delivery refactor broke 4 simulation tests — nomination stalls at round 1 (deploy still blocked) #2703, HERDER: Align timer delivery and trigger scheduling with stellar-core #2700)

• 🔴 Post-deploy catchup-behind recovery noise (metrics: recovery-stalled burst (delta=240) + 155k overlay backpressure on post-deploy catchup-behind #2713 open)

  • Window: 2026-05-15T10:28Z → 10:39Z (11m post-deploy)
  • Symptom: 155k overlay backpressure WARNs + 4.5k per-slot-cap hits + 5 state transitions before peer_gap=0
  • Root cause: metrics: henyey_post_catchup_hard_reset_total — recurring ArchiveBehindStallWallClock livelock (Related to #2568) #2664's at-tip suppression doesn't cover catchup-behind; archive_confirmed_behind arming logic
  • Resolved by: still open

Issues activity

Filed today (30):

• 🔴 URGENT-CI: test_issue_1773_receive_tx_set_frees_event_loop_during_drain — drain blocks event loop after a9cc892d #2716 URGENT-CI: test_issue_1773 drain blocks event loop after a9cc892
• ⚪ Extract shared tracking-state transition helper from advance_tracking_to/advance_tracking_slot #2715 Extract shared tracking-state transition helper (CLOSED)
• 🔴 URGENT-CI: scp_stage_attribution row_05_slot_too_new_prefilter — Range prefilter accepts slot+10000 envelope after 1f82d1b7 #2714 URGENT-CI: scp_stage_attribution row_05_slot_too_new_prefilter (CLOSED)
• ⚪ metrics: recovery-stalled burst (delta=240) + 155k overlay backpressure on post-deploy catchup-behind #2713 metrics: recovery-stalled burst (delta=240) + 155k overlay backpressure
• ⚪ Wire catchup rapid-close through tracking_slot advancement #2712 Wire catchup rapid-close through tracking_slot advancement (CLOSED)
• ⚪ metrics: stellar_quorum_fail_at — value=1 with missing=0 (catalog threshold likely too tight) #2711 metrics: stellar_quorum_fail_at — value=1 with missing=0 (catalog threshold)
• ⚪ metrics: overlay errors — delta=57 >= 50 (calibration?) #2710 metrics: overlay errors — delta=57 >= 50 (calibration?)
• ⚪ CatchupRange enum refactor: encode replay-only vs bucket-apply at type level #2709 CatchupRange enum refactor
• 🔴 URGENT-CI: INV-H2 strict-bound panic (herder.rs:887) regresses #2693 — History Publish failing 2 days running #2708 URGENT-CI: INV-H2 strict-bound panic regresses URGENT-CI: INV-H2 LCL guard assertion (herder.rs:868) panics on normal post-close timing race — breaks Quickstart #2693 (CLOSED)
• ⚪ Implement in-flight merge dedup (getMergeFuture/putMergeFuture parity) #2707 Implement in-flight merge dedup (CLOSED)
• ⚪ HERDER: Preserve checkpoint slot during SCP purge #2706 HERDER: Preserve checkpoint slot during SCP purge
• 🔴 URGENT-CI: Quickstart friendbot tx never finalizes on local-mode captive-core (post-#2701 layer-2 issue; deploy queue blocked 13h) #2705 URGENT-CI: Quickstart friendbot tx never finalizes (CLOSED)
• ⚪ Wire SCP setup_timer/stop_timer through herder driver to eliminate reconcile pattern #2704 Wire SCP setup_timer/stop_timer (CLOSED)
• 🔴 URGENT-CI: a99bd9a2 timer-delivery refactor broke 4 simulation tests — nomination stalls at round 1 (deploy still blocked) #2703 URGENT-CI: a99bd9a broke 4 simulation tests (CLOSED)
• ⚪ HERDER: Event-driven consensus trigger scheduling (Phase 2) #2702 HERDER: Event-driven consensus trigger scheduling (Phase 2)
• 🔴 URGENT-CI: Quickstart friendbot still fails after #2699 fix — captive-core /getledgerentry returns 404 (deploy gate now blocked ~10h) #2701 URGENT-CI: Quickstart friendbot still fails after URGENT-CI: Quickstart friendbot fails — captive-core RPC returns 404 on bot-account-detail (post-#2696 follow-up; blocks 65-commit deploy queue) #2699 (CLOSED)
• ⚪ HERDER: Align timer delivery and trigger scheduling with stellar-core #2700 HERDER: Align timer delivery and trigger scheduling (CLOSED)
• 🔴 URGENT-CI: Quickstart friendbot fails — captive-core RPC returns 404 on bot-account-detail (post-#2696 follow-up; blocks 65-commit deploy queue) #2699 URGENT-CI: Quickstart friendbot fails (CLOSED)
• ⚪ HERDER: Wire ScpPersistenceManager GC timer into production lifecycle #2698 HERDER: Wire ScpPersistenceManager GC timer
• ⚪ SCP persistence: implement hash-aware tx set cleanup #2697 SCP persistence: hash-aware tx set cleanup
• 🔴 URGENT-CI: Quickstart friendbot test timing out for 16h on origin/main since 1e339515 (#2654 overlay work) #2696 URGENT-CI: Quickstart friendbot test timing out 16h (CLOSED)
• ⚪ Close externalization publish-before-advance race window structurally #2695 Close externalization publish-before-advance race window (CLOSED)
• ⚪ Normalize remaining invalid HAS test fixtures across history crates #2694 Normalize remaining invalid HAS test fixtures (CLOSED)
• 🔴 URGENT-CI: INV-H2 LCL guard assertion (herder.rs:868) panics on normal post-close timing race — breaks Quickstart #2693 URGENT-CI: INV-H2 LCL guard assertion panics (CLOSED)
• 🔴 URGENT-CI: 22 simulation tests timing out — manual_close_until hangs at ledger 2, scp_sent=0 (likely #9862d65f MANUAL_CLOSE wiring regression) #2692 URGENT-CI: 22 simulation tests timing out (CLOSED)
• ⚪ Normalize remaining v2 HAS test fixtures to valid v2 or explicit v1 #2691 Normalize remaining v2 HAS test fixtures (CLOSED)
• 🔴 URGENT: validator panic — Resource::can_add size mismatch (2 vs 7) on f486ea12 after 20h #2690 URGENT: validator panic Resource::can_add 2 vs 7 (CLOSED)
• 🔴 CI: test_history_work_chain failing on every origin/main run since b27a2ef0 (~6h, 9 consecutive failures) #2689 CI: test_history_work_chain failing 9 consecutive runs (CLOSED)
• ⚪ HERDER: Suppress sync-recovery timers in MANUAL_CLOSE+RUN_STANDALONE mode #2688 HERDER: Suppress sync-recovery timers in MANUAL_CLOSE+RUN_STANDALONE
• ⚪ Extract shared helper for archive-behind state clearing #2686 Extract shared helper for archive-behind state clearing (CLOSED)

Closed today (30): mass-closure window — see Incidents for narrative; root causes resolved by 8b2c578c, c14afa94, 9c223fb2, d30c5652, b41c22f5, 20034d9c, 7353f5cf, dd118a99, 92baba70, f4b5f89f.

Still open (3 long-running):

• ⚪ Overlay: 2.4× per-peer SCP amplification persists after #2622 — likely pull-mode flooding gap, not duplicate envelopes #2644 Overlay: 2.4× per-peer SCP amplification persists after Overlay: 2.4× more SCP envelopes received per peer than stellar-core; 2× nominate-timeout rate #2622 — last activity 2026-05-13 (Stream 1 landed as 7ee6ef3f; Stream 2 tracked in Overlay: receive-to-relay latency diagnostic (Stream 2 of #2644) #2648)
• ⚪ Spec adherence audit: suite overview (Protocol 26) #2660 Spec adherence audit: suite overview (Protocol 26) — last activity 2026-05-14
• ⚪ Add prompts/ to deploy-gate allowlist when directory reappears #2662 Add prompts/ to deploy-gate allowlist when directory reappears — last activity 2026-05-13

Watch items

• post_catchup_hr: appeared in 10 ticks (steady-state archive_behind churn; tracked indirectly by metrics: recovery-stalled burst (delta=240) + 155k overlay backpressure on post-deploy catchup-behind #2713 and pre-existing metrics: henyey_post_catchup_hard_reset_total — recurring ArchiveBehindStallWallClock livelock (Related to #2568) #2664)
• deploy_blocked: 8 ticks (mostly during Quickstart 16h saga and the herder.rs regression chain)
• frag: 18% → 27% transient post-restart → 20-22% steady, no #
• rss_mb: 11G pre-deploy → 5.7G post-deploy fresh process → 7.5G after module_cache rebuild, no #
• #2696/#2699 still open: 2 ticks each; both closed by 10:00Z
• recovery_completed peer_gap=0: 1 tick (post-deploy recovery checkpoint)

Tick aggregates (last 24h)

• Cron: n/a (CronList not accessible from this runtime)
• Outcomes: 29 OK / 10 ACTION / 5 WARNING / 2 DEPLOY / 0 OFFLINE (46 ticks total)
• Alarm-firings: 2 ticks with ≥1 firing alarm (3 total alarm-firings: 1 overlay-errors + 1 quorum-fail-at-low + 1 recovery-stalled burst)
• metrics_ratio: 0 ticks reported breach, 7 skipped (post-restart warmup or synced-only gate)
• recovery_stalled: 1 non-zero-delta tick (the 240-burst post-deploy at 10:42Z, metrics: recovery-stalled burst (delta=240) + 155k overlay backpressure on post-deploy catchup-behind #2713)
• Self-reflection events: 9 (mixture of legacy-field "not clean" markers and the explicit filed #2711 Tier-2 entry)
• Skill commits today: 1 (1f07680f Make deploy quarantine gate content-aware to auto-clear after revert)

Open questions

• #2713 (post-deploy recovery-loop noise) is the only open urgent-eligible item from today's incidents. The fix landscape touches crates/app/src/app/catchup_impl.rs _archive_confirmed_behind_until arming logic and is structurally adjacent to closed metrics: henyey_post_catchup_hard_reset_total — recurring ArchiveBehindStallWallClock livelock (Related to #2568) #2664. Worth bumping to Backlog priority if next deploy reproduces the 11-min recovery thrash.