Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable

[tomerweller]

Finding

The mainnet validator's event-loop watchdog has an auto-abort path that calls
std::process::abort(), and on the deployed configuration it is armed at 120 s.
The longest event-loop freeze in this process's 27-day log reached 101.28 s —
84.4% of that threshold, with a margin of 18.7 s ≈ 1.9 watchdog sample
intervals.

That freeze (2026-07-12, tracked as #3723) self-recovered. Had it run two
more watchdog samples, the node would have SIGABRT'd itself — and per #3702 the
validator wedges on every restart, so the recovery path from an abort is an
operator-authorized wipe + cold catchup, not a restart.

Neither #3723 nor #3756 (24 comments) mentions watchdog_abort_secs anywhere.
I checked before filing.

The mechanism

crates/app/src/app/mod.rs, deployed 0ac84d42:

pub(crate) fn should_abort(&self) -> bool {
    self.abort_threshold_secs > 0 && self.stale_secs >= self.abort_threshold_secs
}

// Auto-abort: independent of the tier check so that
// any configured threshold (even < 30s) is respected.
if snap.should_abort() {
    if snap.tier() != WatchdogTier::Error { snap.emit_error(); }
    tracing::error!(…, "WATCHDOG: Auto-aborting after {}s freeze at phase={}", …);
    std::process::abort();
}

The watchdog is a std::thread sampling every 10 s (start_event_loop_watchdog),
so stale_secs advances in 10 s steps: the observed ladder on 07-12 was
23 → 33 → 43 → 53 → 63 → 73 → 83 → 93, and the loop resumed 8.1 s after the
last sample.

It is armed, not disabled

crates/app/src/config.rs:596    #[serde(default = "default_watchdog_abort_secs")]
crates/app/src/config.rs:3806   assert_eq!(AppConfig::mainnet().diagnostics.watchdog_abort_secs, 120);

configs/validator-mainnet-rpc.toml (the deployed config) has no
[diagnostics] section — grep -i 'watchdog\|abort' returns nothing — so the
120 s default applies. watchdog_abort_secs = 0 disables it; the config does not
set it.

The margin, measured

The freeze is measured two ways and they disagree, which matters here. The
watchdog's last sample said stale_secs=93; the loop's own park WARN, emitted on
resume, says 101,280 ms:

2026-07-12T05:04:11.683  WARN  WATCHDOG: Event loop slow (>15s) stale_secs=23 phase=3
2026-07-12T05:04:21.684  ERROR WATCHDOG: Event loop appears frozen! stale_secs=33 phase=3
  … 5 more at 10 s cadence …
2026-07-12T05:05:21.691  ERROR WATCHDOG: Event loop appears frozen! stale_secs=93 phase=3
2026-07-12T05:05:29.113  WARN  Slow event-loop phase … phase_name="broadcast" elapsed_ms=101280
2026-07-12T05:05:29.113  WARN  Consensus stuck - transitioning to out-of-sync
2026-07-12T05:05:29.113  WARN  Lost sync with network - transitioning to syncing state

So the true freeze is 101.28 s, and stale_secs understates it by up to one
sample interval. #3723's title/body say "~90 s"; the park WARN puts it 11 s
higher. Against the abort threshold:

	value
observed max freeze	101.28 s
watchdog_abort_secs (default, unoverridden)	120 s
margin	18.72 s
margin in watchdog sample intervals	1.87

Per-day maximum stale_secs as a fraction of the threshold, whole 27-day process
(PID 1512116, never restarted):

day        max stale_secs  % of 120 s   max park ms
2026-07-12       93            78%        101,280   ← #3723, phase=3 broadcast
2026-07-22       54            45%         60,083   ← phase=28 peer_maintenance
2026-07-20       30            25%         31,865   ← phase=3 broadcast
2026-07-07       25            21%         29,282   ← phase=3 broadcast
2026-07-28       24            20%         30,207   ← phase=29 peer_refresh
(21 other days)  ≤24          ≤20%

I am not claiming a trend toward 120 s. The 101 s point is a single outlier;
the phase=3 maxima across the corpus (25.4 / 27.1 / 27.9 / 29.3 / 30.5 / 31.9 /
101.3 s) do not fit a line. The argument is only that one observed sample
already sits at 84% of a threshold whose trip action is unrecoverable on this
deployment.

Why the abort is net-harmful while #3702 is open

std::process::abort() raises SIGABRT immediately. Consequences on this host:

1. No graceful shutdown. The WatchdogGuard drop, WAL checkpoint, and
   mainnet.lock removal are all skipped. A dirty stop is the documented
   precondition for the VE-12 zero-byte bucket file and the recurring
   knit-to-LCL state-integrity FATAL (FATAL: knit-to-LCL hash mismatch on 6011ec9c — state wipe required #2886/URGENT: knit-to-LCL state-integrity FATAL on b35ea04e during RecoveryEscalation catchup (recurrence of #2886) — validator down ~4.5h, wipe required #2931/URGENT: knit-to-LCL state-integrity FATAL on 384b4f6e during forced near-tip catchup (recurrence of #2931/#2886) — validator down ~5h, wipe required #3282).
2. Restart does not recover. OFFLINE: restore-from-disk near-tip back-fill stall (NOT binary-specific) — wedges on both f850e6da and 0ac84d42 #3702: the validator wedges on every restart
   (restore-from-disk → Tracking-at-stale-slot → near-tip SCP back-fill drops all
   peers). Recovery is an operator-authorized wipe + cold catchup (~25 min+,
   measured peak 27.1 GB anon RSS per Instrument startup/catchup peak RSS #3228).
3. Both freezes that reached ERROR tier self-recovered. 07-12 resumed after
   101.3 s and closed L63438849 at 05:05:50; 07-20 and 07-22 likewise. So on the
   evidence available, the abort would have converted three self-healing events
   into three hard outages and prevented none.

The auto-abort exists to break true deadlocks, and that is a good default in
general. It is specifically wrong here because the thing it escalates to is
broken.

Correction I owe on #3756

In #3756 comment
25
I argued that because the deployed binary predates #3690, the next deploy lands
#3690 + #3765 together and therefore "both arms leave the loop in the same step
… that makes the blast-radius cap complete rather than partial."

That is too strong. Park-phase census over the full 27-day log:

phase=28 peer_maintenance  21,649    ← #3690 offloads
phase=29 peer_refresh       5,521    ← #3765 offloads
phase=3  broadcast            263    ← neither
phase=13 maybe_buffered_catchup  2

broadcast is 1.0% of parks but 8 of the 11 ≥30 s ERROR-tier freezes,
including the 101.3 s one. It is a third arm with a much heavier tail, and
neither offload touches it. "Complete" was wrong; the two offloads cap the two
high-frequency arms, not the worst observed freeze.

Suggested action (operator decision — the monitor does not change validator config)

1. While OFFLINE: restore-from-disk near-tip back-fill stall (NOT binary-specific) — wedges on both f850e6da and 0ac84d42 #3702 is open, set watchdog_abort_secs = 0 (or ≥ 300) in
   configs/validator-mainnet-rpc.toml. Rationale: the abort's recovery path is
   currently a manual wipe, and every observed ERROR-tier freeze self-recovered.
   This is a config change on a running node, so it needs sign-off and — because
   config is read at startup — a restart, which is itself gated on OFFLINE: restore-from-disk near-tip back-fill stall (NOT binary-specific) — wedges on both f850e6da and 0ac84d42 #3702. It may
   therefore have to ride along with the next deploy rather than land alone.
2. Reconsider the 120 s default. A watchdog whose escalation is abort()
   should be paired with a supervisor that can actually restart the process
   cleanly. Until OFFLINE: restore-from-disk near-tip back-fill stall (NOT binary-specific) — wedges on both f850e6da and 0ac84d42 #3702 closes there is no such supervisor here.
3. Emit a startup line recording the effective watchdog_abort_secs. There
   is currently no way to read the armed threshold from a running node — I had to
   derive it from config.rs defaults plus the absence of a config section. A
   single INFO at boot would make this auditable.
4. Consider a warn-only pre-abort tier at, say, 0.75 × threshold, so a
   near-miss is visible as a distinct signal rather than having to be reconstructed
   from a stale_secs ladder after the fact.

Not labelled urgent

No abort has occurred; there is no current validator symptom. Filing per the
label policy as an unlabelled risk/tracking issue.

Related: #3723 (the 101.3 s freeze), #3756 (the park growth that feeds the
stale_secs ladder), #3702 (why the abort is unrecoverable), #3228 (cold-catchup
cost), #3582 / #3598 / #3689 / #3690 (prior arm offloads).

Surfaced by /monitor-tick 2826, 2026-07-28T08:0xZ, deployed 0ac84d42,
PID 1512116, uptime 26.4 d.

[tomerweller]

The broadcast arm is not a random freeze — it fires on a ledger schedule, and P(park) is a monotone function of the 2-adic depth of ledger_seq, from 0.014 % to 100 %. That gives this issue a dated risk calendar: the next ledger deeper than anything this process has seen is L63,963,136, ~2026-08-15

Window 2026-07-30T23:43:08 → 2026-07-31T00:18:55Z, 2,147.0 s, coverage 100 % (965 WARN in 16 shapes, 0 ERROR). Deployed 0ac84d42, PID 1512116, uptime 29 d 02 h 30 m, 0 log rotations — the same unbroken process this issue was filed against.

Re-read the issue body in full before writing. I am not restating what is already here: that broadcast (phase=3) is a third arm neither #3690 nor #3765 offloads, that it is ~1 % of parks but carries 8 of the 11 ERROR-tier freezes including the 101.3 s one, the 120 s watchdog_abort_secs default with no [diagnostics] section, the per-day max stale_secs table, and the four asks. All of that stands unchanged.

What the body does not say is when a broadcast park happens. It turns out to be almost entirely determined in advance.

1. The law

Let v₂(L) be the largest power of two dividing the ledger sequence. Over the whole 29-day process — ledgers 63,285,824 … 63,725,847, 439,947 closes, 279 broadcast parks:

v₂	2^v₂	deepest level spilling	ledgers	parks	P(park)	p50 ms	max ms
0–5	—	—	433,148	62	0.014 %	~7,400	16,073
6	64	2	3,438	0	0 %	—	—
7	128	3	1,719	0	0 %	—	—
8	256	3	860	4	0.47 %	2,275	2,304
9	512	4	429	22	5.13 %	3,101	6,167
10	1,024	4	215	33	15.35 %	3,428	7,880
11	2,048	5	107	51	47.66 %	3,676	10,984
12	4,096	5	54	54	100 %	4,365	11,855
13	8,192	6	27	27	100 %	4,866	16,529
14	16,384	6	14	13	92.9 %	14,704	25,416
15	32,768	7	6	6	100 %	21,881	27,891
16	65,536	7	3	3	100 %	27,148	29,282
17	131,072	8	2	2	100 %	31,865	31,865
18	262,144	8	1	1	100 %	30,645	30,645
19	524,288	9	1	1	100 %	101,280	101,280

Both columns are monotone in v₂: frequency spans four orders of magnitude, p50 duration spans 2.3 s → 101.3 s. Every broadcast park longer than 16.1 s in the entire corpus — all four ≥ 30 s, and the 101.28 s freeze this issue is built around — sits at v₂ ≥ 13. The 101.28 s event was L63,438,848 = 2¹⁹ × 121.

2. Two controls, because a ledger-anchoring artifact would look identical

(a) The other arms sit at chance. Same anchor (last Ledger closed before the park release), same corpus:

arm                 n        seq % 64 == 0
broadcast         279     217  (77.8 %)      <- 50x
peer_maintenance 24903     454  ( 1.82 %)
peer_refresh      6149     113  ( 1.84 %)
chance                           1.5625 %

So the anchoring is unbiased; only broadcast is aligned.

(b) It is not "checkpoint work". The obvious alternative is that something expensive runs every 64 ledgers. Something does — Memory report summary and Adding state size window entry fire at 6,876 ledgers with 6,875/6,875 inter-arrival deltas of exactly 64, i.e. at every checkpoint including all of v₂ = 6 and v₂ = 7. Those 5,157 checkpoints — 75 % of all of them — produced zero broadcast parks. Constant per-checkpoint cost is therefore ruled out: the discriminator is strictly the depth of the boundary, not the boundary itself.

3. The modulus ladder is generated verbatim by the bucket-list spill predicate

0ac84d42:crates/bucket/src/bucket_list.rs:

pub(crate) fn bl_level_half(level: usize) -> u32 { 1u32 << (2 * level + 1) }   // 2, 8, 32, 128, 512, 2048, …
pub(crate) fn bl_level_size(level: usize) -> u32 { 1u32 << (2 * (level + 1)) } // 4, 16, 64, 256, 1024, 4096, …

/// Which simplifies to: ledger is a multiple of levelHalf(level).
pub(crate) fn bl_level_should_spill(ledger_seq: u32, level: usize, num_levels: usize) -> bool {
    …
    ledger_seq % half == 0 || ledger_seq % size == 0
}

size(i) = 2·half(i), so the predicate reduces to ledger % 2^(2i+1) == 0 — the odd powers of two. Hence the deepest level spilling at L is i_max = ⌊(v₂ − 1)/2⌋, which is the "deepest level spilling" column above. Collapsed onto it:

i_max   v2        ledgers   parks   P(park)
  2     6           3,438       0     0 %
  3     7,8         2,579       4     0.16 %
  4     9,10          644      55     8.54 %
  5     11,12         161     105    65.2 %
  6     13,14          41      40    97.6 %
  7     15,16           9       9   100 %
  8     17,18           3       3   100 %
  9     19              1       1   100 %

There is also a clean second-order confirmation. Within each i_max pair, even v₂ additionally satisfies ledger % size(i_max) == 0 — a full-level rollover rather than a half-spill — and the even member is the heavier one at every pair where both are populated: v₂=10 (15.35 %) > 9 (5.13 %); 12 (100 %) > 11 (47.66 %); and in p50, 10 > 9, 12 > 11, 14 > 13, 16 > 15.

I have not traced the causal path from a bucket merge to a stall inside the phase=3 select! arm; the arm itself does channel recv + overlay.broadcast().await. What is established is the trigger, not the transmission.

4. This clock is not #3756's clock — which matters for how you read the tail

Normalising each broadcast park by that day's non-broadcast park p50 (so the #3756 ramp divides out), every v₂ class trends down, and in absolute ms too:

v2=14, one event per ~2.2 d, whole corpus:
07-02  25,416 ms      07-13  14,704 ms      07-24  15,978 ms
07-04  22,469 ms      07-17  13,437 ms      07-26  12,613 ms
07-06  22,437 ms      07-19  13,692 ms      07-28  11,512 ms
07-08  23,364 ms      07-21  14,164 ms      07-30  10,022 ms
07-11  17,521 ms
ratio to day's peer-park p50:  8.88 (07-04)  ->  0.61 (07-30)

So the spill park is a stationary-to-shrinking phenomenon sitting inside a process whose other parks are growing +802…+897 ms/day. Two consequences: the 101.28 s outlier on 07-12 happened when the peer-park p50 was only 10.2 s, so it is not a ramp artifact and #3756's fix will not remove it; and conversely nothing here is on a trajectory toward 120 s by itself.

5. The risk calendar this produces

From L63,725,867 at the measured 5.70 s/ledger (see the note below on how that was measured):

next	ledger	ETA	observed at this v₂
v₂=15	63,733,760	~2026-07-31 13:00Z (+0.5 d)	6/6 parks, p50 21.9 s, max 27.9 s
v₂=16	63,766,528	~2026-08-02 (+2.7 d)	3/3, p50 27.1 s, max 29.3 s
v₂=17	63,832,064	~2026-08-07 (+6.9 d)	2/2, 30.5 s and 31.9 s
v₂=20	63,963,136	~2026-08-15 (+15.5 d)	never seen

63,963,136 = 2²⁰ × 61. Its i_max is 9 — the same deepest level as the 07-12 freeze — but where L63,438,848 was a level-9 half-spill, this is the first level-9 full rollover in this process's life, i.e. the even/odd case that is heavier at every lower level.

What I am not claiming. That it will exceed 101.28 s, or trip the 120 s abort. n = 1 at both v₂=18 and v₂=19; the maxima from v₂=14→18 flatten around 30 s and the 101.28 s point does not sit on that line; and §4's downward trend cuts the other way. Two of the three near-term rows above (07-31, 08-02) are cheap out-of-sample tests of the whole law and I will report them either way.

What it does change is the shape of the decision in this issue's ask 1. "Disarm the abort while #3702 is open" was an argument about an unbounded risk with no timetable. It now has one: the single highest-prior ledger of the next month is L63,963,136 on ~2026-08-15, and it is known 15 days ahead. That is enough lead time either to land ask 1 alongside the next deploy, or to decide deliberately not to.

Ask 3 gets slightly cheaper to justify too: a boot-time INFO recording the effective watchdog_abort_secs would let this calendar be checked against a live node instead of against config.rs defaults.

Reproduce

python3 - <<'PY'
import re, bisect, collections
from datetime import datetime
def pt(s): return datetime.strptime(s[:26],'%Y-%m-%dT%H:%M:%S.%f').timestamp()
def v2(n):
    v=0
    while n%2==0 and v<24: n//=2; v+=1
    return v
bc=[]; closes=[]
for l in open('monitor.log', errors='replace'):
    m=re.match(r'^(\S+Z)\s+WARN.*phase_name="broadcast".*elapsed_ms=(\d+)', l)
    if m: bc.append((pt(m.group(1)), int(m.group(2)))); continue
    m=re.match(r'^(\S+Z)\s+INFO.*Ledger closed ledger_seq=(\d+)', l)
    if m: closes.append((pt(m.group(1)), int(m.group(2))))
ct=[c[0] for c in closes]
hit=collections.Counter(); dur=collections.defaultdict(list)
for t, ms in bc:                                   # anchor = last close BEFORE the stall
    s = closes[bisect.bisect_right(ct, t) - 1][1]
    hit[v2(s)] += 1; dur[v2(s)].append(ms)
pop = collections.Counter(v2(s) for s in range(closes[0][1], closes[-1][1] + 1))
for v in sorted(pop):
    d = sorted(dur.get(v, []))
    print("v2=%2d  ledgers=%7d  parks=%4d  P=%7.3f%%  p50=%7s  max=%7s"
          % (v, pop[v], hit[v], 100.0*hit[v]/pop[v],
             d[len(d)//2] if d else "-", d[-1] if d else "-"))
PY

Run the same loop keyed on phase_name="peer_maintenance" for control (a); grep Memory report summary and check set(deltas) == {64} for control (b).

Node state

Validating at L63,725,934 (last close in the parsed corpus: L63,725,867), protocol 27, build henyey-v27.0.0-alpha.1, deployed 0ac84d42 (/info.commit_hash confirms), PID 1512116, uptime 29 d 02 h 30 m, 0 log rotations, 47 threads (#3796 base 45–47). 32 authenticated (13 in / 19 out) / 1 pending. Quorum phase=EXTERNALIZE fail_at=1 agree=18 disagree=0 missing=3 delayed=0 lag_ms=449, transitive intersection=true node_count=25 critical=[]. Parks n=37, sum 720.89 s == union 720.88 s, overlap 0.00 s (11th consecutive disjointness check, no double-counting), duty 33.6 %, arms 30 peer_maintenance / 6 peer_refresh / 1 broadcast (3,066 ms at L63,725,568 = 2¹³ × 7,779 — v₂=13, i.e. exactly the 100 % row). Max park 26,751 ms; 0 ≥ 30 s. 0 ERROR / FATAL / panic / hash mismatch / database is locked / Lost sync / Consensus stuck / Catching Up / heard_from_quorum=false. 5 forcing catchup (all gap=1, self-recovered). henyey_db_busy_retry_total 28 (flat). jemalloc resident 21.38 GiB, frag 41.28 % (#3759 sawtooth — do not single-sample). Disk free 2.4 GiB / 100 % (#3749). Deploys held on #3702.

(The 5.70 s/ledger used in §5 is measured independently of the log: [rpc] retention_window = 360 is enforced only by the [maintenance] period_secs = 900 job, so getHealth's retained span sawtooths 360 → 360 + closes-per-900 s. Observed peak 518 immediately before Performing database maintenance at 00:21:05.290Z, then 360. 900/158 = 5.70 s.)

Not filed urgent, and not filed as a new issue — §1–§5 are the trigger and timetable for the arm this issue already identifies, so they belong here. The node is Validating with agree=18 disagree=0; urgent on this repo is reserved for validator-blocking symptoms.

— /monitor-tick 2866

[tomerweller]

Boot-diagnostics de-dup with #3795 (ask 4)

Cross-linking at plan time, per the #3795 triage's request to de-duplicate this issue's ask 3 (emit the armed abort threshold at boot) with #3795's ask 4 (emit the effective sample period + armed abort threshold at boot).

Resolution: the boot line belongs here, not in #3795. The #3795 plan converged with the boot-diagnostics item cut from its scope (Critic C, scope lens) precisely so the two issues do not both edit crates/app/src/app/mod.rs:4086 and conflict. #3795 ships only the blind-window fix (loop-side exact stall accounting + a structurally de-phased sampler).

When this issue implements its boot [diagnostics] line, please include the full armed-threshold set — the existing tracing::info!("Event loop watchdog started") at crates/app/src/app/mod.rs:4086 is the natural site (a field addition, zero new log sites):

• sample_period_base_ms / sample_jitter_max_ms — the watchdog sampler delay range. Note: post-Event-loop watchdog is phase-locked to the 60 s park cycle (10 s sampler, 10 | 60): effective threshold is 16.60 s not 15 s, 45 % of ≥15 s stalls are unlogged, and the 540× WARN step on 07-28 is a sawtooth wrap #3795 this is a jittered [7 s, 10 s) range, not a fixed 10 s, so a single sample_period_secs field would be misleading. The constants will be named WATCHDOG_SAMPLE_PERIOD_BASE_MS / WATCHDOG_SAMPLE_JITTER_MAX_MS in mod.rs.
• warn_threshold_secs / error_threshold_secs — hoisted by Event-loop watchdog is phase-locked to the 60 s park cycle (10 s sampler, 10 | 60): effective threshold is 16.60 s not 15 s, 45 % of ≥15 s stalls are unlogged, and the 540× WARN step on 07-28 is a sawtooth wrap #3795 into shared constants (currently inline literals 15 / 30 at mod.rs:4369-4372), so read them from the constants rather than re-typing the numbers.
• abort_threshold_secs — config.diagnostics.watchdog_abort_secs (default 120, crates/app/src/config.rs:600/603), this issue's ask 3.

Ordering note: this is easiest after #3795 lands, since the constants it needs do not exist yet on main. Neither issue blocks the other.

Also relevant to this issue from the #3795 plan's source verification: #3795 adds a compile-time const _: () = assert!(BASE_MS + JITTER_MAX_MS <= 10_000) so the sampler change cannot regress the worst-case abort latency (still ≤10 s beyond the 120 s threshold; mean improves 5 s → 4.25 s). And a correction that matters here: a freeze that does not recover is never blind to should_abort — stale_secs is monotone, so the first sample at-or-above 120 s necessarily lands in [120, 130). That matches this issue's own "must reach ~120–130 s to trip"; the blind window only ever swallows stalls that recover inside it.