Event-loop watchdog is phase-locked to the 60 s park cycle (10 s sampler, 10 | 60): effective threshold is 16.60 s not 15 s, 45 % of ≥15 s stalls are unlogged, and the 540× WARN step on 07-28 is a sawtooth wrap

[tomerweller]

Summary

The event-loop watchdog samples on a rigid 10 s grid while the peer_maintenance /
peer_refresh park cycle is exactly 60 s. 10 divides 60, so the sampler visits every
park at the same phase forever. The consequence is that the coded threshold
(stale_secs >= 15) is not the threshold that operates:

• The effective detection threshold on this node right now is 16.60 s, and it is a
  knife edge — a 16.545 s stall is invisible, a 16.646 s stall is logged. Measured over
  40 consecutive parks: perfectly separable, 0/40 misclassified.
• 45 % of stalls that exceeded the coded 15 s threshold were not logged in the current
  window (18 of 40), because the single sensitive sample lands at a fixed 16.59 s
  (spread 76 ms over 22 detections).
• That effective threshold drifts +0.574 s/day and wraps every ~17 days, sawtoothing
  between ~15 s and ~25 s. Two wraps are in the corpus.
• The 540× step in the WARN count on 07-28 (1/day → 538/day) is one of those wraps.
  It is not a regression: park p50 moved 13.7 → 14.7 s that day, a 7 % change.
• In the 10 days before the wrap (07-18 → 07-27) the threshold sat at 19.7–24.6 s. On
  07-27 the loop spent 87.5 minutes in states where it had been continuously
  un-ticked for ≥15 s. The watchdog logged one line.

Deployed 0ac84d42, PID 1512116, uptime 28 d 23 h 20 m, 0 log rotations —
so every number below comes from one process and one binary.

1. Mechanism (deployed source)

crates/app/src/app/lifecycle.rs:531 — the freshness stamp is bumped once per loop
iteration, at the top, immediately before the select! whose duration becomes
elapsed_ms:

loop {
    select_iteration += 1;
    self.tick_event_loop();          // <-- the only call site in the tree
    self.set_phase(0);
    …
    let phase_dispatch_start = std::time::Instant::now();
    tokio::select! { … }

So stale_secs and the park WARN's elapsed_ms measure the same interval from the
same anchor — one sampled mid-flight by the watchdog thread, one recorded by the loop
on resume. That makes the comparison below exact rather than approximate.

crates/app/src/app/mod.rs:3620-3653 — the sampler:

let (lock, cvar) = &*condvar_thread;
let guard = lock.lock().unwrap_or_else(|e| e.into_inner());
let _ = cvar.wait_timeout(guard, Duration::from_secs(10));   // <-- 10 s, fixed
…
let stale_secs = now_ms.saturating_sub(last_tick) / 1000;

and mod.rs:4075-4079 routes >=30 → ERROR, >=15 → WARN. There is no dedup and no
cooldown: every sample in [15, 30) emits a line.

The grid is rigid, not approximate. On 07-30, 647 of 647 inter-arrival gaps between
WARN lines are exact multiples of 10 s.

2. The effective threshold, measured deterministically

Current window 20:25:50 → 21:06:05Z (2,415.9 s), 40 parks, all 40 above the coded 15 s
threshold (min 15,944 ms). 22 detected, 18 missed:

detected  elapsed_ms:  16,646 … 26,796      n=22
MISSED    elapsed_ms:  15,944 … 16,545      n=18   (45.0 % miss rate)

best single cutoff C = 16,545 ms   ->   misclassified 0/40
max(missed) 16,545  <  min(detected) 16,646     PERFECTLY SEPARABLE

Detection is not probabilistic — it is a step function of stall duration at a cutoff the
code never states. The reason is visible in the offset of the detecting sample into its
park:

n=22   min 16.524 s   p50 16.589 s   max 16.600 s   spread 76 ms

Park onsets are pinned at (t mod 60) = 5.80 / 15.80 s; the grid fires at 22.4 /
32.4. Both differences are 16.6 s. Of the six 10 s slots available per minute, only
one is ever inside the sensitive band.

3. The threshold is a 17-day sawtooth, and 07-28 is a wrap

Effective threshold per day = (grid phase − park onset phase) lifted into [15, 30),
both measured from the log. parks>thr is the resulting prediction; observed is the
actual WARN count:

day	parks	onset (mod 60)	grid (mod 60)	eff. threshold	parks > thr	observed
07-02	5	5.79	46.09	20.30	0	1
07-04	638	5.80	47.39	21.60	0	1
07-06	681	5.80	38.31	22.51	0	2
07-08	659	5.80	59.85	24.06	0	1
— wrap A —
07-12	675	5.80	11.68	15.89	1	1
07-14	1,147	5.80	52.91	17.11	0	1
07-16	1,440	15.80	54.13	18.33	0	1
07-19	1,440	15.80	25.46	19.67	0	1
07-20	1,444	15.80	26.40	20.61	2	3
07-22	1,462	15.80	27.74	21.95	37	41
07-23	1,458	15.79	27.82	22.02	23	24
07-25	1,440	15.80	48.99	23.19	1	2
07-26	1,440	15.79	29.94	24.15	1	1
07-27	1,440	15.79	30.42	24.63	1	1
— wrap B —
07-28	1,458	15.79	21.13	15.33	687	538
07-29	1,442	15.80	21.56	15.77	671	673
07-30	1,289	15.80	22.20	16.40	644	648

The prediction tracks the observation on 19 of 21 days, including the two intermediate
cases (07-22: 37 vs 41; 07-23: 23 vs 24) and the plateau (07-29: 671 vs 673; 07-30: 644
vs 648). Only 07-28 is loose (687 vs 538) — that is the transition day, so a single daily
modal phase does not describe it.

Drift, fit per segment:

segment 1  07-02..07-08   n=7    +0.6118 s/day   R2=0.99217
segment 2  07-12..07-27   n=11   +0.5763 s/day   R2=0.99603
segment 3  07-28..07-30   n=3    +0.5350 s/day   R2=0.98960
mean +0.5743 s/day  ->  wrap period = 10 s / 0.574 = 17.4 days

Observed wraps: between 07-08 (24.06 s) and 07-12 (15.89 s), and between 07-27 (24.63 s)
and 07-28 (15.33 s) — 16 days apart.

4. What was lost during the blind phase

Because elapsed_ms and stale_secs share an anchor (§1), the log records what the
watchdog should have seen. Continuous ≥15 s un-ticked time per day, against what was
reported:

day	eff. threshold	≥15 s un-ticked time	WARN lines
07-24	~22.6 s	3,630 s (60.5 min)	0
07-25	23.19 s	4,220 s (70.3 min)	2
07-26	24.15 s	4,963 s (82.7 min)	1
07-27	24.63 s	5,251 s (87.5 min)	1

On 07-27 the event loop was in a ≥15 s stall for 87.5 minutes of the day and the
watchdog emitted one line. Any consumer keying on this signal — a dashboard, an alarm, a
monitor tick — was dark for ten days and then saw a 540× step change with no underlying
regression.

5. What this does not change

Stating this explicitly so the finding is not over-read:

1. Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767's abort risk stands. For a 120 s freeze, ≥12 samples land inside it, so
   phase-lock cannot defeat should_abort(). The phase-lock only matters when
   stall − 15 s < 10 s, i.e. exactly the current park population.
2. Event loop parked 13-29s once per minute (28% duty-cycle loss): peer_refresh arm awaits inline, 69k-row peers table — recurrence of #3582 #3756's duty-cycle numbers stand. They are computed from loop-side elapsed_ms,
   not from watchdog samples. This window: 40 parks, sum 841.47 s, merged union
   841.47 s (6th consecutive sum == union disjointness check), duty cycle 34.8 %.
3. Event-loop freeze ~90s in broadcast phase (phase=3, fetch_channel_depth_max=1054) — self-recovered, 2026-07-12 #3723's ERROR-tier freezes were caught fairly. Per #3723 comment
   5125155721,
   broadcast (phase=3) parks are aperiodic — they do not land on the pinned
   :05.80 / :15.80 onsets, so they sample uniformly and are not phase-locked. That is
   why the 101.3 s and 60.1 s freezes produced full stale_secs ladders.
4. One refinement to Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767. Its headline "101.3 s = 84 % of 120 s" mixes scales: the
   101.3 s is elapsed_ms, but should_abort() trips on stale_secs, whose last sample
   read 93. On the scale the abort actually uses, that freeze reached 77.5 %, and a
   true freeze must reach ~120–130 s to trip. Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767 already notes the truncation; this
   just puts the number on the right axis.

6. Prediction (falsifiable)

The threshold reaches the top of the sawtooth (~25 s) in 15.0 days → wrap ~2026-08-14.
By then #3756's ramp puts park p50 near 29.8 s, well above the post-wrap 15.3 s. So
detection should stay near 100 % across this wrap rather than collapsing as it did on
07-08. If instead the count drops back to ~1/day around 08-14, the ramp has flattened —
either outcome is informative, and both are checkable from the WARN count alone.

7. Asks

1. Break the 10 s / 60 s commensurability. Any sample period coprime with 60
   (7 s, 11 s, 13 s) makes the sampler walk the park phase instead of standing still, so
   the effective threshold converges on the coded one. One-line change; removes the whole
   class.
2. Better: don't sample at all. The loop already computes the exact stall duration
   (phase_dispatch_start.elapsed()) and already emits a WARN at ≥2,000 ms. Bump a
   counter / histogram there, gated at 15 s, and the measurement becomes exact and
   complete. Sampling an interval the measured party can report directly is the root
   defect.
3. Do not use the WARN-line count as a rate or health signal until (1) or (2) lands.
   Its 07-28 step would read as a regression to anyone who did not measure the grid phase.
4. Emit the effective sample period and armed abort threshold at boot — this composes
   with Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767 ask 3. Neither is readable from a running node today; I had to derive both
   from source plus the absence of a [diagnostics] section.

Asks 1 and 3 are independent of 2 and small.

Reproduce

cd ~/data/$MONITOR_SESSION_ID/logs
# the grid is rigid: every gap an exact multiple of 10 s
grep -a 'WATCHDOG: Event loop slow' monitor.log | grep -a '^2026-07-30' \
  | cut -c1-23 > /tmp/wd.txt
python3 - <<'PY'
import datetime as dt
t=[dt.datetime.fromisoformat(l.strip()) for l in open('/tmp/wd.txt')]
g=[(t[i+1]-t[i]).total_seconds() for i in range(len(t)-1)]
print(sum(1 for x in g if x%10<0.05 or x%10>9.95), "/", len(g), "gaps are 10s multiples")
PY

# only one of six slots ever fires
grep -a 'WATCHDOG: Event loop slow' monitor.log | grep -a '^2026-07-29' \
  | cut -c18-19 | sort | uniq -c | sort -rn

# the perfectly-separable cutoff: pair each park [ts-elapsed, ts] against the grid
grep -a 'Slow event-loop phase' monitor.log | grep -a '^2026-07-30T2[01]'

Node state

Validating at L63,723,711, age 3 s, protocol 27, build henyey-v27.0.0-alpha.1,
deployed 0ac84d42, PID 1512116, uptime 28 d 23 h 20 m, 0 log rotations.
33 authenticated (14 in / 19 out) / 1 pending. Quorum phase=EXTERNALIZE hash=66d2e0 fail_at=1 validated=true agree=18 disagree=0 missing=3 delayed=0 lag_ms=445, transitive
intersection=true node_count=25 critical=[]. Window 20:25:50 → 21:06:05Z, coverage
100.0 %: 851/851 WARN in 16 shapes, 0 ERROR / FATAL / panic / hash mismatch /
database is locked / Lost sync / Consensus stuck in-window. Close band :30-:33
= 170/431 = 39.4 %, :12-:29 = 0/431. Nine ≥30 s close gaps, 9/9 paired,
gap − park mean +4.80 s (5th replication; prior means +5.40 / +5.18 / +5.14 /
+5.16). Disk free 2.4 GiB / 100 % (#3749). Deploys held on #3702.

Not filed urgent. This is an observability defect, not a node defect — the validator
is Validating with agree=18 disagree=0, and the stalls themselves are already tracked
on #3756 / #3723 / #3767 with correct loop-side measurements. urgent on this repo is
reserved for validator-blocking symptoms. The reason to fix it anyway is that the signal
was dark for ten days and its recovery looked exactly like a regression.

Related: #3756 (the parks), #3723 (the aperiodic broadcast arm, unaffected), #3767 (the
120 s abort, unaffected), #3689 / #3765 (arm offloads, neither deployed), #3702 (why the
abort would be unrecoverable).

— /monitor-tick 2861

[tomerweller]

Triage Report

Verdict: ACCEPT

Type: bug
Kind: bug-fix
Severity: high
Crate: crate:app
Size: M

Summary: The event-loop watchdog samples staleness on a re-armed 10 s relative sleep while the parking arm (peer_refresh) fires on an absolute 60 s tokio-interval grid. Because the sampler's phase relative to the park onsets is frozen on hour-to-day timescales, the effective WARN threshold is a fixed value in [15 s, 25 s) rather than the coded 15 s, and stalls shorter than that value are deterministically unreported. Fix by measuring the stall loop-side (where it is already known exactly) and/or de-commensurating the sample period.

Self-modifying: NO. The defect and both candidate fixes live in crates/app/src/app/mod.rs and crates/app/src/app/lifecycle.rs — node code. This is the node-side sampler, distinct from the agent-side log-reading defect in #3793 (.claude/skills/monitor-tick/SKILL.md). Not held.

---

Structural verification against origin/main (df2abf83)

Everything material in the issue's mechanism section is confirmed on main, but every cited line number is stale. The citations are accurate for the deployed binary 0ac84d42; they are not for main:

Issue cites	Actual on origin/main	Verdict
lifecycle.rs:531 — tick_event_loop()	crates/app/src/app/lifecycle.rs:624	code identical, line stale (correct at 0ac84d42)
mod.rs:3620-3653 — sampler	crates/app/src/app/mod.rs:3918-3940; the wait_timeout(guard, Duration::from_secs(10)) is at mod.rs:3922	confirmed, lines stale
mod.rs:4075-4079 — >=30 ERROR / >=15 WARN	crates/app/src/app/mod.rs:4368-4375 (WatchdogSnapshot::tier)	confirmed, lines stale
"no dedup and no cooldown"	mod.rs:4028-4032 — match snap.tier() calls emit_warn() unconditionally every sample	confirmed
loop already emits a >= 2000 ms phase WARN	EVENT_LOOP_PHASE_SLOW_THRESHOLD = CONSENSUS_TICK_SLOW_SUBSTEP_THRESHOLD = Duration::from_secs(2) (mod.rs:4170-4171, 4221-4222); emitted by warn_phase_if_slow (mod.rs:4288-4298) from lifecycle.rs:1564-1568	confirmed
abort at 120 s	default_watchdog_abort_secs() = 120 (crates/app/src/config.rs:603, asserted config.rs:3928/3933/3938); should_abort at mod.rs:4382-4384	confirmed

The park source is also confirmed present on main, not just on the deployed binary: peer_refresh_interval = tokio::time::interval(Duration::from_secs(60)) with MissedTickBehavior::Skip (lifecycle.rs:500-501), and the arm at lifecycle.rs:1369-1374 still .awaits refresh_known_peers() inline on the loop thread. So #3756's 60 s park is unfixed on main and the aliasing premise holds there.

phase_time_us (mod.rs:1136, accumulated lifecycle.rs:1584, dumped lifecycle.rs:2195) is log-only — there is no Prometheus series for event-loop stalls today, so the issue's ask 2 (counter/histogram) is genuinely additive, not a duplicate of existing plumbing.

The aliasing is SYSTEMATIC, not probabilistic

Derived from the constants above, not from the issue's arithmetic:

• Samples occur at t_k = t_0 + k*(10 s + eps). The sleep is a relative wait_timeout re-armed after each iteration's work (mod.rs:3918-3923), so eps = per-iteration work.
• Park onsets are pinned to an absolute grid: tokio::time::interval + MissedTickBehavior::Skip re-anchors to multiples of the 60 s period from start.
• A park [s, s+D] is reported at the WARN tier iff some t_k falls in [s+15, s+D]. Since (t_k - s) mod 10 is a constant phi on hour-to-day timescales, the effective threshold is exactly T_eff = 15 + ((phi - 15) mod 10).

Therefore any park with 15 <= D < T_eff is guaranteed unreported — a deterministic blind spot up to 10 s wide, not reduced detection odds. This matches the issue's headline "perfectly separable, 0/40 misclassified": that separability is the predicted signature, not a coincidence.

Correction to the issue's method: §3 says the phase difference is "lifted into [15, 30)". The lift modulus is the sampler period (10 s), so the range is [15, 25). All 21 of the issue's own daily values (15.33 → 24.63) fall in [15, 25), so this is a mis-stated method line, not a data error — but a planner reproducing the table from the stated recipe would get it wrong.

The drift is explained by the same code, which the issue does not connect. The sampler is relative; the park grid is absolute; the beat frequency is eps. The reported +0.574 s/day implies eps ≈ 0.574 s / 8640 iterations ≈ 66 µs of per-iteration watchdog work. That is a falsifiable prediction of the mechanism and a cheap cross-check for /plan.

Escalation the issue does not make: the ERROR tier has the same blind spot

The issue argues (§5.1) that phase-lock "only matters when stall - 15 s < 10 s". By the identical arithmetic applied to tier()'s >= 30 branch (mod.rs:4369), T_eff,ERROR = 30 + ((phi - 30) mod 10), i.e. a systematic blind spot for freezes in [30 s, 40 s). That tier emits watchdog_freeze = true, which per its own doc comment (mod.rs:4324-4338) is the sentinel external monitoring greps to restart the node. So a 30-40 s freeze can be systematically invisible to auto-restart. This — not the WARN miss rate — is why I am rating this high rather than medium. The issue's §5.1 assessment of the abort is correct in substance: should_abort needs a sample in [s+120, s+D], so the blind spot is D in [120 s, T_eff,abort) with T_eff,abort in [120, 130) — matching the issue's own "a true freeze must reach ~120-130 s to trip" in ask 4. But that is the same defect one decade up, not an exemption from it; §5.1's stated reason ("≥12 samples land inside it") is the wrong argument, since what matters is samples above the threshold, not samples inside the freeze.

Correction to ask 1

Ask 1 says a coprime period makes "the effective threshold converge on the coded one". That is true only in aggregate over many parks. Per event it converts a systematic blind spot into a probabilistic one: P(detect | D) = min(1, (D - 15) / T). For D = 16 s, T = 7 s that is ~14 %. Ask 1 is a real improvement (it removes the guaranteed-miss class and the fake 540x step) but it is not a fix for per-event detection. Ask 2 is the actual fix — the loop already holds the exact interval in phase_dispatch_start.elapsed() (lifecycle.rs:648, 1565). A planner should treat ask 1 as mitigation and ask 2 as the fix, not as alternatives.

Two more constraints on ask 1 that the issue misses:

1. peer_maintenance_interval is 10 s (lifecycle.rs:498) — exactly 1:1 with the sampler, the worst possible commensurability. It is offloaded on main (phase=28 peer_maintenance event-loop stall >30s recurs on 0ac84d42 despite #3582 fix — caused lost-sync #3689, dispatch_peer_maintenance, lifecycle.rs:1353-1366) so it should not park, but the new period must be coprime with 10 as well as 60. 7/11/13 satisfy both; 20 or 30 would not.
2. The 10 s period is load-bearing elsewhere: BACKLOG_STALE_TICKS = 3 (mod.rs:3893-3895) makes the scp-verify stuck-heartbeat window 3 x period = 30 s. Changing the period silently retunes that window; it must be re-expressed in seconds. It also changes worst-case abort latency, which couples to Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767.

Not a duplicate

• monitor-tick: log scan is line-budgeted, not time-budgeted — a burst collapses the window 1,908x and silently zeroes the critical greps (tick 2783 saw 1 of 11 Lost sync, 0 of 16 database-is-locked) #3793 (OPEN, self-modifying) is agent-side: monitor-tick's line-budgeted log scan. It loses lines that were emitted. Event-loop watchdog is phase-locked to the 60 s park cycle (10 s sampler, 10 | 60): effective threshold is 16.60 s not 15 s, 45 % of ≥15 s stalls are unlogged, and the 540× WARN step on 07-28 is a sawtooth wrap #3795 is node-side: lines that were never emitted. Neither fix subsumes the other; both are needed for the signal to be trustworthy end-to-end. Independently confirmed — monitor-tick: log scan is line-budgeted, not time-budgeted — a burst collapses the window 1,908x and silently zeroes the critical greps (tick 2783 saw 1 of 11 Lost sync, 0 of 16 database-is-locked) #3793 lives in .claude/skills/monitor-tick/, Event-loop watchdog is phase-locked to the 60 s park cycle (10 s sampler, 10 | 60): effective threshold is 16.60 s not 15 s, 45 % of ≥15 s stalls are unlogged, and the 540× WARN step on 07-28 is a sawtooth wrap #3795 in crates/app/.
• Event loop parked 13-29s once per minute (28% duty-cycle loss): peer_refresh arm awaits inline, 69k-row peers table — recurrence of #3582 #3756 (OPEN, urgent) is the underlying fault: the inline 60 s peer_refresh await that creates the parks (verified still inline at lifecycle.rs:1369-1374). Event-loop watchdog is phase-locked to the 60 s park cycle (10 s sampler, 10 | 60): effective threshold is 16.60 s not 15 s, 45 % of ≥15 s stalls are unlogged, and the 540× WARN step on 07-28 is a sawtooth wrap #3795 is the sensor that fails to see it. Genuinely separate: fixing Event loop parked 13-29s once per minute (28% duty-cycle loss): peer_refresh arm awaits inline, 69k-row peers table — recurrence of #3582 #3756 removes the current park population but leaves the sampler blind to the next 15-25 s stall class from any other arm; fixing Event-loop watchdog is phase-locked to the 60 s park cycle (10 s sampler, 10 | 60): effective threshold is 16.60 s not 15 s, 45 % of ≥15 s stalls are unlogged, and the 540× WARN step on 07-28 is a sawtooth wrap #3795 removes no parks. Note the ordering interaction — if Event loop parked 13-29s once per minute (28% duty-cycle loss): peer_refresh arm awaits inline, 69k-row peers table — recurrence of #3582 #3756 lands first, Event-loop watchdog is phase-locked to the 60 s park cycle (10 s sampler, 10 | 60): effective threshold is 16.60 s not 15 s, 45 % of ≥15 s stalls are unlogged, and the 540× WARN step on 07-28 is a sawtooth wrap #3795 becomes much harder to validate empirically, so Event-loop watchdog is phase-locked to the 60 s park cycle (10 s sampler, 10 | 60): effective threshold is 16.60 s not 15 s, 45 % of ≥15 s stalls are unlogged, and the 540× WARN step on 07-28 is a sawtooth wrap #3795's regression test must be a deterministic unit test, not a log-based one.
• SQLite busy_timeout expiry silently drops 113 of 139 writes in 26d — including the retention trims that would relieve the contention, and catchup LedgerCloseMeta #3772 is CLOSED (SQLite busy_timeout write drops) and is a plausible contributor to park duration, not a duplicate.
• Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767 (abort armed at 120 s) and Event-loop freeze ~90s in broadcast phase (phase=3, fetch_channel_depth_max=1054) — self-recovered, 2026-07-12 #3723 (aperiodic phase=3 freezes) are adjacent; Event-loop watchdog is phase-locked to the 60 s park cycle (10 s sampler, 10 | 60): effective threshold is 16.60 s not 15 s, 45 % of ≥15 s stalls are unlogged, and the 540× WARN step on 07-28 is a sawtooth wrap #3795's ask 4 (emit sample period + armed abort threshold at boot) overlaps Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767 ask 3 and should be de-duplicated at plan time into a single boot-time [diagnostics] line.
• Searches run: watchdog (30 results), phase-lock OR aliasing OR sampler (20 results). No duplicate.

Test obligation: regression test that fails on current main and passes on the fix. This is unit-testable without a live node — good, because the production validator is down (unrelated knit-to-LCL hash mismatch), so no live measurement or baseline is available.

Failing-mode seed (for the regression test):

• Observable: with the sampler period at 10 s and park onsets quantised to a 10 s-divisible grid, a park train of duration D = 16.0 s (comfortably above the coded 15 s WARN gate) produces zero WARN-tier detections across an arbitrarily long run — WatchdogSnapshot::tier() never returns Warn — because every sample lands at the same sub-park offset.
• Reproducer (deterministic, no I/O, no timers): drive WatchdogSnapshot::tier() (mod.rs:4368) from a pure simulation — sample times t_k = phi + k*T, parks [s_j, s_j + D] with s_j = 5.8 + 60j, stale = t_k - s_j where t_k is in-park. Assert detections == 0 for T = 10, D = 16.0 (fails on main once the period is a readable constant) and > 0 for T = 7. A second, stronger test: assert gcd(WATCHDOG_SAMPLE_PERIOD_SECS, P) == 1 for every event-loop timer period P in {60, 30, 10, 5} — gcd(10, 60) = 10 fails today; gcd(7, 60) = 1 passes. Note the period is currently an inline literal at mod.rs:3922, so step 1 of the fix is naming it.
• Where: crates/app/src/app/mod.rs (start_event_loop_watchdog sampler + WatchdogSnapshot), with the exact-accounting variant in crates/app/src/app/lifecycle.rs (phase_dispatch_start, lifecycle.rs:648/1565).

NOT verified — load-bearing premises for /plan to check

1. Every empirical number in the issue is unverified by me. The 16.60 s effective threshold, the 45 % / 18-of-40 miss rate, the 0/40 separability, the 21-day threshold table, the three drift regressions, the two wraps, and the "87.5 min un-ticked on 07-27" all come from a log corpus I do not have; the validator is down. The mechanism is verified from source and the shape of the findings is predicted by it, but no figure is independently confirmed.
2. The deployed-vs-main gap. The cited lines resolve correctly at 0ac84d42, so the measured binary is not main. I confirmed the watchdog and peer_refresh code is substantively identical between the two, but I did not diff the whole tree; other main-only changes could alter park duration.
3. §1's "exact rather than approximate" anchor claim is an approximation. tick_event_loop() is at lifecycle.rs:624; phase_dispatch_start is at lifecycle.rs:648; between them sits self.deferred_catchup.lock().await (lifecycle.rs:628-633), a tokio Mutex acquire that can yield, plus a debug!. So stale_secs >= elapsed_ms / 1000, not ==. Negligible in steady state, but it is the premise for the whole §4 "what was lost" reconstruction, so the planner should not treat that table as exact.
4. Attribution of the parks to phase=29 specifically rests on Event loop parked 13-29s once per minute (28% duty-cycle loss): peer_refresh arm awaits inline, 69k-row peers table — recurrence of #3582 #3756's evidence, not on anything I checked. I verified only that the inline 60 s peer_refresh await is a sufficient mechanism for 60 s-pinned onsets.
5. §5.3's claim that phase=3 broadcast parks are aperiodic (via a Event-loop freeze ~90s in broadcast phase (phase=3, fetch_channel_depth_max=1054) — self-recovered, 2026-07-12 #3723 comment) is unverified; if any other periodic arm also parks, T_eff differs per arm and the single-phi model in §3 under-describes the system.

Label adjustments: added bug, high, crate:app (issue was filed with no labels).

[tomerweller]

📝 Plan Draft (Round 1)

Summary: Make the ≥15 s event-loop stall signal complete and exact by adding loop-side inter-tick stall reporting (which sees every stall the sampler is structurally blind to, at both the WARN and ERROR tiers) and by structurally de-phasing the watchdog sampler with a jittered 7–10 s delay so its phase can no longer lock to any event-loop timer grid.

Line-number note for /do: all three citations in the issue body are stale — they were correct at the deployed commit 0ac84d42, not at origin/main (df2abf83). Every line below is re-derived from origin/main.

---

Verification against origin/main (df2abf83)

claim	origin/main location	status
sampler is a relative, re-armed wait_timeout(guard, Duration::from_secs(10))	crates/app/src/app/mod.rs:3922 (inline literal, unnamed)	✅
tier routing >=30 ERROR / >=15 WARN	mod.rs:4368-4376 (WatchdogSnapshot::tier)	✅
no dedup / no cooldown	mod.rs:3956-4032 — match snap.tier() emits every sample	✅
freshness stamp once per loop iteration	lifecycle.rs:624 self.tick_event_loop()	✅
park source on an absolute grid	lifecycle.rs:500-501 interval(Duration::from_secs(60)) + MissedTickBehavior::Skip; arm still .awaits refresh_known_peers() inline at lifecycle.rs:1369-1374	✅ (#3756 unfixed on main)
peer_maintenance_interval = 10 s (1:1 with sampler)	lifecycle.rs:498-499	✅
BACKLOG_STALE_TICKS = 3 ⇒ 30 s window derived from the period	mod.rs:3893-3895, used mod.rs:4067	✅
loop already holds the exact interval	lifecycle.rs:648 phase_dispatch_start, consumed lifecycle.rs:1566-1569 (warn_phase_if_slow, ≥2 s)	✅
abort at 120 s	config.rs:600/603, mod.rs:4382-4384	✅

Mechanism confirmed: samples are at t_{k+1} = t_k + 10 s + eps_k (relative, re-armed after the body's work); park onsets are pinned modulo 60 s (tokio::time::interval computes deadlines as start + n*period, and Skip re-anchors to the next multiple — both variants are absolute-grid). So (t_k − s) mod 10 is quasi-frozen (drifting only by the accumulated eps), and a park [s, s+D] is reported iff some t_k ∈ [s+15, s+D]. Effective threshold T_eff = 15 + ((φ − 15) mod 10); every park with 15 ≤ D < T_eff is a guaranteed miss for as long as φ holds (days). Same arithmetic one decade up gives T_eff,ERROR = 30 + ((φ − 30) mod 10) ∈ [30, 40).

Two corrections I am making to the issue / triage before designing:

1. The blind window only swallows stalls that RECOVER inside it. For a stall that does not recover, stale_secs grows monotonically and the first sample at or above the threshold necessarily lands in [threshold, threshold + 10 s) — so an unrecovered freeze is always detected, merely ≤10 s late. Therefore the triage's escalation ("a 30–40 s freeze can be systematically invisible to auto-restart") is true only for freezes that self-recover in that band, and the 120 s abort is likewise late-not-blind (consistent with the issue's own "must reach ~120–130 s"). The escalation still has real teeth in a sharper form: a node that repeatedly parks 30–40 s and recovers each time emits no watchdog_freeze at all, so the only automated signal stays silent for the entire degradation and then fires abruptly once parks cross ~40 s — losing exactly the early warning that precedes an abort. That recovering class is what the loop-side path captures by construction, which is why this plan covers the ERROR tier and not just the WARN tier.
2. §3's "lifted into [15, 30)" is a mis-stated method line — the lift modulus is the sampler period (10 s), so the range is [15, 25), which is where all 21 of the issue's own daily values sit. Noted so /do does not reproduce the wrong recipe.

Evidentiary state (honest): every empirical figure in the issue (16.60 s effective threshold, 45 % / 18-of-40, 0/40 separability, the 21-day table, the three drift fits, the two wraps, "87.5 min un-ticked on 07-27") is unverified — the validator is down, there is no log corpus, and no number was independently confirmed. Also unverified: the deployed-vs-main full-tree diff, the phase=29 attribution (rests on #3756), and §5.3's claim that broadcast (phase=3) parks are aperiodic. The design below depends on none of these numbers. It depends only on the source-verified structural facts in the table above: relative sampler + absolute timer grid + threshold-crossing detection. If §5.3 is wrong and other arms also park periodically, that strengthens the case for jitter (a per-arm φ makes a single magic period even less defensible) and changes nothing in the fix.

---

Files to modify:

• crates/app/src/app/mod.rs — (a) name the sampler period and replace the fixed 10 s wait with a jittered [7 s, 10 s) delay from a pure generator; (b) hoist the tier thresholds into shared constants used by both the sampler and the new loop-side path; (c) add the pure event_loop_stall_tier() + report_event_loop_stall() loop-side reporter; (d) re-express BACKLOG_STALE_TICKS as a duration-based window; (e) enrich the existing watchdog-start info! with the armed thresholds; (f) all new unit tests.
• crates/app/src/app/lifecycle.rs — measure the inter-tick gap at the top of the event loop (same anchor as tick_event_loop()) and hand it to report_event_loop_stall() before set_phase(0) clobbers the previous arm's attribution.
• crates/app/src/metrics.rs — one histogram + one counter in metric_catalog!.
• crates/app/src/app/log_throttle.rs — doc-comment correction (once per 10s tick → once per 7–10 s sample).

---

Approach:

1. Loop-side exact accounting — the actual fix (issue ask 2). In run_event_loop (lifecycle.rs:621-625), hold a loop-local last_tick_at: Instant initialised immediately before loop {. At the top of each iteration, before self.tick_event_loop() and before self.set_phase(0), take one Instant::now(), compute gap = now - last_tick_at, restamp last_tick_at = now, and call self.report_event_loop_stall(gap). Reading the phase there is deliberate: set_phase clears event_loop_phase_sub (mod.rs:3813-3822), so at loop top both event_loop_phase and event_loop_phase_sub still hold the arm that just ran — exact attribution, no extra plumbing. report_event_loop_stall early-returns when event_loop_stall_tier(gap) == None, so steady-state cost is one clock read plus one compare (the loop already does two Instant reads per iteration at lifecycle.rs:648/1583).

Measuring the gap at loop top — rather than reusing phase_dispatch_start — is load-bearing: phase_dispatch_start (lifecycle.rs:648) starts after the self.deferred_catchup.lock().await preamble (lifecycle.rs:628-633), a tokio Mutex acquire that can itself block. The loop-top anchor is byte-identical to the watchdog's last_event_loop_tick_ms anchor, so the loop-side number and the sampler's stale_secs are the same quantity measured two ways — which also retires the triage's objection #3 (stale_secs >= elapsed_ms/1000, not ==).

This is complete by construction: every stall that ends produces exactly one report at its exact duration, independent of any sampler phase. It covers both tiers — [15, 30) at WARN severity and ≥30 at ERROR severity — which is what closes the watchdog_freeze-adjacent gap for the recovering-freeze class identified above.

2. The loop-side ERROR line must not trip the auto-restart grep. .claude/skills/monitor-tick/SKILL.md:801 restarts the node on watchdog_freeze"?\s*[=:]\s*true or the literal message WATCHDOG: Event loop appears frozen. The loop-side event fires only after the loop has demonstrably resumed, so a restart is the wrong action; it therefore uses a distinct message ("Event loop stall (loop-side exact accounting)") and carries neither restart pattern, with stall_recovered = true, stall_ms, stall_secs, phase, phase_name, phase_sub. This preserves the reserved-field invariant documented at mod.rs:4324-4338 and its mechanical guard (watchdog_freeze_field_tests) untouched — no doc or test relaxation. Consequence, stated plainly: monitoring's restart response to a recovered 30–40 s stall stays non-monotonic in duration (a 45 s recovered stall still gets watchdog_freeze from the sampler; a 35 s one now gets an ERROR line without it). That non-monotonicity lives in monitor-tick's restart rule, not in the node, so I am filing it as a follow-up rather than papering over it node-side.

3. Structural de-phasing of the sampler (supersedes issue ask 1). Ask 1 (a coprime period) is mitigation only: per event it converts a guaranteed miss into P(detect|D) = min(1, (D−15)/T) ≈ 14 % at D = 16 s, T = 7 s. It is also unmaintainable as a guarantee: flood_tx_period() / flood_demand_period() (lifecycle.rs:492-493) are config-derived, so no fixed integer can be proven coprime with the whole timer set, and any future arm can silently re-create the aliasing. So the sampler is de-phased structurally: Duration::from_millis(WATCHDOG_SAMPLE_PERIOD_BASE_MS + (rng % WATCHDOG_SAMPLE_JITTER_MAX_MS)), i.e. a fresh delay in [7 s, 10 s) each iteration from a pure xorshift64* generator (watchdog_next_sample_delay(&mut u64)) seeded once at thread start from SystemTime nanos ⊕ pid. (t_k − s) mod P then becomes a random walk against every grid, present and future, and different nodes de-phase differently. Belt-and-braces, the base is still chosen coprime with {60, 30, 10, 5, 1} (7 s) and a test pins that, with a comment saying why gcd alone is insufficient. Invariants: BASE_MS + JITTER_MAX_MS <= 10_000 enforced by const _: () = assert!(...) so worst-case abort latency does not regress (should_abort, mod.rs:4382, and hence #3767, is unaffected); mean period drops 10 s → 8.5 s, so detection latency for unrecovered freezes strictly improves. Shutdown latency is unchanged — WatchdogGuard::drop wakes the wait via notify_one (mod.rs:4108-4119).

4. Every duration derived from the period, re-expressed absolutely (triage constraint 3). I grepped the tree for period-derived quantities; there are exactly two, plus one doc comment: (a) BACKLOG_STALE_TICKS = 3 (mod.rs:3893-3895/4067) silently means "30 s" — replaced by BACKLOG_STALE_WINDOW = Duration::from_secs(30) with the sampler tracking hb_stuck_since: Option<Instant> and a pure predicate backlog_heartbeat_is_stuck(stuck_for, backlog), so the window is 30 s under any period; repeat-emission behaviour (fires on every sample once past the window) is preserved. (b) The doc comments at mod.rs:3873 ("Every 10 seconds"), mod.rs:3887, mod.rs:4304 ("10-second poll interval") are corrected. (c) log_throttle.rs:86-89 asserts the watchdog SCP-verifier errors are "naturally rate-limited to at most once per 10s tick" — now once per ≥7 s; the comment is corrected (a 43 % rate increase on an already-throttle-exempt error line; no code change). mod.rs:246/7486 (OUT_OF_SYNC_RECOVERY_TIMER_SECS) and log_throttle.rs's RECOVERY_THROTTLE_SECS are the recovery tick, not the watchdog — unaffected.

5. Metric, so the signal is countable rather than grep-counted (issue ask 3). Two metric_catalog! entries: histogram EVENT_LOOP_STALL_SECONDS = "henyey_event_loop_stall_seconds" and counter EVENT_LOOP_STALL_ERROR_TOTAL = "henyey_event_loop_stall_error_total", both recorded only from the loop-side path (gated at ≥15 s so the hot path stays free; the HELP text states the truncation explicitly). Deliberately unlabeled: labelled counters carry exactly one label key and are excluded from ALL_PREREGISTERED_COUNTER_NAMES (metrics.rs:270-272), whereas an unlabeled counters { } entry auto-populates every required list and histograms are never pre-registered (metrics.rs:243), so the catalog tests stay green with no extra bookkeeping.

6. Ask 4, at zero new log sites. The existing tracing::info!("Event loop watchdog started") (mod.rs:4086) gains sample_period_base_ms, sample_jitter_max_ms, warn_threshold_secs, error_threshold_secs, abort_threshold_secs. This is a field addition to a line that already exists, and composes with #3767's ask 3 for a fuller boot [diagnostics] line rather than duplicating it.

Explicitly out of scope: #3756 (the inline peer_refresh await that creates the parks — a separate in-flight urgent; this issue is the blind sensor and removes no parks), #3793 (agent-side log-budget line loss), #3767 (the 120 s abort itself), and making the sample period configurable.

---

Kind: bug-fix

Test plan — all deterministic: pure functions, fixed PRNG seed, no sleeps, no timers, no I/O, no live park required. This is deliberate per the triage's ordering note: if #3756 lands first the empirical signal disappears, so a log-based or timing-based test would silently stop testing anything.

Regression tests (all in crates/app/src/app/mod.rs's #[cfg(test)] mod tests, alongside the existing watchdog_tier_routing at mod.rs:8174):

1. test_watchdog_sampler_has_no_systematic_blind_window — pure geometry simulation. For each event-loop timer period P ∈ {1, 5, 10, 30, 60} s and each stall duration D ∈ {15.5, 16.0, 20.0, 24.9} s, generate 200 park onsets s_j = φ₀ + P·j and a sample train from the production watchdog_next_sample_delay() with a fixed seed; count parks with at least one sample in [s_j + 15, s_j + D]. Asserts detections > 0 for every (P, D) pair. Why it fails on origin/main: the helper takes the delay sequence as a closure; substituting the constant Duration::from_secs(10) that origin/main uses at mod.rs:3922 yields exactly 0 detections out of 200 for (P = 60, D = 16.0) at the deployed geometry (onset residue 5.8 s, sample residue 2.4 s). On main the test additionally cannot compile, because the period is an unnamed inline literal — naming it is step 1 of the fix. The failing number is pinned in-tree by the sibling below so the regression is asserted, not just asserted-about.
2. test_fixed_period_sampler_is_blind_by_construction — the same helper driven by a constant 10 s delay; asserts detections == 0 for (P = 60, D = 16.0) and detections > 0 for D = 26.0. Documents the exact defect and stays green after the fix (it asserts about a constant sequence, not about production).
3. test_loop_side_reporting_is_complete_for_both_tiers — this is the ERROR-tier regression test. Park train of 200 parks on a 60 s grid, run twice with D = 16.0 s and D = 35.0 s: asserts (a) the constant-10 s sampler detects 0/200 in both cases at the blind geometry, and (b) the loop-side path reports 200/200, at WatchdogTier::Warn for D = 16 and WatchdogTier::Error for D = 35. Fails on origin/main: there is no loop-side tier path at all, so the D = 35 s recovering-freeze class produces no ERROR-tier record whatsoever — the guaranteed-blind ERROR window is asserted directly.
4. test_loop_side_stall_event_does_not_trip_monitor_restart_grep — renders the loop-side ERROR-tier event through the tracing_subscriber::fmt harness already present in watchdog_freeze_field_tests (mod.rs, BufWriter) and asserts the output contains neither watchdog_freeze nor WATCHDOG: Event loop appears frozen (monitor-tick's two restart patterns, .claude/skills/monitor-tick/SKILL.md:801), while WatchdogSnapshot::emit_error() rendered the same way still contains both. Fails on origin/main: the loop-side event does not exist; and it guards the future regression of someone reusing emit_error() there and converting recovered stalls into automated node restarts.
5. test_watchdog_sample_base_period_coprime_with_event_loop_timer_periods — asserts gcd(WATCHDOG_SAMPLE_PERIOD_BASE_SECS, P) == 1 for P ∈ {1, 5, 10, 30, 60}. Fails on origin/main: gcd(10, 60) = 10 and gcd(10, 10) = 10. Carries the comment that gcd is necessary but not sufficient (config-derived flood_tx_period/flood_demand_period), which is the recorded reason jitter exists.

New coverage:

6. test_event_loop_stall_tier_boundaries — pure event_loop_stall_tier(): 14.999 s → None, 15.0 s → Warn, 29.999 s → Warn, 30.0 s → Error; plus a cross-check that it agrees with WatchdogSnapshot::tier() for every integral stale_secs in 0..=60, so the two paths cannot drift off the shared constants.
7. test_watchdog_sample_delay_range_and_determinism — every delay from watchdog_next_sample_delay lies in [7 s, 10 s); the same seed reproduces the same sequence; two different seeds diverge; and BASE_MS + JITTER_MAX_MS <= 10_000 (the abort-latency invariant, also a const _: () = assert!).
8. test_backlog_stale_window_is_duration_based — backlog_heartbeat_is_stuck(29 s, backlog=5) == false, (30 s, 5) == true, (999 s, 0) == false; the 30 s window is asserted in seconds, independent of the sample period. This is the guard for the BACKLOG_STALE_TICKS retune trap.
9. test_watchdog_start_log_reports_armed_thresholds — renders the enriched watchdog-start info! and asserts the five threshold fields are present (ask 4).

Existing tests preserved (cargo test -p henyey-app, run pre- and post-change):

• crates/app/src/app/mod.rs: watchdog_tier_routing (mod.rs:8174), watchdog_should_abort_routing (mod.rs:8189), test_watchdog_diagnostic_hint_content (mod.rs:7975), test_watchdog_guard_drop_signals_shutdown (mod.rs:8012), test_watchdog_thread_exits_on_guard_drop (mod.rs:8043 — the mock still uses its own 5 s wait, and the real guard path wakes via notify_one, so jitter cannot slow shutdown), watchdog_freeze_field_tests::{test_watchdog_freeze_emits_field_text_format, ..._json_format} (must stay byte-identical in behaviour — the restart sentinel is unchanged).
• crates/app/src/metrics.rs: the catalog coverage tests over ALL_COUNTER_NAMES / ALL_HISTOGRAM_NAMES / ALL_PREREGISTERED_COUNTER_NAMES (metrics.rs:1789/1830/1857/1905).
• crates/app/tests/watchdog_shutdown.rs::test_watchdog_clean_shutdown_no_abort — the subprocess no-spurious-SIGABRT harness; sensitive to sampler timing, so it is the key end-to-end guard on the period change.
• crates/app/src/config.rs: test_diagnostics_config_default_watchdog_abort_secs, test_testnet_config_watchdog_abort_secs (config.rs:3927/3932) — abort threshold untouched.

---

Parity considerations:

n/a — does not change the observable surface. docs/PARITY.md scopes parity to ledger/bucket hashes, transaction result & meta XDR, SCP/overlay wire bytes, history archive format, HTTP/RPC/CLI contracts and crypto outputs, and classifies metrics and logging as freely divergeable. This change touches only a henyey-internal liveness sampler (std::thread watchdog), henyey-internal log lines, and two henyey_-prefixed Prometheus series. stellar-core has no counterpart to this watchdog at all, so there is no upstream function to match. No XDR, no hashing, no wire bytes, no consensus sequencing, no HTTP contract change. The one interop-ish contract in play is internal: monitor-tick's restart grep, which the plan preserves exactly and pins with test 4.

---

Risks:

• Log volume up at the WARN tier. Every park ≥15 s now emits one loop-side line where previously only the phase-locked subset did (on the issue's own unverified numbers, ~1440/day vs ~670/day). Same order as the existing ≥2 s warn_phase_if_slow stream (lifecycle.rs:1566), and bounded by the park count, so no new class of unboundedness — but the monitor's line budget (monitor-tick: log scan is line-budgeted, not time-budgeted — a burst collapses the window 1,908x and silently zeroes the critical greps (tick 2783 saw 1 of 11 Lost sync, 0 of 16 database-is-locked) #3793) will see more lines, which is the trade for a signal that is actually complete.
• New ERROR-severity lines. Recovered stalls ≥30 s now produce ERROR lines. That is intended (they deserve attention), but monitor-tick's ERROR/FATAL counters will move off zero on a node that today reads clean, and a reader could misread it as a new fault. Mitigated by the distinct message text and stall_recovered = true; flagged in the follow-up below.
• Restart-response non-monotonicity persists (§2 above): a recovered 35 s stall gets an ERROR line but no restart, a recovered 45 s one still gets watchdog_freeze from the sampler. Deliberate — the fix belongs in monitor-tick's rule. Follow-up to file.
• Jitter makes sampler-side behaviour statistical. Any future test asserting "a sample lands within 10 s" must use the worst case (< 10 s, guaranteed by the const assert) rather than the mean. Existing timing-sensitive test: crates/app/tests/watchdog_shutdown.rs (60 s budget, 25 s observation) — comfortable.
• Hot-path cost. One extra Instant::now() per loop iteration (~20–25 ns) on a loop that already takes two. maxtps is sensitive to loop-top work, so /do should keep report_event_loop_stall #[inline] with the tier check first and nothing else before the early return.
• Unverifiable empirically right now. The validator is down, so none of this can be confirmed against a live node before merge; that is precisely why every test is a pure deterministic unit test. Post-deploy, the falsifiable check is that the loop-side count ≥ the sampler count on every day, with equality only when all parks exceed T_eff.
• Interaction with Event loop parked 13-29s once per minute (28% duty-cycle loss): peer_refresh arm awaits inline, 69k-row peers table — recurrence of #3582 #3756. If Event loop parked 13-29s once per minute (28% duty-cycle loss): peer_refresh arm awaits inline, 69k-row peers table — recurrence of #3582 #3756 lands first the park population vanishes and the new signal reads ~0 — which is the correct outcome and does not weaken any test here (none depends on observing a real park).

[tomerweller]

🔍 Critic A (correctness) — Round 1

Verdict: REVISE-MINOR

Key concerns:

• Test 3's sampler assertion is wrong as written. For D = 35 s the WARN window [s+15, s+35] is 20 s wide — wider than the 10 s sampler period — so the constant-10 s sampler always fires at least one WARN. The claim "the constant-10 s sampler detects 0/200 in both cases" is only true at the ERROR tier for D = 35. A doer implementing the assertion literally writes a test that fails for the wrong reason. The load-bearing part (0/200 ERROR-tier detections at D = 35, which is the watchdog_freeze blind window) is correct and does fail on main; it just needs the tier qualifier.
• Test 1's detections > 0 at D = 15.5 s is thin and seed-dependent. With delays in [7, 10) s the per-park detection probability is ≈ 0.5/8.5 ≈ 5.9 %, and successive samples are correlated (a random walk, not i.i.d. draws), so 200 parks is not a comfortable margin. Deterministic under a fixed seed, but the doer must actually run it and pin a seed; better to raise the park count for the near-threshold D values and assert a floor on the aggregate rate rather than a bare > 0 per pair.
• BACKLOG_STALE_TICKS → duration conversion drops one reset edge if transcribed naively. On main (mod.rs:4065-4079) the counter resets when backlog == 0 even if the heartbeat has not advanced. The plan says "cleared when hb advances or backlog == 0" in prose but the pure predicate signature backlog_heartbeat_is_stuck(stuck_for, backlog) puts the reset outside the predicate — the doer must keep the backlog == 0 clear of hb_stuck_since in the sampler body, not just in the predicate.
• The loop-top gap inherits an undocumented liveness assumption. gap at loop top includes the select! wait, so a genuinely idle loop would look like a stall. It cannot happen only because consensus_interval (lifecycle.rs:489, 1 s, unconditional) guarantees a wakeup. That is the same assumption the existing sampler makes on last_event_loop_tick_ms, so there is no new false-positive class — but it should be written down next to the new code, because anyone who later makes the 1 s tick conditional silently converts idleness into a 15 s "stall" report.

Full review

What the plan gets right

• The anchor choice is correct and non-obvious. Using a loop-top Instant rather than phase_dispatch_start (lifecycle.rs:648) is the right call: phase_dispatch_start begins after self.deferred_catchup.lock().await (lifecycle.rs:628-633), so a stall inside that tokio-Mutex acquire would be invisible to a phase_dispatch_start-based measure. The loop-top anchor is identical to tick_event_loop()'s (lifecycle.rs:624), which makes the loop-side number and stale_secs the same quantity — and, as the plan notes, retires triage objection Add side-by-side comparison CI workflow (henyey vs stellar-core) #3.
• The phase-attribution ordering is correct. set_phase clears event_loop_phase_sub (mod.rs:3813-3822), so reading event_loop_phase/event_loop_phase_sub at loop top before set_phase(0) yields the arm that just ran, with its sub-phase intact. Reading it after would attribute every stall to phase=0.
• Completeness argument is sound. Every stall that ends produces exactly one loop-side report at its exact duration, with no dependence on sampler phase. That is a strictly stronger property than any period choice can give, and it is the only construction that closes a guaranteed blind window rather than converting it to a probability.
• The correction to the triage's escalation is right, and I re-derived it. For an unrecovered freeze stale_secs is monotone, so the first sample at-or-above threshold lands in [threshold, threshold + max_period) — detection is late, never absent. So should_abort (mod.rs:4382-4384) and Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767 are unaffected, and the genuinely-lost class is recovering stalls in [30, T_eff,ERROR). The plan's reframing (loss of early warning ahead of an abort, not loss of the abort) is the accurate statement and it is exactly the class the loop-side path captures.
• Not reusing emit_error() loop-side is correct, and for a reason the plan states precisely: monitor-tick greps both watchdog_freeze and the literal WATCHDOG: Event loop appears frozen (.claude/skills/monitor-tick/SKILL.md:801), so reusing it would convert every recovered 30 s park into an automated node restart. Test 4 pins both patterns. Good.
• Const-asserted BASE_MS + JITTER_MAX_MS <= 10_000 is the right way to keep worst-case abort latency from regressing; it makes the Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767 coupling a compile-time invariant instead of a comment.

Concerns in detail

1. Test 3 tier qualifier (must fix). Detection at tier X requires a sample in [s + X, s + D], a window of width D − X. For X = 15, D = 35 that width is 20 s > 10 s period ⇒ at least two samples land inside ⇒ WARN always fires. For X = 30, D = 35 the width is 5 s ⇒ blind whenever the frozen residue φ mod 10 ∈ (5, 10). So the assertion must read: D = 35 s ⇒ sampler WARN detections = 200/200, sampler ERROR detections = 0/200, loop-side ERROR reports = 200/200. That is a sharper and more damning test than the draft's version — it shows the sampler seeing the stall and still never emitting the restart sentinel.

2. Test 1 robustness. Suggest: D ∈ {15.5, 16.0, 20.0, 24.9} × P ∈ {1, 5, 10, 30, 60} with 2,000 parks for D < 17 and a documented seed; assert per-pair detections > 0 and aggregate rate within a factor of 3 of the analytic min(1, (D−15)/8.5). The rate bound is what would catch a jitter generator that is accidentally degenerate (e.g. a xorshift seeded to 0, which never advances — a classic xorshift footgun the doer must guard: seed must be forced non-zero).

3. Xorshift seed-zero. xorshift64* with state 0 emits 0 forever ⇒ constant 7 s period ⇒ the aliasing silently returns with a worse period (gcd(7,·) is fine, but a constant period re-freezes φ). watchdog_next_sample_delay must force a non-zero seed (seed | 1), and test 7 should include an explicit seed = 0 case asserting the sequence still varies. This is the one way the fix could regress to the bug in production while all other tests stay green.

4. Double-reporting semantics. A 60 s park yields ~5 sampler WARN + ~3 sampler ERROR during, plus one loop-side line after. That is intended, but it means the two log streams must not be summed. The new histogram counts each park exactly once, which is the durable answer to the issue's ask 3 — worth one sentence in the HELP text or the code comment so a dashboard author does not add the streams.

5. Shutdown tail. The final stall before loop exit is never reported loop-side (the loop exits instead of reaching the next top). Correct behaviour — an unrecovered stall is the sampler's job — but the guard drops at lifecycle.rs:1593, so between the last loop top and the drop there is no coverage from either path. Non-blocking; not worth code.

6. Nothing in the plan narrows visibility or removes code, so the dead-code/-Dwarnings build-verify gate does not apply. All new items have non-test callers in the same crate (mod.rs → lifecycle.rs), so pub(crate) is safe by construction; the pure helpers used only by tests (e.g. the simulation harness) should live inside the #[cfg(test)] module rather than being added as pub(crate) production items — worth a word to /do, since a pub(crate) helper called only from #[cfg(test)] trips dead_code under this repo's global -Dwarnings.

Verdict rationale

Every REVISE-MINOR item above is a wording/parameter fix inside the existing design; none of them changes the approach, and the two load-bearing claims (loop-side completeness at both tiers; unrecovered freezes are late-not-blind) survived independent re-derivation. Not REVISE-MAJOR.

[tomerweller]

🔍 Critic B (parity) — Round 1

Verdict: APPROVE

Key concerns:

• This is a non-parity path. Nothing in the plan touches the MUST-match surface enumerated in docs/PARITY.md:23-38 (ledger header/hashes, bucket list, tx results, tx/ledger meta XDR, event XDR, SCP wire, overlay wire, history archive format, HTTP API contract, JSON-RPC, CLI, crypto outputs). The plan changes a std::thread liveness sampler, two log-line shapes, and two Prometheus series.
• Metrics and logging are explicitly carved out as divergeable, verbatim: docs/PARITY.md:55 "Metrics — metric names, labels, existence, and values" and :56 "Logging — log formats, levels, and content". So the new henyey_event_loop_stall_seconds histogram / henyey_event_loop_stall_error_total counter and the new/changed WARN/ERROR lines are non-parity by definition, not by argument.
• stellar-core has no counterpart to match — confirmed, not assumed. I searched the pinned submodule (stellar-core/ @ 7696c069, v27.0.0): the only watchdog hits in src/ are a Rust build artifact (src/rust/soroban/p25/target/.../libperf_event-*.rmeta), i.e. zero source matches. The nearest upstream construct is ApplicationImpl::scheduleSelfCheck / mSelfCheckTimer (src/main/ApplicationImpl.h:145,255,269), which is a bucket/archive integrity self-check driven off VirtualTimer, not an event-loop-liveness sampler, and it has no shared observable output with this watchdog. There is therefore no upstream function whose sequencing or output this plan could diverge from.
• The one contract genuinely at risk is internal, and the plan pins it. monitor-tick's auto-restart grep (.claude/skills/monitor-tick/SKILL.md:801) matches watchdog_freeze"?\s*[=:]\s*true or the literal WATCHDOG: Event loop appears frozen. The plan deliberately keeps the new loop-side ERROR event clear of both patterns and pins that with a rendered-output test, and leaves WatchdogSnapshot::emit_error() (mod.rs:4402-4416) and WATCHDOG_FREEZE_FIELD (mod.rs:4338) byte-identical. Under docs/PARITY.md this is not a parity surface at all (it is an internal monitoring consumer), but it is the right thing to guard and it is guarded.

Full review

Surface-by-surface check against docs/PARITY.md:23-38

MUST-match surface	touched?	why not
Ledger header & hashes	no	no ledger/bucket code in the file list
BucketList ordering / bucket bytes	no	—
Transaction results XDR	no	—
Transaction/ledger meta XDR	no	the file list includes crates/app/, but only app/mod.rs watchdog region, app/lifecycle.rs loop-top, metrics.rs, log_throttle.rs doc comment — no meta emission path
Classic & Soroban event XDR	no	—
SCP wire (signed bytes, nomination/balloting sequencing, quorum decisions)	no	the SCP-verifier health block (mod.rs:4056-4081) is touched, but only its log-emission gating (BACKLOG_STALE_TICKS → duration window). It reads v.state(), v.heartbeat(), v.backlog() and emits tracing::error!; it does not verify, drop, delay, reorder, or sign any envelope. Message-processing sequencing is untouched.
Overlay P2P wire	no	peer_refresh_interval / peer_maintenance_interval cadence and the refresh_known_peers await are read for analysis only; the plan explicitly leaves #3756 out of scope, so no timer that affects peer-visible behaviour changes
History archive format	no	—
HTTP API contract (/info, /tx, /getledgerentry, upgrades, surveys)	no	none of these handlers change. /metrics output does change, but metric names/existence/values are explicitly divergeable (:55) and /metrics is not in the MUST table's HTTP row
JSON-RPC contract	no	—
CLI contract	no	no new flags/subcommands; the plan explicitly declines to make the period configurable
Crypto outputs	no	—

Timing-change parity analysis

The only behavioural change with any timing content is the sampler period (fixed 10 s → jittered [7, 10) s) and the abort-latency invariant. Two things to confirm from a parity standpoint:

1. Does the sampler influence anything peer- or consumer-observable? No. The watchdog thread only reads atomics (last_event_loop_tick_ms, event_loop_phase, event_loop_phase_sub, fetch_channel_depth*) and the verifier handle, and its only effects are tracing emission and std::process::abort(). It never writes ledger state, never touches a socket, and never participates in SCP.
2. Does the abort path shift? should_abort (mod.rs:4382-4384) compares stale_secs against config.diagnostics.watchdog_abort_secs (default 120, config.rs:600/603) — the threshold is untouched, and the plan's const _: () = assert!(BASE_MS + JITTER_MAX_MS <= 10_000) keeps the worst-case sampling latency at its present ≤10 s. So the observable "when does a wedged node die" envelope is unchanged (mean latency actually improves 5 s → 4.25 s). A process abort is not itself a parity surface — stellar-core has no equivalent auto-abort, and an aborted node is indistinguishable from a crashed one to peers.

Adjacent-code parity note for /do

crates/app/src/app/lifecycle.rs is parity-adjacent as a whole file (it drives consensus triggering and close sequencing), so the usual caution applies — but the plan's edit there is confined to the loop preamble: one Instant::now(), one Duration compare, one conditional log/metric, before tick_event_loop()/set_phase(0). No select! arm is added, removed, reordered, or gated; no .await is introduced (the reporter must stay synchronous — if /do were to make it async and .await inside it, that would add a yield point at loop top and I would want to re-review). Worth stating as the single parity-relevant implementation constraint: report_event_loop_stall must be a synchronous, non-awaiting, non-blocking call.

Conclusion

Internal-only change on the divergeable surface. No stellar-core function to mirror, no wire bytes, no XDR, no hashes, no API schema. APPROVE.

[tomerweller]

🔍 Critic C (scope) — Round 1

Verdict: REVISE-MINOR

Key concerns:

• Not too narrow — it addresses the root cause, not the symptom. The issue's own ask 1 (a coprime period) is the band-aid, and the plan explicitly refuses to ship it alone, correctly identifying the root defect as "sampling an interval the measured party can report directly". Had the plan stopped at a magic period I would have called it a band-aid; it does not.
• Borderline on breadth, and one piece should be cut. Item 6 (the boot-time armed-thresholds line, issue ask 4, plus test 9) is the only element that is not needed to close the blind window. Triage explicitly asked for it to be de-duplicated with Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767 ask 3 at plan time. Doing it here as a field-addition is cheap, but it is a second, separable deliverable and it is the natural home for Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767's version. Recommend cutting items 6 and test 9 from this PR and adding the fields as part of Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767's [diagnostics] boot line instead — one boot line, one owner, no merge conflict between two PRs editing mod.rs:4086.
• The remaining five items are one atomic concern and are not separable. (1) loop-side exact reporting, (2) jittered sampler, (3) BACKLOG_STALE_TICKS → duration window, (4) the metric, (5) the period doc-comment corrections. (3) is forced by (2) — shipping (2) without (3) silently retunes the scp-verify stuck window from 30 s to 21–30 s, which is a new bug. (4) is 2 metric_catalog! lines plus 2 record calls and is what retires the issue's ask 3 (stop using a grep count as a rate); splitting it would leave the issue's stated fix half-done. (5) is comment-only and must ship with (2) or the tree documents a period that no longer exists.
• Size is within one reviewable PR but near the top of the band. My estimate: ~130 lines of production change (mod.rs ~95, lifecycle.rs ~20, metrics.rs ~8, log_throttle.rs ~3) plus ~230-260 lines of pure unit tests ≈ ~370-390 net, under the ~500 line ceiling. Cutting item 6 takes it to ~360. Nine tests is a lot of test surface, but they are all pure arithmetic with no fixtures, so review cost is low.

Full review

Root-cause test

The reported symptom is "45 % of ≥15 s stalls unlogged and a fake 540× step". The proximate cause is gcd(10, 60) = 10. The root cause is one level deeper: a sampled detector for a quantity the event loop measures exactly, where the sampler's phase relative to the measured events is quasi-static. A plan that only changed 10 → 7 would fix the reported statistic while leaving (a) per-event detection probabilistic at ~14 % for D = 16 s, (b) the same class re-creatable by any future timer, including the config-derived flood_tx_period()/flood_demand_period() (lifecycle.rs:492-493) which no fixed integer can be proven coprime against, and (c) the ERROR-tier watchdog_freeze window still blind for recovering freezes. The plan attacks all three. Correct depth.

Breadth test — deliverable by deliverable

#	deliverable	separable?	verdict
1	loop-side exact inter-tick reporting (both tiers)	this is the fix	keep
2	jittered sampler [7,10) s	needed for the ongoing-stall path and for abort latency de-correlation; loop-side alone cannot report a stall that has not ended	keep
3	BACKLOG_STALE_TICKS → BACKLOG_STALE_WINDOW	forced by (2) — omitting it is a regression	keep
4	1 histogram + 1 counter	issue ask 2 verbatim ("bump a counter/histogram there"); retires ask 3	keep
5	period doc-comment corrections (mod.rs:3873/3887/4304, log_throttle.rs:86-89)	comment-only, must accompany (2)	keep
6	boot-time armed-thresholds line (ask 4) + test 9	yes — triage asked to de-dup with #3767 ask 3	cut → #3767

On what is correctly excluded

The plan's out-of-scope list is the right one and I want it preserved verbatim in the converged plan:

• Event loop parked 13-29s once per minute (28% duty-cycle loss): peer_refresh arm awaits inline, 69k-row peers table — recurrence of #3582 #3756 (the inline peer_refresh await at lifecycle.rs:1369-1374 that creates the parks) — folding it in would be the classic scope error here: it is the fault, this is the sensor, and it is separately in flight as urgent. Fixing it removes no blindness and fixing this removes no parks.
• monitor-tick: log scan is line-budgeted, not time-budgeted — a burst collapses the window 1,908x and silently zeroes the critical greps (tick 2783 saw 1 of 11 Lost sync, 0 of 16 database-is-locked) #3793 (agent-side monitor-tick log-budget line loss) — different tree (.claude/skills/), different failure (lines emitted but dropped vs never emitted).
• Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767 (the 120 s abort) — the plan makes the abort-latency invariant a compile-time assert and otherwise leaves it alone, which is the right seam.
• Re-architecting the event loop's timer set — the plan deliberately does not touch the ~15 select! arms or their periods, which is what kept this under 500 lines. Explicitly resisting "make all the timers coprime" is the correct call: that would be a multi-hundred-line change across a parity-adjacent file for a weaker guarantee than jitter gives.

One scope risk to watch during /do

The crates/app/src/app/lifecycle.rs edit is the only touch in a parity-adjacent hot file. If implementing it starts requiring new App fields, an async reporter, or moving tick_event_loop(), that is scope creep into the event loop and should stop — the design as written needs exactly one loop-local Instant and one synchronous call, no App struct change. If /do finds it needs more, bounce rather than expand.

Verdict rationale

One atomic concern, right depth, ~370-390 net lines, single reviewable PR. The only actionable item is cutting deliverable 6 to #3767 per triage's own de-dup instruction — a REVISE-MINOR, not a blocker, and no sub-issues need filing (the receiving issue already exists).

[tomerweller]

✅ Converged Plan

Summary: Make the ≥15 s event-loop stall signal complete and exact by adding loop-side inter-tick stall reporting — which by construction sees every stall the sampler is structurally blind to, at both the WARN and the ERROR/watchdog_freeze-adjacent tier — and by structurally de-phasing the watchdog sampler with a jittered [7 s, 10 s) delay so its phase can no longer lock to any event-loop timer grid, present or future.

Line-number warning for /do: all three citations in the issue body are stale. They were correct at the deployed commit 0ac84d42, not at origin/main. Every line reference below is re-derived from origin/main (df2abf83) and verified.

---

Verified structural facts (origin/main = df2abf83)

claim	location on origin/main	status
sampler is a relative, re-armed wait_timeout(guard, Duration::from_secs(10)) — an unnamed inline literal	crates/app/src/app/mod.rs:3922	✅
tier routing >=30 ERROR / >=15 WARN (inline literals)	mod.rs:4368-4376 (WatchdogSnapshot::tier)	✅
no dedup / no cooldown — every sample emits	mod.rs:3956-4032	✅
freshness stamp once per loop iteration	lifecycle.rs:624 self.tick_event_loop()	✅
park onsets pinned to an absolute grid	lifecycle.rs:500-501 interval(Duration::from_secs(60)) + MissedTickBehavior::Skip; arm still .awaits refresh_known_peers() inline at lifecycle.rs:1369-1374 (#3756 unfixed on main)	✅
peer_maintenance_interval = 10 s — 1:1 with the sampler	lifecycle.rs:498-499	✅
BACKLOG_STALE_TICKS = 3 ⇒ a 30 s window derived from the period	mod.rs:3893-3895, used mod.rs:4067	✅
loop already holds the exact interval	lifecycle.rs:648 phase_dispatch_start, consumed at lifecycle.rs:1566-1569 (warn_phase_if_slow, ≥2 s)	✅
abort armed at 120 s	config.rs:600/603; should_abort mod.rs:4382-4384	✅
monitor-tick's auto-restart grep	.claude/skills/monitor-tick/SKILL.md:801 — matches watchdog_freeze"?\s*[=:]\s*true OR the literal WATCHDOG: Event loop appears frozen	✅

Mechanism (re-derived, not taken on trust). Samples occur at t_{k+1} = t_k + 10 s + eps_k (relative, re-armed after the body's work). Park onsets are pinned modulo 60 s — tokio::time::interval computes deadlines as start + n·period, and Skip re-anchors to the next multiple, so both variants are absolute-grid. Hence (t_k − s) mod 10 is quasi-frozen (drifting only by accumulated eps), and a park [s, s+D] is reported at tier X iff some t_k ∈ [s+X, s+D]. Effective threshold T_eff = X + ((φ − X) mod 10), so every park with X ≤ D < T_eff is a guaranteed miss for as long as φ holds (days). At X = 30 that gives a guaranteed blind window inside [30, 40) — the tier that emits watchdog_freeze.

Two corrections carried into the design

1. The blind window only swallows stalls that RECOVER inside it. For a stall that does not recover, stale_secs is monotone, so the first sample at-or-above the threshold necessarily lands in [X, X + max_period): an unrecovered freeze is always detected, merely ≤10 s late. So the triage's escalation ("a 30–40 s freeze can be systematically invisible to auto-restart") holds only for freezes that self-recover in that band, and should_abort/Watchdog auto-abort is armed at 120s on the mainnet validator; the 07-12 freeze already reached 101.3s (84%) and #3702 makes the abort unrecoverable #3767 is late-not-blind (consistent with the issue's own "must reach ~120–130 s"). The escalation keeps its teeth in a sharper form: a node that repeatedly parks 30–40 s and recovers emits no watchdog_freeze at all, so the only automated signal stays silent through the entire degradation and then fires abruptly once parks cross ~40 s — losing exactly the early warning that precedes an abort. That recovering class is what the loop-side path captures by construction, which is why this plan covers the ERROR tier and not only the WARN tier.
2. The issue's §3 "lifted into [15, 30)" is a mis-stated method line — the lift modulus is the sampler period (10 s), so the range is [15, 25), where all 21 of the issue's own daily values sit. Do not reproduce the stated recipe verbatim.

Evidentiary state — read this before trusting any number

Every empirical figure in the issue is UNVERIFIED: the 16.60 s effective threshold, 45 % / 18-of-40 miss rate, 0/40 separability, the 21-day threshold table, the three drift regressions, the two wraps, and "87.5 min un-ticked on 07-27". The production validator is down (unrelated knit-to-LCL hash mismatch), so there is no log corpus and no live measurement. Also unverified: the deployed-vs-main full-tree diff, the phase=29 park attribution (rests on #3756), and §5.3's claim that phase=3 broadcast parks are aperiodic.

The design below depends on none of those numbers. It depends only on the source-verified structural facts in the table above: relative sampler + absolute timer grid + threshold-crossing detection. If §5.3 is wrong and other arms also park periodically, that strengthens the case for jitter (a per-arm φ makes a single magic period even less defensible) and changes nothing in the fix. Every test is a pure deterministic unit test for the same reason.

---

Files to modify:

• crates/app/src/app/mod.rs — (a) name the sampler period and replace the fixed 10 s wait with a jittered [7 s, 10 s) delay from a pure generator; (b) hoist the tier thresholds into shared constants used by both the sampler and the loop-side path; (c) add pure event_loop_stall_tier() + synchronous report_event_loop_stall(); (d) re-express BACKLOG_STALE_TICKS as a duration-based window; (e) correct the period doc comments; (f) all new unit tests.
• crates/app/src/app/lifecycle.rs — measure the inter-tick gap at the top of the event loop (same anchor as tick_event_loop()) and report it before set_phase(0) clobbers the previous arm's attribution. One loop-local Instant, one synchronous call — no new App field, no new .await.
• crates/app/src/metrics.rs — one histogram + one counter in metric_catalog!.
• crates/app/src/app/log_throttle.rs — doc-comment correction at log_throttle.rs:86-89 (once per 10s tick → once per ≥7 s sample).

---

Approach:

1. Loop-side exact accounting — the actual fix (issue ask 2). In run_event_loop, hold a loop-local last_tick_at: Instant initialised immediately before loop { (lifecycle.rs:622). At the top of each iteration, before self.tick_event_loop() and before self.set_phase(0), take one Instant::now(), compute gap = now − last_tick_at, restamp last_tick_at = now, and call self.report_event_loop_stall(gap). Reading the phase there is deliberate: set_phase clears event_loop_phase_sub (mod.rs:3813-3822), so at loop top both event_loop_phase and event_loop_phase_sub still hold the arm that just ran — exact attribution, no extra plumbing. report_event_loop_stall early-returns when event_loop_stall_tier(gap) == None, so steady-state cost is one clock read plus one compare (the loop already takes two Instant reads per iteration at lifecycle.rs:648/1583).

Measuring at loop top rather than reusing phase_dispatch_start is load-bearing: phase_dispatch_start (lifecycle.rs:648) starts after the self.deferred_catchup.lock().await preamble (lifecycle.rs:628-633), a tokio Mutex acquire that can itself block — a stall there would be invisible to a phase_dispatch_start-based measure. The loop-top anchor is identical to last_event_loop_tick_ms's, so the loop-side number and the sampler's stale_secs are the same quantity measured two ways (this also retires triage objection #3: the anchors differ by the preamble, and the loop-top measure removes the discrepancy).

This is complete by construction: every stall that ends produces exactly one report at its exact duration, independent of sampler phase, at WARN severity for [15, 30) and ERROR severity for ≥30. Document alongside the new code the liveness assumption it inherits: gap includes the select! wait, so an idle loop would look like a stall — it cannot, only because consensus_interval (lifecycle.rs:489, 1 s, unconditional) guarantees a wakeup. This is the same assumption the existing sampler already makes, so there is no new false-positive class, but anyone who later makes that 1 s tick conditional would silently turn idleness into a 15 s "stall".

2. The loop-side ERROR line must not trip the auto-restart grep. It fires only after the loop has demonstrably resumed, so a restart is the wrong action. It therefore uses a distinct message — "Event loop stall (loop-side exact accounting)" — carrying neither of monitor-tick's two restart patterns (SKILL.md:801), with fields stall_recovered = true, stall_ms, stall_secs, phase, phase_name, phase_sub. WatchdogSnapshot::emit_error() (mod.rs:4402-4416), WATCHDOG_FREEZE_FIELD (mod.rs:4338) and the reserved-field invariant documented at mod.rs:4324-4338 are left byte-identical — no doc or test relaxation. Consequence, stated plainly: monitoring's restart response to a recovered stall stays non-monotonic in duration (a recovered 45 s stall still gets watchdog_freeze from the sampler; a recovered 35 s one now gets an ERROR line without it). That belongs in monitor-tick's rule, not the node — filed as #3815.

3. Structural de-phasing of the sampler — supersedes issue ask 1. Ask 1 (a coprime period) is mitigation only: per event it converts a guaranteed miss into P(detect|D) = min(1, (D−15)/T) ≈ 14 % at D = 16 s, T = 7 s. It is also unmaintainable as a guarantee — flood_tx_period() / flood_demand_period() (lifecycle.rs:492-493) are config-derived, so no fixed integer can be proven coprime with the whole timer set, and any future arm can silently re-create the aliasing. So de-phase structurally: each iteration waits Duration::from_millis(WATCHDOG_SAMPLE_PERIOD_BASE_MS + (rng % WATCHDOG_SAMPLE_JITTER_MAX_MS)), i.e. a fresh delay in [7 s, 10 s) from a pure xorshift64* generator watchdog_next_sample_delay(&mut u64), seeded once at thread start from SystemTime nanos ⊕ pid. (t_k − s) mod P becomes a random walk against every grid, and different nodes de-phase differently.

Seed-zero is a hard requirement, not a nicety (Critic A): xorshift64* with state 0 emits 0 forever ⇒ a constant 7 s period ⇒ φ re-freezes and the bug returns in production while every other test stays green. The generator MUST force a non-zero state (seed | 1), and test 7 pins a seed = 0 case.

Belt-and-braces the base is still coprime with {60, 30, 10, 5, 1} (7 s), pinned by test 5 with a comment recording that gcd is necessary but not sufficient. Invariants: const _: () = assert!(WATCHDOG_SAMPLE_PERIOD_BASE_MS + WATCHDOG_SAMPLE_JITTER_MAX_MS <= 10_000) so worst-case abort latency cannot regress (should_abort, mod.rs:4382, and hence #3767, unaffected); mean period drops 10 s → 8.5 s so detection latency for unrecovered freezes strictly improves. Shutdown latency is unchanged — WatchdogGuard::drop wakes the wait via notify_one (mod.rs:4108-4119).

4. Every duration derived from the period, re-expressed absolutely (triage constraint 3). The tree was grepped for period-derived quantities; there are exactly two, plus doc comments:

• (a) BACKLOG_STALE_TICKS = 3 (mod.rs:3893-3895, used mod.rs:4067) silently means "30 s". Replace with BACKLOG_STALE_WINDOW = Duration::from_secs(30); the sampler tracks hb_stuck_since: Option<Instant> and calls a pure predicate backlog_heartbeat_is_stuck(stuck_for, backlog). Preserve both existing reset edges exactly (Critic A): on main the counter resets when the heartbeat advances or when backlog == 0, even if the heartbeat has not advanced — the backlog == 0 clear of hb_stuck_since must live in the sampler body, since the predicate signature alone does not express it. Repeat-emission behaviour (fires on every sample once past the window) is preserved.
• (b) Doc comments at mod.rs:3873 ("Every 10 seconds"), mod.rs:3887, mod.rs:4304 ("10-second poll interval"), and log_throttle.rs:86-89 (which asserts the watchdog SCP-verifier errors are "naturally rate-limited to at most once per 10s tick" — now once per ≥7 s; a 43 % rate increase on an already-throttle-exempt error line, comment-only, no code change).
• Not affected: mod.rs:246 / mod.rs:7486 (OUT_OF_SYNC_RECOVERY_TIMER_SECS) and log_throttle.rs's RECOVERY_THROTTLE_SECS are the recovery tick, a different 10 s.

5. Metric, so the signal is countable rather than grep-counted (issue ask 3). Two metric_catalog! entries: histogram EVENT_LOOP_STALL_SECONDS = "henyey_event_loop_stall_seconds" and counter EVENT_LOOP_STALL_ERROR_TOTAL = "henyey_event_loop_stall_error_total", recorded only from the loop-side path (gated at ≥15 s so the hot path stays free). Both deliberately unlabeled: labelled counters carry exactly one label key and are excluded from ALL_PREREGISTERED_COUNTER_NAMES (metrics.rs:270-272), whereas an unlabeled counters { } entry auto-populates every required list and histograms are never pre-registered (metrics.rs:243) — so the catalog tests stay green with no extra bookkeeping. The HELP text must state two things: the ≥15 s truncation (it is a partial distribution by design), and that this series counts each stall exactly once whereas the sampler's WARN/ERROR log lines fire repeatedly during one stall — the two must not be summed.

Explicitly out of scope (preserve this list): #3756 (the inline peer_refresh await that creates the parks — separately in flight as urgent; this issue is the blind sensor, it removes no parks, and #3756 removes no blindness); #3793 (agent-side monitor-tick log-budget line loss — different tree, lines emitted-but-dropped vs never-emitted); #3767 (the 120 s abort itself, plus the boot-time armed-thresholds line, i.e. the issue's ask 4 — cut from this PR by Critic C and handed to #3767 per triage's de-dup request, so the two PRs do not both edit mod.rs:4086; see the note posted on #3767); re-architecting the event loop's timer set; and making the sample period configurable.

---

Kind: bug-fix

Test plan — all deterministic: pure functions, fixed PRNG seed, no sleeps, no timers, no I/O, no live park required. This is deliberate per the triage's ordering note: if #3756 lands first the empirical signal disappears, so a log-based or timing-based test would silently stop testing anything. Test-only simulation helpers live inside the #[cfg(test)] module — a pub(crate) helper called only from #[cfg(test)] trips dead_code under this repo's global -Dwarnings.

Regression tests — all in crates/app/src/app/mod.rs's #[cfg(test)] mod tests, alongside the existing watchdog_tier_routing (mod.rs:8174):

1. test_watchdog_sampler_has_no_systematic_blind_window — pure geometry simulation. For each event-loop timer period P ∈ {1, 5, 10, 30, 60} s and each stall duration D ∈ {15.5, 16.0, 20.0, 24.9} s, generate park onsets s_j = φ₀ + P·j and a sample train from the production watchdog_next_sample_delay() with a fixed, documented seed; count parks with a sample in [s_j + 15, s_j + D]. Asserts per-pair detections > 0 and an aggregate rate within a factor of 3 of the analytic min(1, (D−15)/8.5) — the rate bound is what catches a degenerate generator. Use 2,000 parks for D < 17 s (per-park detection probability is only ≈ 5.9 % at D = 15.5 s and successive samples are correlated, so 200 parks is not a comfortable margin); /do must run it and pin a seed that passes. Why it fails on origin/main: the helper takes the delay sequence as a closure; substituting the constant Duration::from_secs(10) that origin/main uses at mod.rs:3922 yields exactly 0 detections for (P = 60, D = 16.0) at the deployed geometry (onset residue 5.8 s, sample residue 2.4 s). On main the test additionally cannot compile, because the period is an unnamed inline literal — naming it is step 1 of the fix. The failing number is pinned in-tree by test 2.
2. test_fixed_period_sampler_is_blind_by_construction — the same helper driven by a constant 10 s delay; asserts detections == 0 for (P = 60, D = 16.0) and detections > 0 for D = 26.0. Documents the exact defect and stays green after the fix (it asserts about a constant sequence, not about production).
3. test_loop_side_reporting_is_complete_for_both_tiers — the ERROR-tier regression test. 200 parks on a 60 s grid at the blind residue, run for D = 16.0 s and D = 35.0 s. Asserts, with the tier qualifiers made explicit (Critic A — the draft's unqualified version was wrong): for D = 16 s, sampler WARN = 0/200 and loop-side WARN = 200/200; for D = 35 s, sampler WARN = 200/200 (the [15, 35] window is 20 s wide, wider than the 10 s period, so WARN always fires), sampler ERROR = 0/200 (the [30, 35] window is only 5 s wide ⇒ blind whenever φ mod 10 ∈ (5, 10)), and loop-side ERROR = 200/200. Fails on origin/main: there is no loop-side tier path at all, so the D = 35 s recovering-freeze class produces no ERROR-tier record whatsoever. This is the sharpest form of the finding — the sampler sees the stall and still never emits the restart sentinel.
4. test_loop_side_stall_event_does_not_trip_monitor_restart_grep — renders the loop-side ERROR-tier event through the tracing_subscriber::fmt harness already present in watchdog_freeze_field_tests (mod.rs, BufWriter) and asserts the output contains neither watchdog_freeze nor WATCHDOG: Event loop appears frozen (SKILL.md:801), while WatchdogSnapshot::emit_error() rendered the same way still contains both. Fails on origin/main: the loop-side event does not exist; it also guards the future regression of someone reusing emit_error() there and converting every recovered ≥30 s park into an automated node restart.
5. test_watchdog_sample_base_period_coprime_with_event_loop_timer_periods — asserts gcd(WATCHDOG_SAMPLE_PERIOD_BASE_SECS, P) == 1 for P ∈ {1, 5, 10, 30, 60}. Fails on origin/main: gcd(10, 60) = 10 and gcd(10, 10) = 10. Carries the comment that gcd is necessary but not sufficient (config-derived flood_tx_period/flood_demand_period), which is the recorded reason jitter exists.

New coverage:

6. test_event_loop_stall_tier_boundaries — pure event_loop_stall_tier(): 14.999 s → None, 15.0 s → Warn, 29.999 s → Warn, 30.0 s → Error; plus a cross-check that it agrees with WatchdogSnapshot::tier() for every integral stale_secs in 0..=60, so the two paths cannot drift off the shared constants.
7. test_watchdog_sample_delay_range_and_determinism — every delay lies in [7 s, 10 s); the same seed reproduces the same sequence; two different seeds diverge; seed = 0 still produces a varying sequence (the xorshift footgun in §3); and BASE_MS + JITTER_MAX_MS <= 10_000 (also a const _: () = assert!).
8. test_backlog_stale_window_is_duration_based — backlog_heartbeat_is_stuck(29 s, backlog = 5) == false, (30 s, 5) == true, (999 s, 0) == false; the 30 s window asserted in seconds, independent of the sample period. Guard for the BACKLOG_STALE_TICKS retune trap.

Existing tests preserved (cargo test -p henyey-app, run pre- and post-change):

• crates/app/src/app/mod.rs: watchdog_tier_routing (:8174), watchdog_should_abort_routing (:8189), test_watchdog_diagnostic_hint_content (:7975), test_watchdog_guard_drop_signals_shutdown (:8012), test_watchdog_thread_exits_on_guard_drop (:8043 — the mock keeps its own 5 s wait and the real guard path wakes via notify_one, so jitter cannot slow shutdown), watchdog_freeze_field_tests::{test_watchdog_freeze_emits_field_text_format, ..._json_format} — must stay behaviourally byte-identical; the restart sentinel is unchanged.
• crates/app/src/metrics.rs: catalog coverage tests over ALL_COUNTER_NAMES / ALL_HISTOGRAM_NAMES / ALL_PREREGISTERED_COUNTER_NAMES (metrics.rs:1789/1830/1857/1905).
• crates/app/tests/watchdog_shutdown.rs::test_watchdog_clean_shutdown_no_abort — the subprocess no-spurious-SIGABRT harness; the one test sensitive to sampler timing, so it is the key end-to-end guard on the period change (its 60 s budget / 25 s observation window is comfortable against a ≤10 s max delay).
• crates/app/src/config.rs: test_diagnostics_config_default_watchdog_abort_secs, test_testnet_config_watchdog_abort_secs (config.rs:3927/3932) — abort threshold untouched.

---

Parity considerations:

n/a — does not change the observable surface. docs/PARITY.md:23-38 scopes MUST-match parity to ledger/bucket hashes, transaction result & meta XDR, event XDR, SCP wire, overlay wire, history archive format, HTTP API / JSON-RPC / CLI contracts and crypto outputs — none of which this plan touches. docs/PARITY.md:55-56 explicitly classifies Metrics ("names, labels, existence, and values") and Logging ("formats, levels, and content") as freely divergeable, which covers the two new henyey_-prefixed series and every log-line change here.

stellar-core has no counterpart — confirmed, not assumed. The pinned submodule (stellar-core/ @ 7696c069, v27.0.0) has zero watchdog matches in src/ (the only hit is a Rust build artifact). The nearest upstream construct, ApplicationImpl::scheduleSelfCheck / mSelfCheckTimer (src/main/ApplicationImpl.h:145,255,269), is a bucket/archive integrity self-check on a VirtualTimer, not an event-loop-liveness sampler, and shares no observable output. So there is no upstream function whose sequencing or output this could diverge from.

Two internal contracts are in play and both are preserved: monitor-tick's restart grep (SKILL.md:801, pinned by test 4), and — the single parity-relevant implementation constraint — report_event_loop_stall must be synchronous, non-awaiting and non-blocking. lifecycle.rs is parity-adjacent as a whole file (it drives consensus triggering and close sequencing); the edit adds no select! arm, removes/reorders/gates none, and must add no .await at loop top. If it becomes async, that introduces a yield point in the loop preamble and needs a fresh parity review.

---

Minor items to consider during implementation:

• Log volume rises at the WARN tier. Every park ≥15 s now emits one loop-side line where previously only the phase-locked subset did (on the issue's own unverified numbers, ~1440/day vs ~670/day). Same order as the existing ≥2 s warn_phase_if_slow stream (lifecycle.rs:1566) and bounded by the park count, so no new unboundedness — but monitor-tick: log scan is line-budgeted, not time-budgeted — a burst collapses the window 1,908x and silently zeroes the critical greps (tick 2783 saw 1 of 11 Lost sync, 0 of 16 database-is-locked) #3793's line budget will see more lines. That is the trade for a complete signal.
• New ERROR-severity lines. Recovered stalls ≥30 s now produce ERROR lines, so monitor-tick's ERROR/FATAL counters will move off zero on a node that today reads clean. Intended; mitigated by the distinct message and stall_recovered = true; called out in monitor-tick: auto-restart rule should distinguish recovered vs unrecovered event-loop stalls #3815.
• Do not sum the two log streams. A 60 s park yields ~5 sampler WARN + ~3 sampler ERROR during, plus one loop-side line after. The new histogram counts each park exactly once — that is the durable answer to ask 3, and the HELP text should say so.
• The shutdown tail is uncovered by either path (between the last loop top and drop(watchdog_guard) at lifecycle.rs:1593). Correct behaviour, not worth code.
• Any future test asserting "a sample lands within N seconds" must use the worst case (< 10 s, guaranteed by the const assert), never the 8.5 s mean.
• Hot-path discipline: keep report_event_loop_stall #[inline] with the tier check first and nothing before the early return — maxtps is sensitive to loop-top work.
• Scope tripwire for /do: the design needs exactly one loop-local Instant and one synchronous call. If implementation starts requiring new App fields, an async reporter, or moving tick_event_loop(), stop and bounce rather than expand.
• Post-deploy falsifiable check (nothing pre-merge can be verified — validator down): the loop-side stall count must be ≥ the sampler WARN count on every day, with equality only when all parks exceed T_eff.
• Interaction with Event loop parked 13-29s once per minute (28% duty-cycle loss): peer_refresh arm awaits inline, 69k-row peers table — recurrence of #3582 #3756: if it lands first the park population vanishes and the new signal reads ~0. That is the correct outcome and weakens no test here, since none depends on observing a real park.

Sub-issues / follow-ups filed: #3815 (monitor-tick's auto-restart rule should distinguish recovered vs unrecovered stalls — the residual non-monotonicity this plan deliberately does not paper over node-side). Issue ask 4 (boot-time armed thresholds) was cut per Critic C and handed to #3767 via a comment there, rather than a new issue, per triage's de-dup request.

Convergence: Round 1, verdicts: A=REVISE-MINOR, B=APPROVE, C=REVISE-MINOR. No REVISE-MAJOR was raised, so no refute pass was required. All REVISE-MINOR items were folded into this plan: A's test-3 tier qualifier, test-1 seed/park-count + rate bound, the xorshift seed-zero guard, the backlog == 0 reset edge, the idle-wakeup assumption note, and the don't-sum-the-streams note; B's synchronous-reporter constraint; C's cut of the boot-diagnostics deliverable to #3767.