diff --git a/.agents/skills/monitor-loop/SKILL.md b/.agents/skills/monitor-loop/SKILL.md index a4be555bd..7766537aa 100644 --- a/.agents/skills/monitor-loop/SKILL.md +++ b/.agents/skills/monitor-loop/SKILL.md @@ -245,11 +245,11 @@ table is the human reference. These counter-based checks use streak gating rather than immediate-fire thresholds because single-tick increments are often transient self-recovering events that don't warrant operator attention (see #2309). State is tracked independently of -the ratio checks in a separate snapshot (`metrics/counter_streak_snapshot`). +the ratio checks in a separate snapshot (`metrics/recovery_family_streak_snapshot`). | Metric | Delta threshold | Streak threshold | Burst threshold | Severity | Rationale | |--------|-----------------|------------------|-----------------|----------|-----------| -| `henyey_recovery_stalled_tick_total{reason="forcing_catchup_behind"}` | ≥ 1 | 3 ticks | ≥ 10 | WARN | Recovery forced catchup while behind consensus; single occurrences are transient self-recovering events (see #2309); large bursts indicate sustained stalling | +| `henyey_recovery_stalled_tick_total` (sum of all `reason` series, #3824) | ≥ 1 | 3 ticks | ≥ 10 | WARN | Recovery stalled — the delta/streak/burst trigger observes the SUM of every `reason` series (an at-tip stall moves `forcing_catchup_not_behind`/`near_tip_peer_scp_recovery`, not `forcing_catchup_behind`); single-tick blips are transient self-recovering events (see #2309/#3728) absorbed by streak-3 gating; large bursts indicate sustained stalling. Post-restart absolute guard stays scoped to `forcing_catchup_behind` via `post_restart_absolute_label` | **D. Ratio checks — fire on sustained ratio breach (3 consecutive ticks)** @@ -296,7 +296,7 @@ pending_breach_streak= ``` Invalidate on PID/start_ticks change, malformed snapshot, or counter reset (current < previous). -**Counter-streak snapshot** persisted at `~/data//metrics/counter_streak_snapshot` +**Counter-streak snapshot** persisted at `~/data//metrics/recovery_family_streak_snapshot` (format and invalidation rules defined in Check 12b of monitor-tick/SKILL.md; path canonicalized in [`shared/metric-alarms.toml`](../shared/metric-alarms.toml)). Separate from ratio snapshot — runs independently of ratio skip conditions (see diff --git a/.agents/skills/monitor-tick/SKILL.md b/.agents/skills/monitor-tick/SKILL.md index 4db59a532..2c6ce6345 100644 --- a/.agents/skills/monitor-tick/SKILL.md +++ b/.agents/skills/monitor-tick/SKILL.md @@ -121,7 +121,7 @@ All files below live in `/home/tomer/data/$MONITOR_SESSION_ID/`: | `metrics/prev.prom` | previous Prometheus scrape | check 12 | | `metrics/scrape_identity` | process identity of the scrape now in prev.prom | check 12 | | `metrics/ratio_snapshot` | counter-ratio history (check 12) | check 12 | -| `metrics/counter_streak_snapshot` | counter-streak state (check 12b) | check 12b ([metric-alarms](../shared/metric-alarms.toml)) | +| `metrics/recovery_family_streak_snapshot` | counter-streak state (check 12b; #3824 family-union) | check 12b ([metric-alarms](../shared/metric-alarms.toml)) | | `metrics/anomaly_cooldown.json` | alert dedup state | check 9 | | `metrics/archive/` | Per-tick snapshot dirs (current.prom + prev.prom + metadata.env), rolling 500, atomic write | check 12 | | `logs/monitor.log` | node stdout/stderr (rotated on restart) | node process | @@ -1119,7 +1119,7 @@ against a node that is in real-time sync with age=2s). breach on the next tick before firing. - All other §GAUGES are unaffected — they are point-in-time readings from `current.prom` only. - - Do NOT skip ratio_snapshot or counter_streak_snapshot checks — they have + - Do NOT skip ratio_snapshot or recovery_family_streak_snapshot checks — they have their own independent PID/start_ticks invalidation logic and snapshot files. Independence is safe: each check reads PID/start_ticks from `/proc` and compares against its own snapshot. @@ -1357,12 +1357,25 @@ eval_result=$(python3 scripts/lib/eval-alarms.py \ > This section is authoritative for the state machine *logic*; inline literals > are cross-validated against the TOML by `scripts/test-monitor-skill-snippets.sh`. -This check tracks `henyey_recovery_stalled_tick_total{reason="forcing_catchup_behind"}` -using a streak-gated alert, independent of Check 12's ratio checks. It runs on -its own state machine because ratio checks are globally skipped during unsync +This check tracks the **sum of every `reason` series** of +`henyey_recovery_stalled_tick_total` (family-union, `extraction = "form2-sum-all"`, +#3824) using a streak-gated alert, independent of Check 12's ratio checks. It runs +on its own state machine because ratio checks are globally skipped during unsync states (ledger age > 30s, gap > 5, etc.), but the recovery-stalled counter fires precisely during recovery transitions when the node is briefly unsynced. +> **Why the family sum, not a single label (#3824):** Check 12b previously keyed +> only on `{reason="forcing_catchup_behind"}`. During an *at-tip* stall the node +> takes the `forcing_catchup_not_behind` branch by construction, so the one label +> the alarm watched was exactly the branch that could not move — a real recovery +> episode incremented two uncovered labels and the tick reported `ok (delta=0)`. +> Summing the whole family makes the trigger observe whichever branch moves and +> covers any future `reason`. On a fire a per-reason breakdown is appended +> (`[by reason: