Follow-up from #3799 / PR #3929 (Critic C concern in the converged plan).
Context
PR #3929 adds the (3g) catchup-stalled report-only detector to the monitor-tick LLM skill. It is implemented as a shell predicate (classify_catchup_stalled) over the live /info state field, the henyey_startup_peak_anon_rss_mb gauge (absent-or-0), node uptime, and frozen-lcl dwell.
The gap
The (3g) predicate cannot be expressed as a metric-alarms.toml catalog entry:
- The catalog evaluator (
scripts/lib/eval-alarms.py) has no access to the /info state string or a wall-clock lcl-dwell predicate — it operates on Prometheus scrape deltas/gauges.
- An
eval_gauge alarm on the absent startup_peak gauge returns skipped/inert (the discriminator is the gauge's absence, which the catalog treats as no-data, not as a firing condition).
So any monitoring path that runs only eval-alarms.py — notably the #3757 headless watchdog tick — remains blind to this wedge, exactly as the #3799 issue body documented (the #3757 headless tick 2872 reported self_reflect: clean on the live wedge, same as the human-driven tick before (3g) existed).
Ask
Add a corroborating surface for the headless path so the catchup wedge is detectable there too. Options to evaluate during planning:
- A catalog-expressible proxy (e.g. a
apply_ledger_chain_{success,failure}==0 + herder_state==Booting + ledger-age gauge composite) that approximates the wedge without needing /info state, or
- Extend the headless watchdog to run the relevant shell gate(s) directly, or
- A Grafana SLO rule per
ALARM_SURFACES.md (fleet-visible, operator paging) since process-health-adjacent stalls are a Grafana-appropriate surface.
Related
Follow-up from #3799 / PR #3929 (Critic C concern in the converged plan).
Context
PR #3929 adds the (3g)
catchup-stalledreport-only detector to the monitor-tick LLM skill. It is implemented as a shell predicate (classify_catchup_stalled) over the live/infostatefield, thehenyey_startup_peak_anon_rss_mbgauge (absent-or-0), node uptime, and frozen-lcl dwell.The gap
The (3g) predicate cannot be expressed as a
metric-alarms.tomlcatalog entry:scripts/lib/eval-alarms.py) has no access to the/infostatestring or a wall-clock lcl-dwell predicate — it operates on Prometheus scrape deltas/gauges.eval_gaugealarm on the absentstartup_peakgauge returnsskipped/inert (the discriminator is the gauge's absence, which the catalog treats as no-data, not as a firing condition).So any monitoring path that runs only
eval-alarms.py— notably the #3757 headless watchdog tick — remains blind to this wedge, exactly as the #3799 issue body documented (the #3757 headless tick 2872 reportedself_reflect: cleanon the live wedge, same as the human-driven tick before (3g) existed).Ask
Add a corroborating surface for the headless path so the catchup wedge is detectable there too. Options to evaluate during planning:
apply_ledger_chain_{success,failure}==0+herder_state==Booting+ ledger-age gauge composite) that approximates the wedge without needing/infostate, orALARM_SURFACES.md(fleet-visible, operator paging) since process-health-adjacent stalls are a Grafana-appropriate surface.Related