Observability: crash visibility + supervised background tasks (instance-local first)

[coaxk]

Why (critical, should have existed from day 1)

The v1.2.0 #79 regression (fixed in 1.2.1) crashed every coverage walk on foreign-language libraries, logged a warning, and kept silently failing for 24h+ with NO signal to the user or maintainer. subarr has no real crash visibility. With ~2k installs this class of silent failure is almost certainly happening unseen.

Design principle: the privacy line is the TRANSMIT boundary, not the data

A self-hoster seeing their own full stack traces in their own UI is their data on their own box — NOT a privacy issue. Sanitization only matters when something phones home. So rich local visibility ships to everyone for free; only the fleet channel gets sanitized.

Phase 1 — instance-local crash visibility (ship-to-all, no privacy cost) [DO FIRST]

The root danger #79 exposed: background loops that catch-log-and-keep-looping fail silently. Fix = an architectural primitive + a loud surface:

• Supervised background tasks. Wrap every long-running loop (coverage refresh, audio audit walker, scheduler, completion watcher, subgen watchdog) to record per loop: last_success_at, last_error (type + full traceback, local), consecutive_failures. A loop with no success in N cycles = unhealthy.
• Loud UI surface. Header badge + a Health/Errors panel: "Coverage refresh failing — NameError in coverage_engine:1639, last success 2h ago", with the full traceback available. Forced-only embedded English: subarr flags a gap that subgen always skips (forced≠full seam) #79 would've been a red banner on minute one instead of a 24h silent freeze.
• Reuse existing infra: ErrorStore + error_events table (feat(telemetry): real walks_per_day_30d + error_counts_30d #34), /api/health, the Queue "Issues" section, the Logs page. Mostly wiring background failures into surfaces that already exist → fast.
• Dev-only verbose knob (SUBARR_DEBUG): extra detail / payload dumps / perf timing beyond the shipped default, for the maintainer's dev box ("go nuts locally").

Phase 2 — fleet crash telemetry (hold the privacy line) [LATER]

Sanitized {exception_type, module:line/function, subarr_version, count} ONLY — never message/locals/paths. Aggregated on the existing self-owned CF Worker + D1 (no third-party SaaS), opt-out, shown in the transparency panel. Rides the federated data pipeline (#95). A spike of NameError @ coverage_engine across installs after a release = instant regression alarm (would've caught #79 within hours of 1.2.0).

Phase 3 — perf/APM trends [LATER, don't cargo-cult]

No per-request tracing. Trend metrics on hot paths (subgen-call latency, scan throughput) — much already surfaced (build_duration_s, GPU, queue depth). Flag degradation.

Sequencing

Phase 1 is the urgent, fast, free-for-everyone piece → next Track-A item after 1.2.1. Phase 2 folds into the federated pipeline. Phase 3 later.

Related: #79 (the bug that motivated this), #34 (ErrorStore/error_events seed), #95 (federated pipeline for Phase 2), #138 (perf).

[coaxk]

Phase 1 shipped (on main: backend e888309, frontend 1b2826f; full suite 625 green, frontend CI green).

• task_health store (migration 012) + all 6 persistent loops supervised (coverage-cache, dashboard-cache, scheduler, completion-watcher, update-checker, subgen-watchdog). Silent loops now capture the traceback they discarded. Best-effort recording (never crashes the loop it monitors). is_unhealthy = ≥3 consecutive failures OR no success in 3× cadence.
• GET /api/health/tasks + any_unhealthy rollup.
• Dedicated Health page (/health): per-task status, cadence, last-success, consecutive failures, expandable last-error traceback. Header pill (red, " failing") + Health-rail badge when anything is unhealthy.

Live-verified: healthy roster renders; a simulated failure (the exact #79 NameError) surfaces as a red row + full traceback + pill + badge, then reverts cleanly.

Phase 2 (sanitized fleet crash telemetry on the federated pipeline) + Phase 3 (perf trends) remain — tracked here.