Add vigil watch headless hang watchdog

CHANGELOG.md

@@ -9,6 +9,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- `vigil watch`: headless hang watchdog for always-on machines. Combines log silence with `nvidia-smi` sampling to alert on hung, exited, silently-busy (possible NCCL deadlock), unreachable and preempted instances, with the cost of the silence in every alert, reminders, and recovery notices
+- `ntfy` webhook format for phone push notifications
+- Instance start time from the Vast.ai and RunPod APIs, used for total cost in alerts
+- `watch_skip_labels` and per-instance `watch: false` to keep `vigil watch` away from dev boxes and servers
+
 - Provider abstraction layer — vigil now supports multiple GPU cloud providers
 - RunPod provider with full support (auto-discovery, cost tracking, instance management)
 - `--provider` flag now accepts `runpod` in addition to `vast`
@@ -30,8 +35,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Contextual onboarding hints (non-intrusive, dismissable, persistent)
 - `--reset-hints` CLI flag to re-show onboarding hints
 
+### Changed
+
+- RunPod API key is sent as an `Authorization: Bearer` header instead of a URL query parameter, and API keys are redacted from error messages (previously a failed request printed the key)
+- Webhook alert titles say "vigil alert" instead of "Vast.ai Alert" (they were wrong for RunPod)
+- `--provider` now defaults to the `provider` set in the config file instead of always `vast`
+
 ### Fixed
 
+- Progress bars that redraw with `\r` (tqdm) are now streamed instead of piling up until the next newline; they count as output for stall detection, and at most one redraw per 10s is logged
 - Tab focus now correctly sets `_focused_id` (unifies Tab/click/1-9)
 - Alert borders auto-clear when NaN/plateau conditions resolve
 - NaN alert deduplication prevents notification spam

README.md

@@ -138,11 +138,59 @@ vigil [OPTIONS]
 
   --config, -c PATH    Config file path (default: ~/.config/vigil/config.yaml)
   --api-key KEY        Provider API key (overrides config/env)
-  --provider NAME      GPU provider (default: vast) [vast, runpod]
+  --provider NAME      GPU provider (default: from config, else vast) [vast, runpod]
   --demo               Run with simulated instances (no API key or SSH needed)
   --reset-hints        Reset onboarding hints so they appear again
 ```
 
+## Hang Watchdog: `vigil watch`
+
+Get pinged when a rented GPU training run hangs, with no changes to your training code. `vigil watch` runs headless, so leave it on any always-on machine (a small VPS, a home server, a Raspberry Pi) instead of your laptop.
+
+```bash
+vigil watch --webhook https://ntfy.sh/<your-secret-topic> --webhook-format ntfy
+```
+
+It streams logs from every running instance over SSH, samples `nvidia-smi` every minute, and combines the two:
+
+| Output | GPU | Alert |
+|---|---|---|
+| Stopped for `stall_threshold_minutes` | Idle, memory still allocated | **hang**: training looks stuck |
+| Stopped | Idle, memory released | **exited**: process finished or crashed, instance still billing |
+| Stopped for `watch_busy_silence_minutes` | Busy | **busy_silent**: long eval, or an NCCL deadlock (these spin at 100%) |
+| Stopped | `nvidia-smi` over SSH failing | **unreachable** |
+| Resumes after an alert | | **recovered** |
+| Instance disappears from the provider | | **gone**: stopped, destroyed or preempted |
+
+The GPU must stay idle across the whole stall window before a hang is reported, so short dips between epochs don't page you. Progress-bar redraws (tqdm and friends) count as output, so a run that only updates its bar between log lines isn't mistaken for a silent one. Each alert says what the silence has cost so far:
+
+> No new output for 47m and GPU idle (0% util) while still holding memory. Training looks hung. Cost: ~$1.88 during this silence, ~$14.40 total (6.0h @ $2.400/hr).
+
+Unresolved alerts repeat every `watch_reminder_minutes` with the updated cost. Logs are persisted to `log_dir` just like the TUI, so they survive the instance being destroyed.
+
+To leave some instances alone (a Jupyter box, an inference server), skip them by label or ID:
+
+```yaml
+watch_skip_labels: [jupyter, dev]   # instances whose label contains any of these
+instances:
+  "12345678":
+    watch: false
+```
+
+Alerts go to any `alert_webhook_url`: [ntfy](https://ntfy.sh) (phone push, no account needed), Slack, Discord, or raw JSON. Check the setup with `vigil watch --test-alert`. To keep it running, use `tmux`, `nohup vigil watch >> ~/vigil-watch.log 2>&1 &`, or a systemd user service.
+
+```text
+vigil watch [OPTIONS]
+
+  --config, -c PATH       Config file path
+  --api-key KEY           Provider API key
+  --provider NAME         vast or runpod (default: from config, else vast)
+  --webhook URL           Alert webhook URL (overrides config)
+  --webhook-format FMT    raw, slack, discord or ntfy
+  --stall-minutes N       Minutes without output before a run is checked for a hang
+  --test-alert            Send one test alert and exit
+```
+
 ## Configuration
 
 Configuration lives at `~/.config/vigil/config.yaml`. All fields are optional with sensible defaults. See [`config.example.yaml`](config.example.yaml) for the full reference.
@@ -244,11 +292,11 @@ plateau_metrics: [loss]    # which metrics to watch
 
 ### Webhook Alerts
 
-Supports raw JSON, Slack Block Kit, and Discord embed formats:
+Supports raw JSON, Slack Block Kit, Discord embed, and ntfy formats:
 
 ```yaml
 alert_webhook_url: "https://hooks.slack.com/services/..."
-alert_webhook_format: "slack"  # "raw", "slack", or "discord"
+alert_webhook_format: "slack"  # "raw", "slack", "discord", or "ntfy"
 
 notifications:
   nan: true        # NaN/Inf detected in metrics
@@ -258,6 +306,16 @@ notifications:
   webhook: true    # Master switch for webhook alerts
 ```
 
+### Watchdog (`vigil watch`)
+
+```yaml
+watch_gpu_poll_seconds: 60       # how often to sample nvidia-smi
+watch_gpu_idle_percent: 5        # max GPU util that still counts as idle
+watch_busy_silence_minutes: 30   # silence before alerting even though the GPU is busy
+watch_reminder_minutes: 60       # repeat unresolved alerts (0 = once)
+watch_skip_labels: []            # skip instances whose label contains any of these
+```
+
 ### Per-Instance Overrides
 
 ```yaml

config.example.yaml

@@ -62,12 +62,34 @@ desktop_notifications: true
 # Payload: {timestamp, instance_id, alert_type, message, metrics?}
 alert_webhook_url: null
 
-# Webhook payload format: "raw" (default), "slack", or "discord"
+# Webhook payload format: "raw" (default), "slack", "discord", or "ntfy"
 # - raw: {timestamp, instance_id, alert_type, message, metrics?, gpu_name?, ...}
 # - slack: Slack Block Kit message with rich formatting
 # - discord: Discord embed with color-coded severity
+# - ntfy: plain-text push to an ntfy.sh topic URL (phone notifications, no account)
 alert_webhook_format: "raw"
 
+# ── Watchdog (`vigil watch`) ─────────────────────────────────────────
+# Headless mode that alerts when output stops, using GPU utilization to
+# tell hung, exited and busy runs apart. Uses stall_threshold_minutes above.
+
+# Seconds between nvidia-smi samples on each instance
+watch_gpu_poll_seconds: 60
+
+# Max GPU utilization (%) that still counts as idle
+watch_gpu_idle_percent: 5
+
+# Minutes of silence before alerting even though the GPU is busy
+# (long evals vs. NCCL deadlocks, which spin at 100% util)
+watch_busy_silence_minutes: 30
+
+# Repeat unresolved alerts with updated cost every N minutes (0 = alert once)
+watch_reminder_minutes: 60
+
+# Skip instances whose label contains any of these (case-insensitive).
+# To skip a single instance instead, set `watch: false` under instances.<id>.
+watch_skip_labels: []
+
 # ── Display ───────────────────────────────────────────────────────────
 
 # Number of metric readings to keep for sparkline rendering
@@ -167,3 +189,4 @@ reconnect_backoff_max: 60
 #     ssh_username: "trainer"
 #   67890:
 #     log_command: "docker logs -f training_container"
+#     watch: false      # `vigil watch` ignores this instance

src/vigil/__main__.py

@@ -1,16 +1,17 @@
 from __future__ import annotations
 
 import argparse
+import asyncio
+import sys
 from pathlib import Path
 
-from .config import DEFAULT_CONFIG_PATH, load_config
-from .providers import get_provider
+from .config import DEFAULT_CONFIG_PATH, Config, load_config
+from .providers import Provider, get_provider
 
+WEBHOOK_FORMATS = ["raw", "slack", "discord", "ntfy"]
 
-def main() -> None:
-    parser = argparse.ArgumentParser(
-        description="Real-time TUI for monitoring cloud GPU training instances",
-    )
+
+def _add_common_args(parser: argparse.ArgumentParser) -> None:
     parser.add_argument(
         "--config", "-c",
         type=Path,
@@ -26,25 +27,20 @@ def main() -> None:
     parser.add_argument(
         "--provider",
         type=str,
-        default="vast",
+        default=None,
         choices=["vast", "runpod"],
-        help="GPU cloud provider (default: vast)",
-    )
-    parser.add_argument(
-        "--reset-hints",
-        action="store_true",
-        help="Reset onboarding hints so they appear again",
-    )
-    parser.add_argument(
-        "--demo",
-        action="store_true",
-        help="Run with simulated GPU instances (no API key or SSH needed)",
+        help="GPU cloud provider (default: from config, else vast)",
     )
-    args = parser.parse_args()
 
+
+def _load(args: argparse.Namespace) -> tuple[Config, Provider]:
     config = load_config(args.config)
-    config.provider = args.provider
-    provider = get_provider(config.provider)
+    if args.provider:
+        config.provider = args.provider
+    try:
+        provider = get_provider(config.provider)
+    except ValueError as exc:
+        sys.exit(f"vigil: {exc}")
 
     # Re-resolve API key now that provider is known (checks RUNPOD_API_KEY, etc.)
     if not config.api_key:
@@ -53,6 +49,103 @@ def main() -> None:
 
     if args.api_key:
         config.api_key = args.api_key
+    return config, provider
+
+
+def _watch_main(argv: list[str]) -> None:
+    parser = argparse.ArgumentParser(
+        prog="vigil watch",
+        description=(
+            "Headless hang watchdog. Streams logs from every running instance and alerts "
+            "when output stops, using GPU utilization to tell hung, exited and busy runs apart."
+        ),
+    )
+    _add_common_args(parser)
+    parser.add_argument("--webhook", type=str, default=None, help="Alert webhook URL (overrides config)")
+    parser.add_argument(
+        "--webhook-format",
+        type=str,
+        default=None,
+        choices=WEBHOOK_FORMATS,
+        help="Webhook payload format (overrides config)",
+    )
+    parser.add_argument(
+        "--stall-minutes",
+        type=int,
+        default=None,
+        help="Minutes without output before a run is checked for a hang (overrides config)",
+    )
+    parser.add_argument(
+        "--test-alert",
+        action="store_true",
+        help="Send one test alert to the configured webhook and exit",
+    )
+    args = parser.parse_args(argv)
+
+    config, provider = _load(args)
+    if args.webhook:
+        config.alert_webhook_url = args.webhook
+    if args.webhook_format:
+        config.alert_webhook_format = args.webhook_format
+    if args.stall_minutes is not None:
+        config.stall_threshold_minutes = args.stall_minutes
+
+    if args.test_alert:
+        if not config.alert_webhook_url:
+            sys.exit("vigil watch: no webhook configured (use --webhook or alert_webhook_url in config)")
+        from .alerts import post_webhook_alert
+        asyncio.run(post_webhook_alert(
+            config.alert_webhook_url,
+            "test",
+            "test",
+            "Test alert from vigil watch. If you can read this, alerts are working.",
+            format=config.alert_webhook_format,
+        ))
+        print(f"Sent test alert ({config.alert_webhook_format}) to {config.alert_webhook_url}")
+        return
+
+    if not config.api_key:
+        names = ", ".join(provider.env_var_names())
+        sys.exit(f"vigil watch: no {provider.display_name} API key (set {names}, --api-key, or api_key in config)")
+    if not config.alert_webhook_url:
+        print("warning: no webhook configured; alerts will only be printed here", file=sys.stderr, flush=True)
+
+    from .watch import Watcher
+
+    print(
+        f"vigil watch: {provider.display_name}, stall after {config.stall_threshold_minutes}m, "
+        f"GPU idle below {config.watch_gpu_idle_percent:g}%, "
+        f"alerts -> {config.alert_webhook_format if config.alert_webhook_url else 'stdout'}",
+        flush=True,
+    )
+    try:
+        asyncio.run(Watcher(config, provider).run())
+    except KeyboardInterrupt:
+        pass
+
+
+def main(argv: list[str] | None = None) -> None:
+    argv = sys.argv[1:] if argv is None else argv
+    if argv and argv[0] == "watch":
+        _watch_main(argv[1:])
+        return
+
+    parser = argparse.ArgumentParser(
+        description="Real-time TUI for monitoring cloud GPU training instances",
+        epilog="Run `vigil watch --help` for the headless hang watchdog.",
+    )
+    _add_common_args(parser)
+    parser.add_argument(
+        "--reset-hints",
+        action="store_true",
+        help="Reset onboarding hints so they appear again",
+    )
+    parser.add_argument(
+        "--demo",
+        action="store_true",
+        help="Run with simulated GPU instances (no API key or SSH needed)",
+    )
+    args = parser.parse_args(argv)
 
     # Handle --reset-hints
     if args.reset_hints:
@@ -62,6 +155,8 @@ def main() -> None:
         print("Onboarding hints have been reset.")
         return
 
+    config, provider = _load(args)
+
     # Load UI state
     from .state import load_state
     state = load_state()