runtime: add NVIDIA model health probes and automatic fallback

[meiiie]

Context\nDuring the 2026-05-09 GCP production rebuild and Cloudflare cutover, infrastructure was healthy but chat initially hung because both requested NVIDIA DeepSeek v4 models timed out from the production VM:\n\n- deepseek-ai/deepseek-v4-flash: chat completions timed out after 35-45s with no bytes\n- deepseek-ai/deepseek-v4-pro: chat completions timed out after 35s with no bytes\n\nThe NVIDIA API itself was reachable:\n\n- /v1/models: 200 in ~0.2s\n- meta/llama-3.1-8b-instruct: simple chat 200 in ~1.8-3.4s\n- qwen/qwen3-next-80b-a3b-instruct: tool-call probe 200 in ~1.35s and production smoke passed 12/12 after hotfix\n\nProduction was temporarily hotfixed to qwen/qwen3-next-80b-a3b-instruct so Wiii is alive while staying on NVIDIA provider.\n\n## Required work\n- Add model-level health probes for configured NVIDIA models at startup and periodically.\n- Mark timed-out models degraded and remove them from routing until recovery.\n- Support same-provider fallback order, e.g. DeepSeek Flash/Pro -> Qwen/Nemotron fallback.\n- Surface model health/fallback telemetry in

untime_latency, logs, and status endpoints without leaking API keys.\n- Update deploy smoke so model fallback behavior is visible and actionable.\n\n## Acceptance criteria\n- A broken configured model cannot stall normal chat for 70-170s.\n- Production chat chooses a healthy NVIDIA model automatically.\n- Visual/tool smoke continues to pass when the preferred DeepSeek model is degraded.\n- Operators can see which model is degraded and which fallback is active.\n\n## Risk\nHigh-runtime-impact area: provider routing, model selection, timeout behavior, streaming finalization. Implement in narrow PRs with targeted tests.

[meiiie]

PR #258 has been merged and deployed to production.\n\nImplemented now:\n- startup + periodic NVIDIA model health probes\n- failure classification for timeout/rate-limit/server/provider/auth errors\n- process-local model degradation using existing model health state\n- LLMPool cache eviction when a cached model has been marked degraded\n- shutdown cancellation for probe tasks\n- targeted regression coverage for probe behavior and stale cache replacement\n\nProduction deployment:\n- commit: 9c20972\n- app/nginx images pinned to sha-9c209722f8b528ee3ea0c724e3ab44c49bb185f3\n- predeploy backup: predeploy_9c209722f8b5_20260509_095829.dump\n- external smoke: 12 passed, 0 failed\n- logs confirm NVIDIA model probe healthy for qwen/qwen3-next-80b-a3b-instruct (~1.8-1.9s)\n\nStill open from this issue:\n- surface model health/fallback telemetry more explicitly in status/admin/runtime endpoints\n- make deploy smoke assert model fallback visibility, not only chat/visual behavior\n- optionally add a staged DeepSeek->Qwen fallback smoke once we can safely configure a deliberately degraded model in non-prod.

[meiiie]

Follow-up after PR #261 production deploy.\n\nCurrent production state:\n- deployed commit: ded67a9\n- external smoke: 14 passed, 0 failed\n- /api/v1/health/llm-models: healthy with model_count=2, healthy_count=2, degraded_count=0\n- model-health partial degradation semantics are now safer: a partially degraded model set is an operator warning, while all-known-models degraded still marks the component degraded.\n\nLive NVIDIA capability findings:\n- deepseek-ai/deepseek-v4-flash timed out and was not stable enough for production tool/visual paths.\n- deepseek-ai/deepseek-v4-pro supports tool calls but was inconsistent as the primary chat model under smoke constraints.\n- qwen/qwen3-next-80b-a3b-instruct currently passes chat and visual smoke, so it is the production primary/general model.\n\nRemaining work for this issue / next PR:\n- make fallback/model selection visible in per-turn chat metadata, not only health endpoints/logs\n- add a staged degraded-model smoke that intentionally proves model fallback without risking production\n- continue real latency work on supervisor/router/provider timeline so Wiii does not feel silent before the answer stream.