feat(grafana): fill the two empty rows, graph the metrics nobody was watching, and alert on the observability path

[OsherElhadad]

The main dashboard had two sections an operator reaches for during an incident —
"Is storage and cold storage healthy?" and "Is anything failing?" — that
rendered as bare header bars. Both rows were collapsed: true with their panels
nested inside them (5 and 9 respectively), so the content existed but was
invisible until someone clicked. The alerting directory shipped with no rules, so
nothing paged.

Correction to an earlier version of this description. I first wrote that those
rows had no panels under them, and that the main dashboard went from 27 to 45
panels. Both were wrong. The panels were there, collapsed; and counting nested
panels the real change is 41 → 45, i.e. 4 genuinely new panels plus
un-collapsing 14 that were already written. The 27→45 figure compared top-level
counts on one side against flattened counts on the other. Corrected below.

This is the companion to #183, which fixes the outage itself. That outage is the
argument for this PR: /metrics had a cache TTL shorter than the scrape interval,
so it never served a scrape, the Prometheus target sat down for hours, and every
cg_* panel read "No data" — indistinguishable, from the dashboard, from the
service being dead. Nothing in Grafana said which.

What changed

context-guru: 41 → 45 panels (4 new), plus three layout bugs fixed.

• Both collapsed rows un-collapsed and flattened to top-level siblings, matching
  what the live deployment already had. Someone had expanded them in Grafana's UI and
  it was never exported back to the repo.
• Rows 1 and 7 were vertically swapped — "Is it up and healthy?" sat at
  gridPos.y=22 and "Am I saving tokens and money?" at y=0, so the dashboard
  rendered saving above up/healthy, contradicting the file's own __comment
  ("Rows answer questions in order: is it up, is it saving..."). Fixed.
• Three stat panels were orphaned at the end of the JSON array despite gridPos
  slotting them into row 7's band — file hygiene, reordered, no visual change.
• One panel violated the file's own colour rule: "Total avoided this month" used
  aqua (#199e70, reserved for values that cannot go negative) for its ≥0 step, on a
  value that can go negative. Its simpler twin does red<0/blue≥0 correctly. Fixed.
• 4 new panels: Net saved this month (cg_net_saved_usd — whose own HELP text
  calls it "the number to alert on" and which was graphed nowhere), Extraction: net
  value, Cache-frozen headroom, Expand recovery failures, by cause. Plus an
  archived-bytes column on the per-tenant table.

context-guru service SLO: 16 → 19 panels (3 new), all in "Correctness of the
observability path itself" — exactly where this week's outage would have shown up and
did not. cg_metrics_age_seconds is the one that matters: an exposition age that stops
advancing distinguishes "slightly stale numbers" from "numbers stopped updating
entirely" in a single number. Those two series come from #183.

Alerting: three rules for failures that were previously silent — service down,

10% of requests refused, and the metrics exposition ageing past its refresh interval.

Validation

Every panel's PromQL was run against this deployment's Prometheus before being
committed, because an unvalidated panel is the bug this PR exists to fix:

• 83 of 86 Prometheus queries return series on live data. The 3 that did not were
  the new cg_metrics_* panels, which only exist once fix(dash): stop the dashboard retry storm that took the service down #183 is deployed — it now is,
  and they return real values (age sawtoothing under ~65s, render ~1.2–1.4s).
• Sample validated values: cg_net_saved_usd = 9.2305; cg_extract_net_value_usd =
  -0.3784 (extract_llm is currently underwater, which no panel could show
  before); cg_frozen_tokens_total = 457,478,707; cg_tenant_archived_bytes per
  tenant e.g. 1,608,835.
• The 17 Loki queries were checked against Loki, healthy and ingesting (~3,145
  lines per 5-minute window).
• All three alert expressions return series.
• Layout re-verified programmatically: zero panel overlaps, zero dead space, in both
  edited files. Both dashboard uids unchanged, so provisioning and existing links
  still resolve. Datasource uids match the provisioned cg-prom / cg-loki.

Colour

Untouched: blue #3987e5 actual, orange #d95926 comparison/baseline, aqua #199e70
savings — previously validated CVD-safe on Grafana dark at all-pairs ΔE 9.4. The one
colour change is the bug fix above, which brings a panel back into that scheme
rather than departing from it.

Drift worth knowing about

This PR changes the deployed dashboard's visible layout, not just its panel set:
row order changes and two rows stop being collapsed. Worth telling whoever expanded
those rows live, in case they have other un-exported UI edits.

The deployed copy of context-guru.json came from d9e2f24, which is not on main —
41 panels that only ever existed on the box. This change is a verified strict superset,
so landing it brings that work into the repo rather than reverting it.
context-guru-host.json is byte-identical to live and untouched.
context-guru-logs.json is untouched and the repo is already ahead of live there.

Deliberately not added

No panel over a metric that does not exist, and no wall of panels for its own sake.
Left out: the extract_llm drill-down series (cg_extract_calls_total,
gate_declines_total, latency_ms) until the new net-value panel says someone needs
them; cg_llm_tokens_total (token twin of an existing dollar panel);
cg_saved_usd/cg_baseline_cost_usd (inputs to the net figure, already visible in
aggregate).

The one thing an operator still cannot see in Grafana is the dashboard API's own 5xx
rate. During this outage nginx logged 15,700 503s while Grafana looked fine. I
grepped proxy/ and dash/ for any HTTP status/route instrumentation — there is none;
cg_dash_events_total is the transcript-capture pipeline, and the refusalReasons set
covers LLM-proxy refusals, not dash route handlers. Closing that needs a
cg_dash_http_requests_total{status}-style counter in middleware that does not exist
today. That is a code change and belongs with #183's family, not here.