fix(dash): stop the dashboard retry storm that took the service down

[OsherElhadad]

The dashboard returned 503 to real users for over a day while the proxy itself was
perfectly healthy, and Grafana showed no data at all. One day's nginx log for the
service: 15,700 503s and 7,737 gateway timeouts against 36,702 OKs, across 19
client addresses. The process sat at 6.8 GB of Go heap against its 8 G cgroup cap
burning 3.66 cores continuously.

It was one feedback loop, and there were six independent places it could have been
broken. Every one of them is a variant of the same mistake: a cache or a bound set
against the wrong number.

The loop

1. The client turned a 5-minute poller into a 5-second hammer on first failure.
   dash/ui/app.js paced its refresh on state.loadedAt, which is assigned only
   after a load succeeds. The first failure froze it, so both backoff guards —
   each measuring now - loadedAt — were permanently satisfied and the 5s tick
   re-fired every 5s forever. The repaint error path also calls markDirty(),
   which removed the staleness cap that was the second brake. Now paced on the
   last attempt, recorded in a finally so an abort counts too, plus an
   in-flight guard so ticks cannot stack.

2. Abandoned reads ran to completion holding a pooled connection.
   WithContext/readCtx already existed in dash/store.go for exactly this
   problem — its doc comment even measures it — and was wired only to the KV-cache
   routes. The other handler call sites and 106 of 111 d.sql.Query/QueryRow
   calls ignored it. Reads now run under the request context through one accessor,
   a.db(r). Writes are untouched: a write must complete.

3. Only 3 of ~30 routes had the handler timeout, attached by hand, so every
   route added afterwards silently had none — including /api/series and
   /api/capture, the two that hung longest (4,049 and 3,549 gateway timeouts in
   a day). Mount now applies a bound by default and routeBounds names the
   exceptions.

4. Two caches had a TTL shorter than the work they cached, so they cached
   almost nothing:

   • /metrics: promCacheTTL 10s against a 15s scrape_interval meant the body
     was always already expired when a scrape arrived, so the cache could never
     serve a single scrape — that part is structural and load-independent. Every
     scrape therefore rendered from scratch, and once the render exceeded the 10s
     scrape_timeout it returned 503 and the target stayed down for hours. That
     is why Grafana was blank.

     Correction to an earlier version of this description, which said the 503 was
     "independent of load". Only the cache-miss is. The render itself measures
     8.3s cold / 3.2s warm on an idle box through the real driver, so it was not
     always over 10s on its own — the permanent 503 needed the TTL bug and the
     memory-pressure slowdown from items 1-3 together. Fixing either would have
     helped; both are fixed here.

   • /api/{stats,facets,components}: dashCacheTTL 5s against a ~6s
     Components(). With no collapsing either, 19 readers meant 19 concurrent
     multi-second scans on one pool.

   Both now hold a body longer than it costs to build, serve the previous body
   while a replacement is computed behind the response, and collapse concurrent
   misses onto one computation. dashCacheStale caps how old a served body may
   get, because the page stamps freshness when the browser fetched and cannot see
   server-side staleness.

5. /api/components failed 100% of the time on its default view. The
   unfiltered all-time aggregate measures 10.2–10.4s against a 10s bound — it lost
   by a fraction of a second, every time. That 10s existed because a comment said
   it "matches Prometheus's own scrape_timeout", which is the wrong reason for
   these three routes: nothing scrapes them. Prometheus reads /metrics. Bound is
   now 45s, well inside nginx's 600s proxy_read_timeout.

6. serveJSON's shared computation inherited the leader's request context, so
   a leader that timed out cancelled the work and singleflight handed that same
   cancelled-context error to every waiter — one reader closing a tab failed the
   tab for all of them, and completed work was discarded rather than cached.
   Measured before the fix: 19 concurrent /api/components all failed in 2ms. It
   is now detached and separately bounded.

Plus GOMEMLIMIT=7GiB in the unit. Go's GC cannot see MemoryMax, so the process
grew until the kernel started refusing it memory and the two fought — measured
live, the CPU was spread evenly across twelve threads, which is the GC's signature
rather than an application loop. This is a seatbelt, not the cure; the cure is
items 1–6 not needing the memory.

Measured

On a copy of the production database (2.23 GB), warm, medians:

call	before	after
Overview	4.0s	1.7s
DeclCreditFor (SelfRemovals)	7.2s	1.5s
Components	13.2s	3.5s
Facets	4.4s	2.8s
Series	0.2s	0.06s

Through the real mounted handlers, all-time default view:

endpoint	before	after (cold)	after (warm)
/api/stats	200 5.8s	200 5.7s	200 0ms (hit)
/api/facets	500	200 1.8s	200 0ms (hit)
/api/components	503 10.0s	200 10.1s	200 0ms (hit)

40 concurrent /api/components: 40× non-200 in 2ms → 40× 200 in 1ms.

Live, deployed on the hosted box

• 5xx rate: ~20–28/min sustained all day → 0. The only 5xx since are 502s inside
  each ~25s restart window.
• Memory: 8.28 GB → 129 MB. CPU: 376% → 29%.
• /metrics: 503 after 10s → 200 in under 1ms; Prometheus target down → up
  with 1.7ms scrapes; Grafana's cg_* series are flowing again, 13.6s behind live.

Not mine, and not fixed here

idx_tooldecl_inventory with its matching SelfRemovals GROUP BY, and not carrying
r.* through three window sorts, come from perf/dash-query-cost-0901 (79a132c,
7471b09, f97bda5) — measured against production and never merged. Its invariant is
kept intact: the index and the GROUP BY ship together, because that branch measured
the rewrite without the index at 10,749ms against the plain scan's 4,498ms. The
index lives in additiveDDL, which runs on every open, so it reaches an existing
database; schemaVersion is untouched, so no database is renamed aside. On this
deployment the one-time build took 12s.

The cold all-time Components() is still ~10s. It is now paid once per refresh
interval by one reader instead of by every reader on every load, and it is honest
about itself through X-Cache and cg_metrics_render_seconds. Making the aggregate
itself cheap needs a rollup table maintained on write — a real change with real
failure modes, which should not be smuggled in behind an outage fix.

MaxOpenConns is deliberately not changed. Reads here are CPU-bound and do not
parallelise (measured: at K=16 concurrent Overview calls each takes 41s), so a pool
of 100 cannot add throughput — but 20/10 was tried before and queued live traffic
past the timeout, and that was under the pathological demand this PR removes. It
should be re-measured against the fixed demand, not guessed at now.

Tests

go test ./... and go test -race ./... green; go vet and gofmt clean.
dash/uirefreshpacing_test.go is new and guards the pacing invariant at source
level (verified to fail on the pre-fix code, not just pass on the new). The
/api/components cache test was rewritten to assert the new
stale-while-revalidate contract including the staleness cap, rather than being
loosened to pass.