perf(stats): fold the dashboard's readings out of the statistics rollups

ARCHITECTURE.md

@@ -244,10 +244,10 @@ Everything is in a single SQLite file (`noadd.sqlite3` by default; a legacy `noa
 | Index | Serves |
 | --- | --- |
 | `timestamp` | the time-window filter every stats query starts with |
-| `(domain, timestamp)` | the query log's domain search; top and unique domains for callers asking for domains alone |
-| `(timestamp, domain, client_ip, doh_token)` | top domains and top clients together, on the Statistics page and the dashboard |
-| `(timestamp, blocked, cached, response_ms, query_type, has_result)` | timeline, query-type breakdown, latency histogram, outcome breakdown, the dashboard summary |
-| `(timestamp, upstream, response_ms) WHERE upstream IS NOT NULL` | top upstreams |
+| `(domain, timestamp)` | the query log's domain search |
+| `(timestamp, domain, client_ip, doh_token)` | nothing since the top domain and client lists moved onto the rollups (*Rollups* below) |
+| `(timestamp, blocked, cached, response_ms, query_type, has_result)` | the Statistics page's scan, the `/api/stats/*` timeline and breakdowns, the query log's action and type filters |
+| `(timestamp, upstream, response_ms) WHERE upstream IS NOT NULL` | nothing since top upstreams moved onto the rollups |
 
 `(domain, timestamp)` puts the grouped column first and `timestamp` last, which made it covering for a `GROUP BY … WHERE timestamp >= ?` shape — the aggregation reads the index alone instead of scanning the window and building a temp b-tree over it. The client index that sat beside it, `(client_ip, doh_token, timestamp)`, did the same for top clients and took them from 143 ms to 20 ms on a 447 k-row database. The cost of that order is that the window cannot restrict it: a 24-hour question skip-scans the whole index, one seek per distinct group.
 
@@ -271,7 +271,11 @@ The quarter hour is there because it is the finest bucket any chart draws and th
 
 Deletes are not a trigger. A `DELETE` trigger would unwind a prune row by row and turn off SQLite's truncate optimisation for Clear All, so both do it in their own transaction instead: Clear All empties the five tables, and `prune_logs_before` calls `unwind_stats_rollups` before its delete. That drops whole units before the cutoff and, for the one quarter and one hour the cutoff falls inside, recounts the rows about to go and subtracts them — so the prune keeps its exact cutoff rather than rounding retention to the hour. Pruning a day from that database writes 5 151 pages and misses 13 971, against 5 020 and 12 282 without rollups. `rollups_follow_every_write_that_changes_query_logs` (`src/db.rs`) is the guard: it compares every table with its recount after batches, a direct SQL insert, prunes inside and on a unit boundary, and Clear All.
 
-Version 16 fills the rollups from the rows already logged, which on that database reads 101 420 pages and writes 3 714. The fill replaces rather than adds, so a migration interrupted before `user_version` moved is safe to run again. Nothing reads the rollups yet; the dashboard and the Statistics page move onto them in later changes.
+Version 16 fills the rollups from the rows already logged, which on that database reads 101 420 pages and writes 3 714. The fill replaces rather than adds, so a migration interrupted before `user_version` moved is safe to run again.
+
+A reader takes every unit from the first whole one inside its window (`first_whole_unit`) from a rollup, and only the rest of the unit the window starts inside from `query_logs`, through `idx_query_logs_timestamp`. Nothing is read from the table at the other end: the rollups are written in the same transaction as the rows, so the unit still filling up is already complete. Both halves are one statement, so they read one snapshot. `summary_multi_since` tells its three windows apart by which arm a table row came from — a row belongs to the one window whose partial quarter it fills, while the wider windows count that quarter through the rollup. `timeline_since` reads `query_stats_quarter` when its bucket is a whole number of quarters and counts the table directly otherwise, which only happens while the log is younger than a few hours.
+
+The dashboard reads nothing else: `summary_multi_since`, `timeline_since`, `traffic_lists_since` and `top_upstreams_since` all fold rollups, and so does `domain_stats_since`, which answers the domain suggestions. On the 1.48 M-row database a dashboard tick went from 10 774 page misses to 227, the first response from 10 444 to 215, and the domain suggestions from 6 192 to 559; a Statistics visit over 30 days went from 31 883 to 11 641, the rest of it being `stats_scan_since`. `dashboard_readings_equal_a_recount_of_the_table` (`tests/stats_db_test.rs`) holds every one of these to the statement it replaced, run on the same rows, for windows starting before the data, on an hour, on a quarter, inside each, and after it.
 
 ### Measuring these queries
 
@@ -291,9 +295,9 @@ The charts did still pay for scans of their own after that: the browser fetched
 
 The Database Health card's row count is the one reading that is not a scan of anything. `SELECT COUNT(*)` has no shortcut in SQLite — it walks the smallest index end to end, 1 386 pages on that database, for a number the card prints and two of its estimates divide by — so the count lives in `settings` under `query_log_count`, seeded by the version-13 migration and moved by the three statements that change how many rows `query_logs` holds: the logger's insert batch, the hourly prune, and Clear All. Each moves it inside its own transaction, which is what makes the counter unable to disagree with the table; `total_log_count` falls back to counting when the row is missing, which is the state the migration seeds it out of. The card went from 1 398 pages to 14. The query log's pager asks for the same number whenever no filter narrows it, so `count_logs` reads the counter in that case through the same `read_log_count`, and only counts once a filter is applied.
 
-The dashboard pays for its readings every 10 seconds rather than once a visit, which makes a scan it repeats the most expensive kind. Its summary asked two statements for totals and blocks, then cache hits and latency, over the same 30 days of the metrics index; `summary_multi_since` moves the allowed-only filter from the `WHERE` into each `CASE` and answers both from one scan, 2 152 pages a tick instead of 4 304. With the upstream index, a tick on that database dropped from 8 304 page misses to 4 755.
+The dashboard pays for its readings every 10 seconds rather than once a visit, which makes a scan it repeats the most expensive kind. Its summary asked two statements for totals and blocks, then cache hits and latency, over the same 30 days of the metrics index; `summary_multi_since` moves the allowed-only filter from the `WHERE` into each `CASE` and answers both from one scan, 2 152 pages a tick instead of 4 304. With the upstream index, a tick on that database dropped from 8 304 page misses to 4 755. Those readings have since moved onto the rollups (see *Rollups*), which no longer read either index.
 
-`INDEXED BY` appears on every statement that reads this index, in both directions. The ones that need `blocked`, `cached` or `has_result` name `idx_query_logs_ts_metrics` because the planner otherwise takes the smaller `idx_query_logs_timestamp` and pays a rowid lookup per row; top upstreams names its partial index so drifting statistics cannot send it back to that lookup; the heatmap, which reads `timestamp` and nothing else, names `idx_query_logs_timestamp` for the opposite reason — left alone the planner took the metrics index and read 2 153 pages where 1 386 answer it.
+`INDEXED BY` appears on every statement that reads this index, in both directions. The ones that need `blocked`, `cached` or `has_result` name `idx_query_logs_ts_metrics` because the planner otherwise takes the smaller `idx_query_logs_timestamp` and pays a rowid lookup per row; the rollup readers' table arms name `idx_query_logs_timestamp`, where that lookup is the point — they read at most one unit of rows; the heatmap, which reads `timestamp` and nothing else, names `idx_query_logs_timestamp` for the opposite reason — left alone the planner took the metrics index and read 2 153 pages where 1 386 answer it.
 
 Every index migration runs `ANALYZE`. A new index alone is not always enough — the planner keeps its old plan until `sqlite_stat1` is refreshed — and the hourly `PRAGMA optimize` lets those statistics drift a long way in the meantime.
 

CLAUDE.md

@@ -133,7 +133,7 @@ Statistics adds the conventions for a page whose readings sit in a **chosen wind
 - **The range is in the URL and the switcher is three `<a>`s** (`/stats?range=30d`), because the range picks the *server's* window. `StatsRange::label()` is the one spelling shared by the link, the parse and every card title. An unrecognised range renders the default rather than 400ing — it is a link an operator can edit, and every window on offer is spelled out right above it.
 - **A date the server can only write in UTC ships as an ISO day plus its timestamp** (`data-date-ts`), and `app.js` restates it in the browser's locale — the same division as the query log's relative times, and for the same reason.
 - **This page is measured in page misses, not milliseconds.** Development is on an SSD and the appliance runs off an SD card, so a duration measured here says nothing about a Raspberry Pi; the pages a query fetches from the file are the same on both. `cargo nextest run --release --no-capture --run-ignored only stats_page_miss` with `BENCH_DB` pointed at a copy of a real database reports them; `dashboard_page_miss` and `logs_page_miss` do the same for the dashboard (first response and tick) and the query log (every filter), and `BENCH_NOW` pins the clock on a copy older than its windows. A wall-clock reading is the thing to distrust when the two disagree — it is what left the outcome breakdown scanning the whole table through version 11.
-- **One scan per index, not one per reading.** `stats_scan_since` and `traffic_lists_since` (`src/db.rs`) are the page's two scans — the second answering top domains, the distinct-domain count and top clients from one grouping of `(domain, client_ip, doh_token)` — and every reading on it — the charts included — is folded out of one of them; `compute_range_stats` (`src/admin/stats.rs`) is what the page calls. `stats_scan_since` streams its rows and folds them in Rust rather than grouping in SQL: a grain carrying both the quarter hour and `response_ms` approaches a group per row, which would be a temp b-tree the size of the window. The single-purpose functions `/api/stats/*` uses are statements of their own — adding a seventh reading to the page means folding it out of one of those two scans, not adding a statement.
+- **One scan per index, not one per reading.** `stats_scan_since` and `traffic_lists_since` (`src/db.rs`) are the page's two reads — the second answering top domains, the distinct-domain count and top clients from one statement over the domain and client rollups — and every reading on it — the charts included — is folded out of one of them; `compute_range_stats` (`src/admin/stats.rs`) is what the page calls. `stats_scan_since` streams its rows and folds them in Rust rather than grouping in SQL: a grain carrying both the quarter hour and `response_ms` approaches a group per row, which would be a temp b-tree the size of the window. The single-purpose functions `/api/stats/*` uses are statements of their own — adding a seventh reading to the page means folding it out of one of those two scans, not adding a statement.
 - **A total nobody can count cheaply is maintained, not counted.** `query_logs`' row count lives in `settings` (`query_log_count`), moved by the insert batch, the prune and Clear All inside their own transactions — `SELECT COUNT(*)` walks an index end to end, and both the Database Health card and the query log's pager (whenever no filter is applied, via `count_logs`) ask on every load. A fourth write path to `query_logs` means a fourth `bump_log_count`, not a fourth reader.
 - **The same holds for the statistics rollups** (`query_stats_*`, see ARCHITECTURE.md *Rollups*), which must always equal a recount of `query_logs`. Inserts are covered by the `query_logs_maintain_stats` trigger whatever writes them; a new path that *deletes* from `query_logs` has to unwind them in its own transaction, as `prune_logs_before` (`unwind_stats_rollups`) and Clear All do.
 - Four bar lists in one template share **one askama macro** (`templates/_macros.html`); `{% call … %}` needs a matching `{% endcall %}` in askama 0.16, and `{% include %}` cannot see a loop variable at all.