Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,13 @@ That distinction is not hypothetical. `outcome_breakdown_since` was left uncover

### One scan per index, not one per reading

The Statistics page's five readings were four foldings of the metrics index over one window and two foldings of `(domain, timestamp)` over the same one, asked as six separate statements. Each re-walked an index another had just finished with, and the read pool round-robins them across connections holding 2 MiB of page cache each, so nothing was ever warm for the next. `range_metrics_since` groups at `(bucket, blocked, cached, has_result)` and `(query_type, response_ms)` — two statements the timeline, both breakdowns and the latency histogram are derived from — and `domain_stats_since` returns the top list and the distinct count from one materialized CTE. The single-purpose functions `/api/stats/*` calls are folds over the same statements, so there is one SQL spelling per fact. Rendering a 7-day window on that 370 k-row database went from 29 274 page misses (114 MiB) to 13 291 (52 MiB).
The Statistics page's five readings were four foldings of the metrics index over one window and two foldings of `(domain, timestamp)` over the same one, asked as six separate statements. Each re-walked an index another had just finished with, and the read pool round-robins them across connections holding 2 MiB of page cache each, so nothing was ever warm for the next. `window_metrics_since` groups at `(blocked, cached, has_result, query_type, response_ms)` — one statement both breakdowns and the latency histogram are derived from — and `domain_stats_since` returns the top list and the distinct count from one materialized CTE. The single-purpose functions `/api/stats/*` calls are folds over the same statements, so there is one SQL spelling per fact. Rendering a 7-day window on that 370 k-row database went from 29 274 page misses (114 MiB) to 9 674 (38 MiB).

The window has no time bucket in it because the page draws no timeline — that chart is the client's, and `timeline_multi_since` is its own scan. Bucketing the shared grain was what the first version did, and it cost a scan: the outcome breakdown sums across every bucket, so the bucket only multiplied the rows the folds read, 68 846 of them against 4 658 at the 7-day range's hourly grain, while the query-type and latency folds still needed a second statement of their own. Collapsing the two into one grain fine enough for all three is 2 157 pages against 4 314.

The Database Health card's row count is the one reading that is not a scan of anything. `SELECT COUNT(*)` has no shortcut in SQLite — it walks the smallest index end to end, 1 386 pages on that database, for a number the card prints and two of its estimates divide by — so the count lives in `settings` under `query_log_count`, seeded by the version-13 migration and moved by the three statements that change how many rows `query_logs` holds: the logger's insert batch, the hourly prune, and Clear All. Each moves it inside its own transaction, which is what makes the counter unable to disagree with the table; `total_log_count` falls back to counting when the row is missing, which is the state the migration seeds it out of. The card went from 1 398 pages to 14.

`INDEXED BY` appears on every statement that reads this index, in both directions. The two that need `blocked`, `cached` or `has_result` name `idx_query_logs_ts_metrics` because the planner otherwise takes the smaller `idx_query_logs_timestamp` and pays a rowid lookup per row; the heatmap, which reads `timestamp` and nothing else, names `idx_query_logs_timestamp` for the opposite reason — left alone the planner took the metrics index and read 2 153 pages where 1 386 answer it.

Every index migration runs `ANALYZE`. A new index alone is not always enough — the planner keeps its old plan until `sqlite_stat1` is refreshed — and the hourly `PRAGMA optimize` lets those statistics drift a long way in the meantime.

Expand Down
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,8 @@ Statistics adds the conventions for a page whose readings sit in a **chosen wind
- **The range is in the URL and the switcher is three `<a>`s** (`/stats?range=30d`), because the range picks the *server's* window. `StatsRange::label()` is the one spelling shared by the link, the parse and every card title. An unrecognised range renders the default rather than 400ing — it is a link an operator can edit, and every window on offer is spelled out right above it.
- **A date the server can only write in UTC ships as an ISO day plus its timestamp** (`data-date-ts`), and `app.js` restates it in the browser's locale — the same division as the query log's relative times, and for the same reason.
- **This page is measured in page misses, not milliseconds.** Development is on an SSD and the appliance runs off an SD card, so a duration measured here says nothing about a Raspberry Pi; the pages a query fetches from the file are the same on both. `cargo nextest run --release --no-capture --run-ignored only stats_page_miss` with `BENCH_DB` pointed at a copy of a real database reports them. A wall-clock reading is the thing to distrust when the two disagree — it is what left the outcome breakdown scanning the whole table through version 11.
- **One scan per index, not one per reading.** `range_metrics_since` and `domain_stats_since` (`src/db.rs`) group at a grain every reading on the page can be folded out of; `compute_range_stats` (`src/admin/stats.rs`) is what the page calls. The single-purpose functions `/api/stats/*` uses are folds over the same statements — adding a seventh reading means folding it out of one of those two, not adding a statement.
- **One scan per index, not one per reading.** `window_metrics_since` and `domain_stats_since` (`src/db.rs`) group at a grain every reading on the page can be folded out of; `compute_range_stats` (`src/admin/stats.rs`) is what the page calls. The single-purpose functions `/api/stats/*` uses are folds over the same statements — adding a seventh reading means folding it out of one of those two, not adding a statement. The window carries no time bucket: the page renders no timeline, and bucketing multiplied the rows the folds read by fifteen for an answer summed across every bucket anyway.
- **A total nobody can count cheaply is maintained, not counted.** `query_logs`' row count lives in `settings` (`query_log_count`), moved by the insert batch, the prune and Clear All inside their own transactions — `SELECT COUNT(*)` walks an index end to end and the Database Health card asks on every load. A fourth write path to `query_logs` means a fourth `bump_log_count`, not a fourth reader.
- Four bar lists in one template share **one askama macro** (`templates/_macros.html`); `{% call … %}` needs a matching `{% endcall %}` in askama 0.16, and `{% include %}` cannot see a loop variable at all.

Account adds the conventions for **actions that need a password proof**:
Expand Down
32 changes: 14 additions & 18 deletions src/admin/stats.rs
Original file line number Diff line number Diff line change
Expand Up @@ -301,13 +301,12 @@ pub async fn compute_breakdowns(
) -> Result<Breakdowns, DbError> {
let (window_secs, _) = range.window();
let since = now - window_secs;
let (query_types, outcomes) = tokio::try_join!(
db.query_type_breakdown_since(since),
db.outcome_breakdown_since(since),
)?;
// Both breakdowns fold out of one statement; asking for them separately is
// two scans of the index that answers either.
let metrics = db.window_metrics_since(since).await?;
Ok(Breakdowns {
query_types,
outcomes,
query_types: metrics.query_types,
outcomes: metrics.outcomes,
})
}

Expand Down Expand Up @@ -337,15 +336,15 @@ pub async fn compute_highlights(
/// Everything the Statistics page reads out of `query_logs` for its window, in
/// the fewest scans the indexes allow.
///
/// The page used to ask for the four readings separately — a query-type
/// breakdown, an outcome breakdown, a latency summary, a unique-domain count
/// alongside the top-domain list, and every one of them re-scanned an index
/// another had just walked. Two of them share
/// [`crate::db::Database::range_metrics_since`] and two share
/// [`crate::db::Database::domain_stats_since`], which is three index scans
/// The page used to ask for the five readings separately — a query-type
/// breakdown, an outcome breakdown, a latency summary, a unique-domain count,
/// a top-domain listand every one of them re-scanned an index another had
/// just walked. Three of them share
/// [`crate::db::Database::window_metrics_since`] and two share
/// [`crate::db::Database::domain_stats_since`], which is two index scans
/// instead of six.
pub struct RangeStats {
pub metrics: crate::db::RangeMetrics,
pub metrics: crate::db::WindowMetrics,
pub domains: crate::db::DomainStats,
}

Expand All @@ -355,13 +354,10 @@ pub async fn compute_range_stats(
range: StatsRange,
top_n: i64,
) -> Result<RangeStats, DbError> {
let (window_secs, bucket_secs) = range.window();
let (window_secs, _) = range.window();
let since = now - window_secs;
// The page renders no timeline — the chart is the client's — so the bucket
// width only bounds how many rows the outcome fold reads. Passing the
// range's own keeps it to one statement shared with the API endpoint.
let (metrics, domains) = tokio::try_join!(
db.range_metrics_since(since, bucket_secs, 0),
db.window_metrics_since(since),
db.domain_stats_since(since, top_n),
)?;
Ok(RangeStats { metrics, domains })
Expand Down
Loading