Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,8 @@ A reader takes every unit from the first whole one inside its window (`first_who

The dashboard reads nothing else: `summary_multi_since`, `timeline_since`, `traffic_lists_since` and `top_upstreams_since` all fold rollups, and so does `domain_stats_since`, which answers the domain suggestions. On the 1.48 M-row database a dashboard tick went from 10 774 page misses to 227, the first response from 10 444 to 215, and the domain suggestions from 6 192 to 559; a Statistics visit over 30 days went from 31 883 to 11 641, the rest of it being `stats_scan_since`. `dashboard_readings_equal_a_recount_of_the_table` (`tests/stats_db_test.rs`) holds every one of these to the statement it replaced, run on the same rows, for windows starting before the data, on an hour, on a quarter, inside each, and after it.

The Statistics page followed. `stats_scan_since` is one statement of four arms — `query_stats_quarter` from the earlier window's first whole quarter, `query_stats_metrics_hour` from the range's first whole hour, and the table rows each window starts inside — told apart by a leading column, so a table row both windows start inside is counted once per window. The API-only readers fold the same tables: `window_metrics_since` the metrics rollup, and `timeline_multi_since` and `hourly_heatmap_since` the quarter rollup whenever the bucket and the offset are whole quarters. The API rounds `tz_offset` to the nearest quarter hour for that reason; an offset no zone uses would otherwise count every row in the window, and after the metrics index goes that means the table. On the 1.48 M-row database a 30-day visit went from 11 641 page misses to 3 644 and a 7-day visit from 10 060 to 1 015; the API's timeline from 9 476 to 45 and its heatmap from 6 030 to 38. `statistics_readings_equal_a_recount_of_the_table` is the guard, over the same window starts, heatmap windows before, equal to and after the range's, and offsets including India, Nepal and a seven-minute one that takes the table path.

### Measuring these queries

Index work here is measured in **page misses** — `SQLITE_DBSTATUS_CACHE_MISS`, the 4 KiB database pages SQLite has to fetch from the file — and not in milliseconds. Development happens on an SSD and the appliance runs off an SD card, where the same page count costs orders of magnitude more; a query that reads the whole table can look free on one machine and take seconds on the other. `tests/stats_page_miss_bench.rs` reports the figure against a real database and `tests/stats_page_miss_test.rs` asserts the properties behind it; `tests/dashboard_page_miss_bench.rs` does the same for the dashboard's first response and its recurring event-stream tick, and `tests/logs_page_miss_bench.rs` for every filter the query log offers, with values drawn from the database under test. All three take `BENCH_DB` and a `BENCH_NOW` that pins the clock, so a copy older than its windows still measures the traffic it holds; `tests/stats_parallel_bench.rs` is the wall-clock companion, and is the one to distrust when the two disagree.
Expand All @@ -291,13 +293,13 @@ The Statistics page's five readings were four foldings of the metrics index over

The window's grain has no time bucket in it. Bucketing the shared grain was what the first version did, and it cost a scan: the outcome breakdown sums across every bucket, so the bucket only multiplied the rows the folds read, 68 846 of them against 4 658 at the 7-day range's hourly grain, while the query-type and latency folds still needed a second statement of their own. Collapsing the two into one grain fine enough for all three is 2 157 pages against 4 314.

The charts did still pay for scans of their own after that: the browser fetched the timeline, which walked the metrics index again, and the heatmap, which walked `idx_query_logs_timestamp`. They now come out of the page's scan too. `stats_scan_since` reads the metrics index once from the earlier of the range's and the heatmap's windows, folds the window readings, and counts queries per UTC quarter hour into a `QuarterSeries` the page embeds; `app.js` folds that into the viewer's hours and days, which is exact because every UTC offset in use is a whole number of quarter hours. It streams rows into Rust rather than grouping in SQL, because a grain carrying both the quarter and `response_ms` approaches one group per row. On the same database with the 30-day range — the whole table — a visit went from 13 298 page misses (51.9 MiB) to 9 758 (38.1 MiB). The API's timeline and heatmap endpoints keep their own statements, and `e2e/tests/specs/stats_charts.rs` holds the browser's folds to them.
The charts did still pay for scans of their own after that: the browser fetched the timeline, which walked the metrics index again, and the heatmap, which walked `idx_query_logs_timestamp`. They now come out of the page's scan too. `stats_scan_since` reads the metrics index once from the earlier of the range's and the heatmap's windows, folds the window readings, and counts queries per UTC quarter hour into a `QuarterSeries` the page embeds; `app.js` folds that into the viewer's hours and days, which is exact because every UTC offset in use is a whole number of quarter hours. It streams rows into Rust rather than grouping in SQL, because a grain carrying both the quarter and `response_ms` approaches one group per row. On the same database with the 30-day range — the whole table — a visit went from 13 298 page misses (51.9 MiB) to 9 758 (38.1 MiB). The API's timeline and heatmap endpoints keep their own statements, and `e2e/tests/specs/stats_charts.rs` holds the browser's folds to them. The scan and both endpoints have since moved onto the rollups (see *Rollups*).

The Database Health card's row count is the one reading that is not a scan of anything. `SELECT COUNT(*)` has no shortcut in SQLite — it walks the smallest index end to end, 1 386 pages on that database, for a number the card prints and two of its estimates divide by — so the count lives in `settings` under `query_log_count`, seeded by the version-13 migration and moved by the three statements that change how many rows `query_logs` holds: the logger's insert batch, the hourly prune, and Clear All. Each moves it inside its own transaction, which is what makes the counter unable to disagree with the table; `total_log_count` falls back to counting when the row is missing, which is the state the migration seeds it out of. The card went from 1 398 pages to 14. The query log's pager asks for the same number whenever no filter narrows it, so `count_logs` reads the counter in that case through the same `read_log_count`, and only counts once a filter is applied.

The dashboard pays for its readings every 10 seconds rather than once a visit, which makes a scan it repeats the most expensive kind. Its summary asked two statements for totals and blocks, then cache hits and latency, over the same 30 days of the metrics index; `summary_multi_since` moves the allowed-only filter from the `WHERE` into each `CASE` and answers both from one scan, 2 152 pages a tick instead of 4 304. With the upstream index, a tick on that database dropped from 8 304 page misses to 4 755. Those readings have since moved onto the rollups (see *Rollups*), which no longer read either index.

`INDEXED BY` appears on every statement that reads this index, in both directions. The ones that need `blocked`, `cached` or `has_result` name `idx_query_logs_ts_metrics` because the planner otherwise takes the smaller `idx_query_logs_timestamp` and pays a rowid lookup per row; the rollup readers' table arms name `idx_query_logs_timestamp`, where that lookup is the point — they read at most one unit of rows; the heatmap, which reads `timestamp` and nothing else, names `idx_query_logs_timestamp` for the opposite reason — left alone the planner took the metrics index and read 2 153 pages where 1 386 answer it.
`INDEXED BY` appears on every statement that reads this index, in both directions. The ones that needed `blocked`, `cached` or `has_result` named `idx_query_logs_ts_metrics` because the planner otherwise takes the smaller `idx_query_logs_timestamp` and pays a rowid lookup per row — none is left, every such reading having moved onto the rollups; the rollup readers' table arms name `idx_query_logs_timestamp`, where that lookup is the point — they read at most one unit of rows; the heatmap, which reads `timestamp` and nothing else, names `idx_query_logs_timestamp` for the opposite reason — left alone the planner took the metrics index and read 2 153 pages where 1 386 answer it.

Every index migration runs `ANALYZE`. A new index alone is not always enough — the planner keeps its old plan until `sqlite_stat1` is refreshed — and the hourly `PRAGMA optimize` lets those statistics drift a long way in the meantime.

Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,12 +128,12 @@ Dashboard adds the conventions for a page that is **all readings and no controls
Statistics adds the conventions for a page whose readings sit in a **chosen window**, and the rule for **what the server cannot render**:

- **Let the data draw the line, not the page.** Only the three charts need a calendar, and a calendar-aligned bucket needs the viewer's UTC offset, which arrives with the browser and not with the request. Everything else is a plain `now - range` window with no calendar in it, so it renders on the server and never moves again: the highlights, both breakdowns, both ranged lists and the health grid are all in the first response.
- **The charts' data is in the first response too, just not in the viewer's calendar.** `<stats-page data-series>` is a `QuarterSeries` (`src/db.rs`) — query counts per quarter hour on UTC boundaries, out of the same scan as the breakdowns — and `timelineFromQuarters` / `heatmapFromQuarters` in `app.js` fold it into the browser's hours and days. That is exact, not approximate: every offset in use is a whole number of quarter hours, so no quarter straddles a local hour. The page makes no request for its charts. `/api/stats/v2/timeline` and `…/heatmap` still take `tz_offset` for API callers, and `e2e/tests/specs/stats_charts.rs` holds the JavaScript folds to them across ranges and offsets — change one and that is what fails.
- **The charts' data is in the first response too, just not in the viewer's calendar.** `<stats-page data-series>` is a `QuarterSeries` (`src/db.rs`) — query counts per quarter hour on UTC boundaries, out of the same scan as the breakdowns — and `timelineFromQuarters` / `heatmapFromQuarters` in `app.js` fold it into the browser's hours and days. That is exact, not approximate: every offset in use is a whole number of quarter hours, so no quarter straddles a local hour. The page makes no request for its charts. `/api/stats/v2/timeline` and `…/heatmap` still take `tz_offset` for API callers — rounded to the nearest quarter hour, the grain of `query_stats_quarter` they fold, which no zone in use notices — and `e2e/tests/specs/stats_charts.rs` holds the JavaScript folds to them across ranges and offsets — change one and that is what fails.
- **`app.js` does not redraw what it did not need to draw.** `StatsPage` is three charts and one date; the bar-list, health-grid and highlights renderers are gone rather than kept as a second copy of the markup. There is no polling here — this page is history, not a live reading.
- **The range is in the URL and the switcher is three `<a>`s** (`/stats?range=30d`), because the range picks the *server's* window. `StatsRange::label()` is the one spelling shared by the link, the parse and every card title. An unrecognised range renders the default rather than 400ing — it is a link an operator can edit, and every window on offer is spelled out right above it.
- **A date the server can only write in UTC ships as an ISO day plus its timestamp** (`data-date-ts`), and `app.js` restates it in the browser's locale — the same division as the query log's relative times, and for the same reason.
- **This page is measured in page misses, not milliseconds.** Development is on an SSD and the appliance runs off an SD card, so a duration measured here says nothing about a Raspberry Pi; the pages a query fetches from the file are the same on both. `cargo nextest run --release --no-capture --run-ignored only stats_page_miss` with `BENCH_DB` pointed at a copy of a real database reports them; `dashboard_page_miss` and `logs_page_miss` do the same for the dashboard (first response and tick) and the query log (every filter), and `BENCH_NOW` pins the clock on a copy older than its windows. A wall-clock reading is the thing to distrust when the two disagree — it is what left the outcome breakdown scanning the whole table through version 11.
- **One scan per index, not one per reading.** `stats_scan_since` and `traffic_lists_since` (`src/db.rs`) are the page's two reads — the second answering top domains, the distinct-domain count and top clients from one statement over the domain and client rollups — and every reading on it — the charts included — is folded out of one of them; `compute_range_stats` (`src/admin/stats.rs`) is what the page calls. `stats_scan_since` streams its rows and folds them in Rust rather than grouping in SQL: a grain carrying both the quarter hour and `response_ms` approaches a group per row, which would be a temp b-tree the size of the window. The single-purpose functions `/api/stats/*` uses are statements of their own — adding a seventh reading to the page means folding it out of one of those two scans, not adding a statement.
- **One statement per rollup family, not one per reading.** `stats_scan_since` and `traffic_lists_since` (`src/db.rs`) are the page's two reads — the first folding the quarter and metrics rollups, the second answering top domains, the distinct-domain count and top clients from one statement over the domain and client rollups — and every reading on it — the charts included — is folded out of one of them; `compute_range_stats` (`src/admin/stats.rs`) is what the page calls. `stats_scan_since` streams its rows and folds them in Rust rather than grouping in SQL, telling its four arms (two rollups, then the table rows each window starts inside) apart by a leading column. The single-purpose functions `/api/stats/*` uses are statements of their own over the same rollups — adding a seventh reading to the page means folding it out of one of those two statements, not adding one.
- **A total nobody can count cheaply is maintained, not counted.** `query_logs`' row count lives in `settings` (`query_log_count`), moved by the insert batch, the prune and Clear All inside their own transactions — `SELECT COUNT(*)` walks an index end to end, and both the Database Health card and the query log's pager (whenever no filter is applied, via `count_logs`) ask on every load. A fourth write path to `query_logs` means a fourth `bump_log_count`, not a fourth reader.
- **The same holds for the statistics rollups** (`query_stats_*`, see ARCHITECTURE.md *Rollups*), which must always equal a recount of `query_logs`. Inserts are covered by the `query_logs_maintain_stats` trigger whatever writes them; a new path that *deletes* from `query_logs` has to unwind them in its own transaction, as `prune_logs_before` (`unwind_stats_rollups`) and Clear All do.
- Four bar lists in one template share **one askama macro** (`templates/_macros.html`); `{% call … %}` needs a matching `{% endcall %}` in askama 0.16, and `{% include %}` cannot see a loop variable at all.
Expand Down
10 changes: 8 additions & 2 deletions src/admin/api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3683,7 +3683,8 @@ async fn get_stats_top_upstreams(
pub struct TimelineV2Query {
pub range: Option<String>,
/// Viewer's east-positive UTC offset in minutes (e.g. 480 for UTC+8), used
/// to align buckets to their local calendar. Clamped to ±14h; missing ⇒ 0
/// to align buckets to their local calendar. Clamped to ±14h and rounded to
/// the nearest 15 minutes, which every zone in use already is; missing ⇒ 0
/// (UTC-aligned).
pub tz_offset: Option<i64>,
}
Expand Down Expand Up @@ -3712,8 +3713,13 @@ fn parse_stats_range(raw: Option<&str>) -> Result<stats::StatsRange, StatusCode>

/// Resolve the viewer's UTC offset to seconds, clamped to the real-world range
/// (±14h) so a malformed value can't shift buckets to nonsense.
///
/// Rounded to a quarter hour because that is the grain of
/// `query_stats_quarter`: a quarter-aligned offset is answered from the rollup,
/// any other would count every row in the window. No zone in use is affected.
fn resolve_tz_offset_secs(tz_offset: Option<i64>) -> i64 {
tz_offset.unwrap_or(0).clamp(-14 * 60, 14 * 60) * 60
let minutes = tz_offset.unwrap_or(0).clamp(-14 * 60, 14 * 60);
(minutes + 7).div_euclid(15) * 15 * 60
}

async fn get_stats_v2_timeline(
Expand Down
Loading