Skip to content

perf(stats): answer the Statistics page in one metrics scan and a maintained row count - #274

Merged
henry40408 merged 2 commits into
mainfrom
perf/stats-one-metrics-scan
Sep 12, 2026
Merged

henry40408 merged 2 commits into
mainfrom
perf/stats-one-metrics-scan

Conversation

@henry40408

@henry40408 henry40408 commented Sep 12, 2026

Copy link
Copy Markdown
Owner

Two independent reductions in what a Statistics page load fetches from the database file.

One scan of the metrics index, not two

The outcome breakdown, the query-type breakdown and the latency percentiles are three foldings of one window of idx_query_logs_ts_metrics, and were asked as two statements: one bucketed by time, one by (query_type, response_ms). Each scanned the index end to end, and the read pool spreads them over connections holding 2 MiB of page cache each, so nothing was warm for the second.

The bucket was never needed. The page renders no timeline — that chart is the client's — and the outcome breakdown sums across every bucket anyway, so bucketing only multiplied the rows the folds read: 68 846 at the 7-day range's hourly grain against 4 658 without it. window_metrics_since groups at (blocked, cached, has_result, query_type, response_ms) and all three readings fall out of it. compute_breakdowns, behind /api/stats/breakdowns, shared the same split and is now one call.

The heatmap gets the opposite hint: it reads timestamp and nothing else, but the planner reached for the metrics index rather than the smallest one that covers it.

A maintained row count, not COUNT(*)

The Database Health card prints how many rows query_logs holds and divides two of its estimates by it. SELECT COUNT(*) has no shortcut in SQLite — it walks the smallest index end to end.

The count lives in settings under query_log_count, seeded by the version-13 migration and moved by the three statements that change how many rows the table holds: the logger's insert batch, the hourly prune, and Clear All. Each moves it inside its own transaction, so the counter cannot report a total the table stopped holding; a database with no counter row counts, which is what the migration seeded it out of. No new table, and nothing enumerates settings keys, so the row is invisible to the settings page.

Measured

370 k-row, 147 MiB database, 7-day window (stats_page_miss_bench):

reading before after
range_stats (breakdowns+latency+domains) 8147 6042
top_clients 3618 3618
db_health 1398 14
timeline (client fetch) 2087 2069
heatmap (client fetch) 2153 1387
breakdowns alone 4174 2069
PAGE TOTAL (first response) 13163 9674

51.4 MiB to 37.8 MiB. What is left is one scan each of idx_query_logs_ts_metrics, idx_query_logs_domain_ts and idx_query_logs_client_ts — the floor for this storage model. Going below it means hourly rollup tables, which is not this PR.

Tests

  • the_window_readings_are_one_scan_between_them, the_heatmap_reads_the_narrowest_index_that_covers_it, the_total_log_count_is_read_rather_than_counted (tests/stats_page_miss_test.rs)
  • the_log_count_follows_every_write_that_changes_it (tests/db_test.rs) — insert, prune, no-op prune, clear
  • migration_v13_seeds_the_log_count_from_the_table (src/db.rs)

cargo nextest run: 670 passed. No UI change, so no screenshots.

🤖 Generated with Claude Code

The outcome breakdown, the query-type breakdown and the latency
percentiles are three foldings of one window of idx_query_logs_ts_metrics,
and were asked as two statements: one bucketed by time, one by
(query_type, response_ms). Each scanned the index end to end, and the read
pool spreads them over connections with 2 MiB of page cache each, so
nothing was warm for the second.

The bucket was never needed. The page renders no timeline — that chart is
the client's — and the outcome breakdown sums across every bucket anyway,
so bucketing only multiplied the rows the folds read: 68 846 at the 7-day
range's hourly grain against 4 658 without it. window_metrics_since groups
at (blocked, cached, has_result, query_type, response_ms) and all three
readings fall out of it.

The heatmap gets the opposite hint. It reads timestamp and nothing else,
but the planner reached for the metrics index rather than the smallest one
that covers it.

Measured on a 370 k-row, 147 MiB database over a 7-day window:

  reading                              before     after
  range_stats                            8147      6053
  heatmap (client fetch)                 2153      1387
  breakdowns alone                       4174      2080
  PAGE TOTAL (first response)           13163     11069

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.22222% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.18%. Comparing base (c5d388a) to head (d720340).

Files with missing lines Patch % Lines
src/db.rs 97.08% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #274      +/-   ##
==========================================
+ Coverage   91.16%   91.18%   +0.02%     
==========================================
  Files          31       31              
  Lines       11168    11224      +56     
==========================================
+ Hits        10181    10235      +54     
- Misses        987      989       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

The Database Health card prints how many rows query_logs holds and divides
two of its estimates by it. SELECT COUNT(*) has no shortcut in SQLite: it
walks the smallest index end to end, 1 386 pages on a 370 k-row database,
on every Statistics page load.

The count lives in settings under query_log_count now, seeded once by the
version-13 migration and moved by the three statements that change how many
rows the table holds — the logger's insert batch, the hourly prune, and
Clear All. Each moves it inside its own transaction, so the counter cannot
report a total the table stopped holding. A database with no counter row
counts, which is what the migration seeded it out of.

settings needs no new table and nothing enumerates its keys, so the row is
invisible to the settings page.

Measured on a 370 k-row, 147 MiB database over a 7-day window:

  reading                              before     after
  db_health                              1398        14
  PAGE TOTAL (first response)           11069      9674

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@henry40408 henry40408 changed the title perf(stats): answer the Statistics page's metrics index in one scan perf(stats): answer the Statistics page in one metrics scan and a maintained row count Sep 12, 2026
@henry40408
henry40408 merged commit 94737c8 into main Sep 12, 2026
7 checks passed
@henry40408
henry40408 deleted the perf/stats-one-metrics-scan branch September 12, 2026 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant