perf(stats): answer the Statistics page in one metrics scan and a maintained row count - #274
Merged
Merged
Conversation
The outcome breakdown, the query-type breakdown and the latency percentiles are three foldings of one window of idx_query_logs_ts_metrics, and were asked as two statements: one bucketed by time, one by (query_type, response_ms). Each scanned the index end to end, and the read pool spreads them over connections with 2 MiB of page cache each, so nothing was warm for the second. The bucket was never needed. The page renders no timeline — that chart is the client's — and the outcome breakdown sums across every bucket anyway, so bucketing only multiplied the rows the folds read: 68 846 at the 7-day range's hourly grain against 4 658 without it. window_metrics_since groups at (blocked, cached, has_result, query_type, response_ms) and all three readings fall out of it. The heatmap gets the opposite hint. It reads timestamp and nothing else, but the planner reached for the metrics index rather than the smallest one that covers it. Measured on a 370 k-row, 147 MiB database over a 7-day window: reading before after range_stats 8147 6053 heatmap (client fetch) 2153 1387 breakdowns alone 4174 2080 PAGE TOTAL (first response) 13163 11069 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #274 +/- ##
==========================================
+ Coverage 91.16% 91.18% +0.02%
==========================================
Files 31 31
Lines 11168 11224 +56
==========================================
+ Hits 10181 10235 +54
- Misses 987 989 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The Database Health card prints how many rows query_logs holds and divides two of its estimates by it. SELECT COUNT(*) has no shortcut in SQLite: it walks the smallest index end to end, 1 386 pages on a 370 k-row database, on every Statistics page load. The count lives in settings under query_log_count now, seeded once by the version-13 migration and moved by the three statements that change how many rows the table holds — the logger's insert batch, the hourly prune, and Clear All. Each moves it inside its own transaction, so the counter cannot report a total the table stopped holding. A database with no counter row counts, which is what the migration seeded it out of. settings needs no new table and nothing enumerates its keys, so the row is invisible to the settings page. Measured on a 370 k-row, 147 MiB database over a 7-day window: reading before after db_health 1398 14 PAGE TOTAL (first response) 11069 9674 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent reductions in what a Statistics page load fetches from the database file.
One scan of the metrics index, not two
The outcome breakdown, the query-type breakdown and the latency percentiles are three foldings of one window of
idx_query_logs_ts_metrics, and were asked as two statements: one bucketed by time, one by(query_type, response_ms). Each scanned the index end to end, and the read pool spreads them over connections holding 2 MiB of page cache each, so nothing was warm for the second.The bucket was never needed. The page renders no timeline — that chart is the client's — and the outcome breakdown sums across every bucket anyway, so bucketing only multiplied the rows the folds read: 68 846 at the 7-day range's hourly grain against 4 658 without it.
window_metrics_sincegroups at(blocked, cached, has_result, query_type, response_ms)and all three readings fall out of it.compute_breakdowns, behind/api/stats/breakdowns, shared the same split and is now one call.The heatmap gets the opposite hint: it reads
timestampand nothing else, but the planner reached for the metrics index rather than the smallest one that covers it.A maintained row count, not
COUNT(*)The Database Health card prints how many rows
query_logsholds and divides two of its estimates by it.SELECT COUNT(*)has no shortcut in SQLite — it walks the smallest index end to end.The count lives in
settingsunderquery_log_count, seeded by the version-13 migration and moved by the three statements that change how many rows the table holds: the logger's insert batch, the hourly prune, and Clear All. Each moves it inside its own transaction, so the counter cannot report a total the table stopped holding; a database with no counter row counts, which is what the migration seeded it out of. No new table, and nothing enumeratessettingskeys, so the row is invisible to the settings page.Measured
370 k-row, 147 MiB database, 7-day window (
stats_page_miss_bench):51.4 MiB to 37.8 MiB. What is left is one scan each of
idx_query_logs_ts_metrics,idx_query_logs_domain_tsandidx_query_logs_client_ts— the floor for this storage model. Going below it means hourly rollup tables, which is not this PR.Tests
the_window_readings_are_one_scan_between_them,the_heatmap_reads_the_narrowest_index_that_covers_it,the_total_log_count_is_read_rather_than_counted(tests/stats_page_miss_test.rs)the_log_count_follows_every_write_that_changes_it(tests/db_test.rs) — insert, prune, no-op prune, clearmigration_v13_seeds_the_log_count_from_the_table(src/db.rs)cargo nextest run: 670 passed. No UI change, so no screenshots.🤖 Generated with Claude Code