Skip to content

perf(stats): fold the Statistics page's readings out of the statistics rollups - #282

Merged
henry40408 merged 1 commit into
mainfrom
perf/stats-rollups
Sep 14, 2026
Merged

henry40408 merged 1 commit into
mainfrom
perf/stats-rollups

Conversation

@henry40408

Copy link
Copy Markdown
Owner

Fifth of the rollup series (#278 benches, #279 log counter, #280 rollups, #281 dashboard).

What changes

  • stats_scan_since — the Statistics page's one read for its breakdowns, latency and chart series — is now a single four-arm statement: query_stats_quarter from the earlier window's first whole quarter, query_stats_metrics_hour from the range's first whole hour, and the query_logs rows each window starts inside (via idx_query_logs_timestamp). A leading column tells the arms apart, so each window keeps its edge to the millisecond.
  • API-only readers fold the same tables: window_metrics_since (breakdowns, highlights' latency), timeline_multi_since and hourly_heatmap_since (quarter rollup when bucket and offset are whole quarters; a table count otherwise).
  • /api/stats/v2/timeline and /api/stats/v2/heatmap round tz_offset to the nearest 15 minutes. Every zone in use already is; the rounding keeps every API call on the rollup path.
  • No statement names idx_query_logs_ts_metrics any more, which unblocks dropping it (next PR). Its plan and migration tests stay until then.

Page misses

stats_page_miss_bench, BENCH_NOW=1789212354, main (d801b9c) → this branch.

Reading 1.48 M rows (31d) 370 k rows (8d)
Statistics visit, 30d range 11 641 → 3 644 2 718 → 946
Statistics visit, 7d range 10 060 → 1 015 2 735 → 980
API timeline, 30d 9 476 → 45 2 153 → 13
API heatmap 6 030 → 38 1 387 → 13
API breakdowns, 30d 9 476 → 1 446 2 153 → 372

dashboard_page_miss_bench and logs_page_miss_bench are identical to main on both databases (tick 227 / 190, logs sum 368 963 / 100 029).

Tests

  • statistics_readings_equal_a_recount_of_the_table — scan, window metrics, timeline (5 buckets) and heatmap against a direct count of query_logs, for window starts before the data, on an hour, on a quarter, inside each and after; heatmap windows before/equal/after the range; offsets 0, +8h, −5h, +5:30, +5:45, −9:30 and an unaligned 7 min. Breaking each table arm in turn (scan, window, timeline, heatmap) makes it fail.
  • the_statistics_readings_fold_rollups_rather_than_the_table — fails on main (stats scan 107 pages vs 168 for a full scan).
  • stats_v2_tz_offset_is_rounded_to_a_quarter_hour — fails with the rounding removed.
  • Removed the_page_and_its_charts_are_one_metrics_scan and the_heatmap_reads_the_narrowest_index_that_covers_it, which asserted index choices no reader makes now.

🤖 Generated with Claude Code

…s rollups

stats_scan_since reads query_stats_quarter and query_stats_metrics_hour,
plus the table rows each window starts inside, in one four-arm statement.
The API-only readers follow: window_metrics_since folds the metrics
rollup, and timeline_multi_since and hourly_heatmap_since fold the
quarter rollup whenever the bucket and offset are whole quarters. The
API rounds tz_offset to the nearest quarter hour so it always is.

No reader names idx_query_logs_ts_metrics any more.

Page misses on a 1.48 M-row database (main -> this):
  Statistics 30d visit  11 641 -> 3 644
  Statistics 7d visit   10 060 -> 1 015
  API timeline 30d       9 476 ->    45
  API heatmap            6 030 ->    38
Dashboard and query log readings unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.85039% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.36%. Comparing base (d801b9c) to head (efee13b).

Files with missing lines Patch % Lines
src/db.rs 96.80% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #282      +/-   ##
==========================================
+ Coverage   91.32%   91.36%   +0.03%     
==========================================
  Files          31       31              
  Lines       11625    11691      +66     
==========================================
+ Hits        10617    10681      +64     
- Misses       1008     1010       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@henry40408
henry40408 merged commit 38276a6 into main Sep 14, 2026
7 checks passed
@henry40408
henry40408 deleted the perf/stats-rollups branch September 14, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant