perf(db): index the query log's filters instead of the statistics scans

[henry40408]

Last of the rollup series (#278 benches, #279 log counter, #280 rollups, #281 dashboard, #282 Statistics).

What changes

Schema version 17:

• Drops idx_query_logs_ts_domain_client, idx_query_logs_ts_metrics, idx_query_logs_ts_upstream. Since perf(stats): fold the dashboard's readings out of the statistics rollups #281/perf(stats): fold the Statistics page's readings out of the statistics rollups #282 no statistic reads them.
• Adds one index per query log filter: (doh_token, timestamp), (blocked, timestamp), (query_type, blocked, timestamp). ANALYZE afterwards.

query_logs with a query type as the only filter reads two runs of the type index, blocked = 0 and blocked = 1. Each run is already newest first, carries only id, timestamp and is cut at offset + limit. The runs are merged, and the table is read for the page's rows only. A negative limit/offset from /api/logs follows SQLite's rules.

Page misses

logs_page_miss_bench / dashboard_page_miss_bench / stats_page_miss_bench, BENCH_NOW=1789212354, main (38276a6) → this branch.

Reading	1.48 M rows (31d)	370 k rows (8d)
Query log, sum of 42 readings	368 963 → 159 649	100 029 → 44 968
token (quiet): page 1 / count	1 200 / 23 130 → 28 / 15	1 198 / 5 770 → 31 / 6
type (quiet): page 1 / count	9 812 / 9 797 → 19 / 4	2 158 / 2 153 → 9 / 4
token + type (both quiet): page 1	23 604 → 19	5 897 → 9
blocked: page 20 / count	33 / 9 797 → 18 / 1 277	29 / 2 153 → 16 / 321
type (busy): page 20	30 → 22	27 → 22
type (busy): page 1	12 → 14	12 → 14
Dashboard tick	227 → 227	190 → 190 *
Statistics visit, 30d / 7d	3 644 / 1 015 → same	946 / 980 → same

type (busy) page 1 is the only reading that got worse: it reads two runs where the metrics index read one. This was reviewed and accepted.

* On the 8d database the tick measured 196 when the bench ran straight after the app migrated in place, with the migration's writes still in the WAL. With v17 applied and checkpointed it measures 190, identical to main, and 184 after VACUUM.

Without (blocked, timestamp), blocked/allowed page 20 regressed to 102/57, because the metrics index had been answering that filter without a table lookup. The index was added after measuring both options.

Cost

• Writes (1.48 M queries replayed in 500-row batches with the rollup trigger, sqlite3 CLI): 885 858 → 957 160 pages written (+8%).
• File size (same replay): 111 291 → 92 224 pages.
• Migration (sqlite3 CLI 3.51.0, SSD): 3.0 s on the 31d database, 0.5 s on the 8d. The 31d database ends with a 25% freelist, past VACUUM_FREELIST_RATIO, so the first hourly maintenance runs one VACUUM (119 603 → 86 876 pages). The 8d database stays at 15%. The migration still needs timing on an appliance.

Tests

• every_database_opens_to_the_same_query_log_indexes (src/db.rs) opens v9, v10, v11, v16 and fresh databases and requires the same five indexes. It then runs every reader that once named a dropped index with INDEXED BY. It fails with the v17 step disabled. It replaces six tests that asserted the dropped indexes exist.
• a_query_type_filter_pages_exactly_like_the_table (tests/db_test.rs) checks the two-run merge against the plain statement for 10 page sizes/offsets, including negative ones. It fails if a run is cut at limit instead of offset + limit, or if the negative-limit guard is removed.
• the_query_log_filters_seek_their_indexes (tests/stats_page_miss_test.rs) covers token page and count, type page and count, blocked page 20, and type + blocked. Each must read under a tenth of the file. It fails on main (token page 290 of 2 160 pages), and fails when the blocked index can't serve the filter (568 of 2 018).

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 98.51852% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.32%. Comparing base (38276a6) to head (720f6fb).

Files with missing lines	Patch %	Lines
src/db.rs	98.51%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #283      +/-   ##
==========================================
- Coverage   91.36%   91.32%   -0.04%     
==========================================
  Files          31       31              
  Lines       11691    11590     -101     
==========================================
- Hits        10681    10585      -96     
+ Misses       1010     1005       -5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.