fix: prune raw metrics through the timestamp index, not a rowid anchor - #22
Merged
Merged
Conversation
The per-batch anchor (max(id) below the cutoff) cannot use any index: it rescans every old row, and on the live router's multi-million-row backlog it consumed the 5-minute prune deadline after deleting only 258 rows — retention never caught up. Each batch now selects its rowids directly with 'timestamp < ? LIMIT n', which the timestamp index answers in O(batch), and the predicate no longer assumes insertion order matches time (the configurable-retention test seeds fresh-then-old rows and caught the anchor's binary-search variant doing exactly that).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Live evidence (0.3.36):
Metric retention pass failed … select interface_metrics prune anchor: context deadline exceededwith deleted=258 — the hourly prune never clears the upgrade backlog. The anchor query (max(id) WHERE timestamp < cutoff) is an unindexable full scan of the old rows; the batched DELETE under it is fine. Fix: drop the anchor, let the batch subselect seek the timestamp index directly (EXPLAIN shows COVERING INDEX + rowid point deletes). Also removes the id↔time ordering assumption — the configurable-retention test (seeds old rows after fresh ones) fails against a binary-search anchor and passes here.