Skip to content

Stop re-deciding what the index already decided - #79

Merged
sepehr-safari merged 1 commit into
mainfrom
stop-rechecking-what-the-index-guaranteed
Aug 11, 2026
Merged

Stop re-deciding what the index already decided#79
sepehr-safari merged 1 commit into
mainfrom
stop-rechecking-what-the-index-guaranteed

Conversation

@sepehr-safari

Copy link
Copy Markdown
Contributor

A query picks the index that suits its filter, walks it, and then checks each candidate against the whole filter again. Two of those checks cannot fail. Reaching an event through its author's index proves the author matches; reading it because its id was named proves the id does.

Filter.matches compares against a list one entry at a time, so the redundant half is a scan of the whole list, per candidate. A feed over a real follow list re-scanned two thousand pubkeys for every note it returned, having reached that note through the very index that answered the question. A client refreshing a feed it already holds by naming the ids it holds pays it squared.

Each branch now nulls out what its index guaranteed and checks the rest. Time bounds stay in: they cost nothing, and they are a check on the index agreeing with the record it points at.

The number

QueryResult.list_checks reports the size of that term, the way examined already reports index entries read. It is zero for every filter an index can answer, which is the property the new tests assert. A stopwatch cannot tell a quadratic scan from a busy machine; this can.

ReleaseFast, best of 50 and the lowest of repeated runs, 2049 authors over 210k events:

shape before after list_checks
same feed by id, 1200 ids 1202 us 871 us 3.9M to 0
same feed by id, 300 ids 260 us 231 us 270k to 0
same feed by id, 60 ids 61 us 56 us 11k to 0
wide feed, limit 60 1506 us 1438 us 123k to 0, inside the noise

The merged path's saving is real but small at that shape, and I am not going to claim otherwise: its cost is the cursor per author and the linear pick across streams, not this. The win is on the by-id path and it grows with how many ids are named.

Tests

Four, and all four fail without the change, with the counts you would predict:

  • a feed over 300 follows: list_checks 6000 to 0
  • 200 ids named: list_checks 40000 to 0
  • a constraint no index proved (a tag, and a kind under the author index) is still enforced, and costs no list walk
  • an id named by the wrong author is still rejected, so dropping the id check did not drop the rest of the filter

Verified by putting the old behaviour back and watching each one go red.

A query picks the index that suits its filter, walks it, and then checks
each candidate against the whole filter again. Two of those checks cannot
fail. Reaching an event through its author's index proves the author
matches; reading it because its id was named proves the id does.

`Filter.matches` compares against a list one entry at a time, so the
redundant half is a scan of the whole list, per candidate. A feed over a
real follow list re-scanned two thousand pubkeys for every note it
returned, having reached that note through the very index that answered
the question. A client refreshing a feed it already holds by naming the
ids it holds pays it squared.

Each branch now nulls out what its index guaranteed and checks the rest.
Time bounds stay in: they cost nothing and they are a check on the index
agreeing with the record it points at.

`QueryResult.list_checks` reports the size of that term, the same way
`examined` reports index entries read. It is zero for every filter an
index can answer, which is the property the new tests assert. A stopwatch
cannot tell a quadratic scan from a busy machine; this can.

Measured on this machine, ReleaseFast, best of 50 and the lowest of
repeated runs, 2049 authors over 210k events:

  same feed by id, 1200 ids : 1202 us -> 871 us   (list_checks 3.9M -> 0)
  same feed by id,  300 ids :  260 us -> 231 us
  same feed by id,   60 ids :   61 us ->  56 us
  wide feed, limit 60       : no measurable change

The merged path's saving is real but small at that shape: its cost is the
cursor per author and the linear pick across streams, not this. The win is
on the by-id path, and it grows with how many ids are named.
@sepehr-safari
sepehr-safari merged commit e3bb6d1 into main Aug 11, 2026
2 checks passed
@sepehr-safari
sepehr-safari deleted the stop-rechecking-what-the-index-guaranteed branch August 11, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant