Skip to content

Fix broken cursor pagination for word_similarity relevance search - #201

Merged
BarryArinze merged 1 commit into
aid-linkk:masterfrom
Petah1:fix/194-cursor-pagination-word-similarity
Aug 21, 2026
Merged

Fix broken cursor pagination for word_similarity relevance search#201
BarryArinze merged 1 commit into
aid-linkk:masterfrom
Petah1:fix/194-cursor-pagination-word-similarity

Conversation

@Petah1

@Petah1 Petah1 commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Summary

closes #194. Cursor pagination for searchCampaignsByRelevance() / searchDonationsByRelevance() in src/services/search.service.ts had two real bugs, not just the float-precision issue described in the issue:

  • A hard crash: the WHERE clause referenced the SELECT-list alias score directly (WHERE (score, id) < (...)). Postgres does not allow referencing an output alias from the same query's WHERE clause — verified against a live Postgres 15 instance that any second-page request threw column "score" does not exist.
  • A type-promotion bug: word_similarity() returns float4. The cursor's score, once round-tripped through Prisma into a JS number and sent back as a query parameter, gets bound as numeric. Comparing a float4 column against a numeric parameter silently produces wrong results — confirmed directly against Postgres that an equality check between a real column and its own round-tripped value returned zero matching rows.

Fix

  • Cast the score expression to double precision everywhere it's used (SELECT, > 0.2 threshold filter, cursor comparison), so every comparison happens in one consistent Postgres type instead of relying on implicit float4/numeric promotion.
  • Replace the tuple comparison (score, id) < (x, y) with an explicit score < x OR (score = x AND id < y), which also removes the alias-in-WHERE dependency causing the crash.
  • Add COALESCE(word_similarity(query, description), 0) for campaigns, matching the existing NULL-handling already used in donation search.
  • Version the cursor payload (v: 2). A pre-fix (unversioned) cursor is now detected and treated as absent — falls back to page one — instead of being silently misinterpreted with the old, broken comparison semantics.

Applied identically to both searchCampaignsByRelevance() and searchDonationsByRelevance(). Beneficiary search and the non-relevance sort paths are untouched (out of scope per the issue).

Test plan

  • Unit tests (mocked Prisma) in src/services/search.service.test.ts: cursor encode/decode roundtrip for a float4-range value, legacy/unversioned cursor is dropped gracefully, adversarial manipulated cursor (score: -1) doesn't crash or leak rows, generated SQL uses CAST(... AS DOUBLE PRECISION) and COALESCE for description.
  • Integration tests in tests/performance/search.performance.test.ts against a real Postgres 15 + pg_trgm instance (validated locally with docker):
    • 50-campaign match set, 10/page: all 50 unique IDs returned, no duplicates, no gaps.
    • 20 campaigns with an identical word_similarity score: partition cleanly across two pages by id DESC, zero overlap.
    • Manipulated cursor score returns empty results without crashing.
  • npx tsc --noEmit clean for the changed file.
  • Existing cursor-based-pagination unit tests still pass (updated to build v2-format cursors). Pre-existing unrelated test failures in the same file (offset-pagination expectations for a different code path, tiebreaker-order mismatches, unrelated mock gaps in searchBeneficiaries/globalSearch tests) predate this change and are unaffected.

Cursor pagination for searchCampaignsByRelevance/searchDonationsByRelevance
had two real bugs, not just the float precision issue: the WHERE clause
referenced the SELECT-list alias "score" in `(score, id) < (...)`, which
Postgres rejects outright (verified against a live Postgres — any second
page request threw "column \"score\" does not exist"); and even once fixed
to reference the expression directly, comparing the float4 word_similarity
output against a JS number bound as `numeric` silently dropped or duplicated
rows, including exact-tie rows that share a score (confirmed: an equality
check between a real column and its own value round-tripped through Prisma
returned zero matches).

Fix: cast the score expression to double precision everywhere it's used
(SELECT, threshold filter, cursor comparison) so every comparison happens
in one consistent Postgres type instead of relying on implicit float4/
numeric promotion, and decompose the cursor condition into an explicit
`score < x OR (score = x AND id < y)` instead of a tuple comparison, so it
never depends on the "score" alias being visible in WHERE. Also add
COALESCE around the description word_similarity call for campaigns, matching
the existing donation search handling of NULL columns.

Cursors are now versioned (`v: 2`) so a pre-fix cursor is detected and
treated as absent (falls back to page one) instead of being silently
misinterpreted with the old, broken comparison semantics.

Validated the SQL-level fix against a real Postgres 15 + pg_trgm instance:
20 identically-scored rows paginate across two pages with zero duplicates
and zero gaps, and a 50-row match set walks cleanly across all pages.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix broken cursor pagination for word_similarity relevance search — floating-point score roundtrip corrupts cursor position

2 participants