fix: give each query lexeme its own BM25 candidate budget

[nuemaan]

Summary

• The FTS prefilter from perf: bound classic BM25 candidates with Postgres FTS #252 bounded candidates with one global ts_rank_cd ordering. That ordering scores term density inside a chunk and ignores corpus-wide rarity, while BM25 weights rare terms heavily, so the chunk BM25 ranks first can be the first one truncated.
• Each query lexeme now draws from its own share of the budget through a lateral join, so a lexeme matching few chunks always contributes them. _MIN_CANDIDATES_PER_LEXEME keeps many-lexeme queries from dividing the budget into slivers.
• Adds a warning when the candidate pool saturates. The debug line reported candidates == limit whether the corpus held exactly that many or far more, so silent truncation was indistinguishable from a healthy query.
• Classic route only. Map-nav default path untouched. No API, worker, deployment, or migration impact.
• Resolves Classic BM25 FTS prefilter can truncate the top BM25 match in large namespaces #278, follows up on Optimize BM25 retrieval candidate loading with Postgres FTS prefilter #195 and perf: bound classic BM25 candidates with Postgres FTS #252.

Measured on 5001 chunks where 5000 densely repeat a common term and one holds a rare term, querying data zebra:

ts_rank_cd position of the rare chunk:   5001 of 5001
rank_rows_by_bm25 position:                 1 of 5001

candidate pool, global ordering, limit 2000:  2000 rows, rare chunk absent
candidate pool, per-lexeme budget:            1001 rows, rare chunk present

The per-lexeme pool is smaller and still keeps the match BM25 wants. Cost is one GIN probe per lexeme rather than one overall, and _MAX_FTS_QUERY_TOKENS already caps that at 50.

Verification

• uv run pytest apps/api/tests/unit/test_bm25_channel_tsquery.py apps/api/tests/contract/test_bm25_fts_prefilter_contract.py — 16 passed against real Postgres 16
• uv run pytest packages/shared-python/shared/tests/test_retrieval_search_channels.py — 4 passed
• uvx ruff check packages/shared-python apps/api/tests — clean
• Reverted the channel change and re-ran the new regression test to confirm it fails without the fix: AssertionError: assert 'common-2' == 'rare-zebra'

test_content_channel_uses_bounded_or_fts_after_scope_filters asserted on the literal LIMIT :fts_candidate_limit as a position marker for the candidate bound. That clause is now LIMIT lb.per_lexeme_limit, so the assertions were repointed. The intent from #244, that scope filters land ahead of any candidate bound so exclusions cannot consume the budget, is unchanged and still asserted.

Not tested: production-scale corpora, and lateral-probe cost at the 50 lexeme ceiling on a large namespace. Both need a dataset I do not have locally.

Deployment Notes

• No new environment variables. RETRIEVAL_POSTGRES_FTS_CANDIDATE_LIMIT keeps its meaning as the total budget, now spent per lexeme.
• No database migrations.
• Backwards compatible. The full-scan fallback is untouched, so rolling back is a code-only revert.

Checklist

• Tests were added or updated when behavior changed
• Public docs, examples, or OpenAPI contracts were updated when needed
• Database migrations are idempotent and safe to deploy
• Logs, errors, and validation paths avoid leaking secrets or user data
• The pull request description explains any breaking or user-visible change