Fix: hybrid lexical + semantic retrieval ordering

[shivansh31414]

Summary

Fix stale documentation that described the hybrid retrieval strategy as using BM25 as a fallback. BM25 was evaluated and removed (PR-NOTES F4c) because its unbounded score scale (~1–3) overwhelms cosine similarity values (0–1) in a shared ORDER BY, causing semantic hits to be pushed out of the top-K entirely. Using it correctly would require rank-based fusion (RRF) or score normalisation, which adds complexity without clear benefit for the "find any session mentioning X" use case.

The actual hybrid strategy is a UNION ALL of:

• Lexical LIKE/ILIKE substring rows, emitting a sentinel score of 1.0 (capped by HIVEMIND_HYBRID_LEXICAL_LIMIT, default 20)
• Semantic cosine-similarity rows, emitting their real 0–1 score

Results are ordered by score descending, so exact keyword matches always lead the ranking while semantic concept matches fill in below. When embeddings are off, retrieval falls back to lexical LIKE/ILIKE only.

Files changed:

• README.md — updated the feature bullet to describe the sentinel-score fusion and document why BM25 was dropped
• src/embeddings/disable.ts — corrected the JSDoc comment from "BM25 / ILIKE matching" to "lexical LIKE/ILIKE matching" with the same rationale

Version Bump

To trigger a release, bump "version" in package.json before merging.

Change type	Version bump	Example
Bug fix	patch (1.2.0 → 1.2.1)	"version": "1.2.1"
New feature	minor (1.2.0 → 1.3.0)	"version": "1.3.0"
Breaking change	major (1.2.0 → 2.0.0)	"version": "2.0.0"

If you don't bump the version, no release will be created.

This is a documentation-only change — no release needed. No version bump required.

Test plan

• Tests pass locally (npm test)
• Relevant new tests added
• Version bumped in package.json, or no release needed for this change ✓ (docs only)

Summary by CodeRabbit

• Documentation
  • Clarified the search documentation’s description of lexical and semantic result scoring and ordering.
  • Documented that lexical-only matching uses LIKE/ILIKE when embeddings are disabled.
  • Added context explaining that BM25 is not used because its score scale is incompatible with cosine similarity without normalization or score fusion.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 702556ba-a411-4c24-969c-c0bdeeeb770f

📥 Commits

Reviewing files that changed from the base of the PR and between 3991d0d and 6f2a362.

📒 Files selected for processing (1)

• README.md

🚧 Files skipped from review as they are similar to previous changes (1)

• README.md

Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review.

---

📝 Walkthrough

Walkthrough

The changes align search documentation with the implemented lexical and cosine retrieval behavior. They also document lexical fallback when embeddings are disabled and the reason BM25 is not used.

Changes

Hybrid search documentation

Layer / File(s)	Summary
Search behavior documentation README.md, src/embeddings/disable.ts	The documentation describes LIKE/ILIKE matching and explains why BM25 is not combined with cosine-similarity scores.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 6f2a3

Documentation clarifies hybrid retrieval ordering and embedding-disabled fallback behavior without changing runtime behavior. No merge-readiness risk is identified.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the documented hybrid lexical and semantic retrieval ordering change.
Description check	✅ Passed	The description includes the required Summary, Version Bump, and Test plan sections. It explains the documentation changes and correctly states that no release or version bump is needed for this docum…
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

🟡 Changes recommended

The updated docs introduce a broken JSDoc sentence and an internal inconsistency in the BM25 rationale wording that should be corrected before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates repository documentation to accurately describe the current hybrid lexical + semantic retrieval strategy (sentinel-score UNION ALL of lexical LIKE/ILIKE rows plus semantic cosine-similarity rows) and clarifies why BM25 was evaluated but removed.

Changes:

• Updated README feature bullet to describe sentinel-score fusion behavior and rationale for dropping BM25.
• Corrected src/embeddings/disable.ts JSDoc to refer to lexical LIKE/ILIKE fallback (not “BM25 / ILIKE”).

File summaries

File	Description
README.md	Updates the hybrid retrieval description and BM25 rationale in the feature list.
src/embeddings/disable.ts	Updates embeddings-disable JSDoc to reflect lexical fallback behavior and BM25 rationale.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[efenocchi]

Closing in favour of #350, which fixes the same stale BM25 wording from #337 with a minimal change to the three affected spots (README, docs/EMBEDDINGS.md, disable.ts JSDoc) and has a green CI run.

Your reading of the hybrid ranking is correct (UNION ALL, 1.0 sentinel for lexical rows, ORDER BY score, BM25 dropped per PR-NOTES F4c), but that explanation already lives in src/shell/grep-core.ts and is too much detail for the README feature list. Two small things here also needed fixing before it could land: the JSDoc in src/embeddings/disable.ts was left with a dangling "readable." line, and the PR title says "Fix" for a docs-only change. Thanks for taking the time to verify it against the code.