From 3991d0d1f4b36ae351d58084c5f5b45f179ed560 Mon Sep 17 00:00:00 2001 From: Shivansh31414 Date: Tue, 8 Sep 2026 22:40:41 +0530 Subject: [PATCH 1/2] Fix: hybrid lexical + semantic retrieval ordering --- README.md | 2 +- src/embeddings/disable.ts | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8a6a1b0fd..16823c1e8 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ On [LoCoMo](https://arxiv.org/abs/2402.17753), the public long-context memory be - 📥 **Captures** every session's prompts, tool calls, and responses as structured traces in Deeplake - 🧠 **Codifies** patterns into reusable `SKILL.md` files, available to every agent on your team -- 🔍 **Searches** traces and skills with hybrid lexical + semantic retrieval (BM25 fallback when embeddings off) +- 🔍 **Searches** traces and skills with hybrid lexical + semantic retrieval: a `UNION ALL` of `LIKE`/`ILIKE` substring rows (sentinel score 1.0, capped by `HIVEMIND_HYBRID_LEXICAL_LIMIT`) and cosine-similarity rows (real 0–1 score), ordered by score — so exact keyword matches always lead while semantic hits fill in below. When embeddings are off, falls back to lexical `LIKE`/`ILIKE` only. (BM25 was evaluated but dropped: its unbounded score scale (~1–3) overwhelmed cosine in a shared `ORDER BY`, requiring rank-based fusion (RRF) or score normalisation to use safely.) - 🔗 **Propagates** capability across sessions, agents, teammates, and machines in real time - 📁 **Intercepts** file operations on `~/.deeplake/memory/` through a virtual filesystem backed by SQL - 📝 **Summarizes** sessions into AI-generated wiki pages via a background worker at session end diff --git a/src/embeddings/disable.ts b/src/embeddings/disable.ts index a96741975..7af74a0f3 100644 --- a/src/embeddings/disable.ts +++ b/src/embeddings/disable.ts @@ -23,7 +23,9 @@ import { getEmbeddingsEnabled } from "../user-config.js"; * * In either case: SessionStart skips the warmup, capture / wiki-worker * write rows with NULL in the embedding column, and `Grep` falls back to - * BM25 / ILIKE matching on text columns. Existing rows' embeddings remain + * lexical `LIKE`/`ILIKE` matching on text columns (BM25 was evaluated but + * dropped — its score scale (~1..3) is incompatible with cosine in a single + * ORDER BY without RRF or normalisation; see grep-core.ts PR-NOTES F4c). * readable. * * Read-once: the status is cached for the lifetime of the (short-lived) From 6f2a3623c41aa3c592ed031d78b8743d4850dbd5 Mon Sep 17 00:00:00 2001 From: Shivansh Sharma Date: Wed, 9 Sep 2026 22:00:33 +0530 Subject: [PATCH 2/2] Fix duplicate search functionality description in README Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 16823c1e8..d5bc81b03 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ On [LoCoMo](https://arxiv.org/abs/2402.17753), the public long-context memory be - 📥 **Captures** every session's prompts, tool calls, and responses as structured traces in Deeplake - 🧠 **Codifies** patterns into reusable `SKILL.md` files, available to every agent on your team -- 🔍 **Searches** traces and skills with hybrid lexical + semantic retrieval: a `UNION ALL` of `LIKE`/`ILIKE` substring rows (sentinel score 1.0, capped by `HIVEMIND_HYBRID_LEXICAL_LIMIT`) and cosine-similarity rows (real 0–1 score), ordered by score — so exact keyword matches always lead while semantic hits fill in below. When embeddings are off, falls back to lexical `LIKE`/`ILIKE` only. (BM25 was evaluated but dropped: its unbounded score scale (~1–3) overwhelmed cosine in a shared `ORDER BY`, requiring rank-based fusion (RRF) or score normalisation to use safely.) +- 🔍 **Searches** traces and skills with hybrid lexical + semantic retrieval: a `UNION ALL` of `LIKE`/`ILIKE` substring rows (sentinel score 1.0, capped by `HIVEMIND_HYBRID_LEXICAL_LIMIT`) and cosine-similarity rows (real 0–1 score), ordered by score — so exact keyword matches always lead while semantic hits fill in below. When embeddings are off, falls back to lexical `LIKE`/`ILIKE` only. (BM25 was evaluated but dropped: its score scale (~1–3) overwhelmed cosine in a shared `ORDER BY`, requiring rank-based fusion (RRF) or score normalisation to use safely.) - 🔗 **Propagates** capability across sessions, agents, teammates, and machines in real time - 📁 **Intercepts** file operations on `~/.deeplake/memory/` through a virtual filesystem backed by SQL - 📝 **Summarizes** sessions into AI-generated wiki pages via a background worker at session end