From 7bb7d5c531b417d49c613c9874dab765cc062aab Mon Sep 17 00:00:00 2001 From: sumitvairagar Date: Fri, 11 Sep 2026 23:55:37 +0530 Subject: [PATCH] docs: replace stale BM25 references with ILIKE in user-facing docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BM25 was tried and dropped (PR-NOTES F4c) because its score range (~1–3) overpowered cosine similarity (0–1) in the UNION ALL query, pushing semantic results out of the top-K ranking entirely. The current lexical path uses ILIKE with a fixed sentinel score of 1.0. Updated five references across three user-facing files: - README.md: feature bullet and embeddings section - docs/EMBEDDINGS.md: intro paragraph and degraded-mode checklist - src/embeddings/disable.ts: JSDoc comment Internal source comments in grep-core.ts and deeplake-api.ts that describe BM25's history are intentionally left as-is — they document the architectural decision for future contributors. Fixes #337 --- README.md | 4 ++-- docs/EMBEDDINGS.md | 4 ++-- src/embeddings/disable.ts | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 8a6a1b0fd..a25f87464 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ On [LoCoMo](https://arxiv.org/abs/2402.17753), the public long-context memory be - 📥 **Captures** every session's prompts, tool calls, and responses as structured traces in Deeplake - 🧠 **Codifies** patterns into reusable `SKILL.md` files, available to every agent on your team -- 🔍 **Searches** traces and skills with hybrid lexical + semantic retrieval (BM25 fallback when embeddings off) +- 🔍 **Searches** traces and skills with hybrid lexical + semantic retrieval (ILIKE lexical fallback when embeddings off) - 🔗 **Propagates** capability across sessions, agents, teammates, and machines in real time - 📁 **Intercepts** file operations on `~/.deeplake/memory/` through a virtual filesystem backed by SQL - 📝 **Summarizes** sessions into AI-generated wiki pages via a background worker at session end @@ -435,7 +435,7 @@ So `org switch` moves everything that *isn't* explicitly pinned; a pin stays put ## Semantic search (optional) -Hivemind ships with a local embedding daemon (nomic-embed-text-v1.5) for hybrid semantic + lexical search over `~/.deeplake/memory/`. **Off by default** because the dependency footprint is ~600 MB. Enable with `hivemind embeddings install` (or `hivemind install --with-embeddings`). Without it, search degrades silently to BM25/lexical-only. +Hivemind ships with a local embedding daemon (nomic-embed-text-v1.5) for hybrid semantic + lexical search over `~/.deeplake/memory/`. **Off by default** because the dependency footprint is ~600 MB. Enable with `hivemind embeddings install` (or `hivemind install --with-embeddings`). Without it, search degrades silently to ILIKE lexical-only. Full guide: **[docs/EMBEDDINGS.md](docs/EMBEDDINGS.md)**. diff --git a/docs/EMBEDDINGS.md b/docs/EMBEDDINGS.md index 7e74c34b8..c355510ac 100644 --- a/docs/EMBEDDINGS.md +++ b/docs/EMBEDDINGS.md @@ -1,6 +1,6 @@ # Embeddings (semantic search) -Hivemind can run a local embedding daemon (nomic-embed-text-v1.5, ~130 MB) so that `Grep` over `~/.deeplake/memory/` uses hybrid semantic + lexical ranking instead of pure BM25. This is **off by default** — the daemon depends on `@huggingface/transformers`, which pulls onnxruntime-node and sharp (~600 MB total with native binaries). Shipping that with every agent install would 60× the install size for a feature most users don't need. +Hivemind can run a local embedding daemon (nomic-embed-text-v1.5, ~130 MB) so that `Grep` over `~/.deeplake/memory/` uses hybrid semantic + lexical ranking instead of pure ILIKE lexical matching. This is **off by default** — the daemon depends on `@huggingface/transformers`, which pulls onnxruntime-node and sharp (~600 MB total with native binaries). Shipping that with every agent install would 60× the install size for a feature most users don't need. ## Install @@ -32,7 +32,7 @@ Restart your agents after enabling. From the next session, captured messages and If `@huggingface/transformers` is **not** present, Hivemind silently degrades to lexical-only mode: - ✅ Capture continues; rows still land in Deeplake. -- ✅ `Grep` still works via BM25 / `ILIKE` matching on text columns. +- ✅ `Grep` still works via ILIKE (lexical substring) matching on text columns. - ⚪ The `message_embedding` / `summary_embedding` columns stay `NULL`. - ⚪ The hook log notes `embeddings: no-transformers` once at session start. diff --git a/src/embeddings/disable.ts b/src/embeddings/disable.ts index a96741975..a199fe135 100644 --- a/src/embeddings/disable.ts +++ b/src/embeddings/disable.ts @@ -23,7 +23,7 @@ import { getEmbeddingsEnabled } from "../user-config.js"; * * In either case: SessionStart skips the warmup, capture / wiki-worker * write rows with NULL in the embedding column, and `Grep` falls back to - * BM25 / ILIKE matching on text columns. Existing rows' embeddings remain + * ILIKE (lexical substring) matching on text columns. Existing rows' embeddings remain * readable. * * Read-once: the status is cached for the lifetime of the (short-lived)