Repository navigation
feat: LanceDB semantic vectors + in-process native Qwen3 embedder (MinSync shape) - #40
Merged
Merged
Conversation
…ative embedder Fixes #39. The semantic arm now matches the MinSync contract: - Vector store: LanceDB table at <data-dir>/semantic.lance (typed FixedSizeList<Float32> column) instead of JSON text in semantic_vectors. SQLite stays the source of truth for messages/chunks/FTS. - Embedder: in-process native default, native:Qwen/Qwen3-Embedding-0.6B (1024-d) through the ONNX Runtime native binding, WebGPU (Metal) on Apple Silicon with CPU fallback, batched. No HTTP sidecar by default. EmbeddingGemma remains as the legacy-onnx opt-in profile; loopback-http stays the explicit #31 override. - Batching is mandatory and configurable (--batch-size, default 4 for the native profile); each batch commits vector upsert + queue completion so a crash resumes from the queue. - Identity is data: provider/model/dimension/prefixes/runtimeBuild persisted at <data-dir>/semantic.identity.json. Mismatch forces a full rebuild and search against a mismatched table fails loudly. Compatible legacy JSON vectors are imported when the legacy profile is explicitly active. - Query is a vector-table query with account/mailbox/thread/from/date predicates pushed down; only the to filter hydrates through SQLite. - doctor/status report semantic health from the store identity and queue; repair --semantic forces a full rebuild.
Windows CI spends ~1s per CLI spawn loading native addons, so five sequential sync/status/embed/repair/search invocations exceeded vitest's default 5s budget.
A Lance table without a readable identity is an unknown vector space. Index reused those rows by content hash and wrote a new identity. Drop and re-embed instead; doctor reports rebuild-required.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #39 (also supersedes the closed #38 — batching alone was not enough).
What changed
Storage:
semantic_vectors(JSONnumber[]text in SQLite) is replaced by a LanceDB table at<data-dir>/semantic.lance— a typedFixedSizeList<Float32>vector column plus account/mailbox/thread/date predicate columns. SQLite remains the source of truth for messages, chunks, and FTS; the Lance table is a rebuildable semantic index (same layout MinSync produces:_versions,_transactions,data).<data-dir>/semantic.identity.jsonpersists the embedder identity next to the table.Embedder: the zero-config default is now the in-process native profile —
native:Qwen/Qwen3-Embedding-0.6B(1024-d) through ONNX Runtime's native Node binding (WebGPU → Metal on Apple Silicon, automatic CPU fallback), last-token pooling, no HTTP sidecar. EmbeddingGemma stays as thelegacy-onnxopt-in (and imports compatible legacy JSON vectors on first index);loopback-httpremains the explicit #31 override.Indexing: mandatory configurable batching (
--batch-size/MAILCRAWL_EMBED_BATCH_SIZE, default 4 for the native profile, mirroring MinSync). Each batch commits the Lance upsert and the queue completion; a crash resumes from the last committed batch. Reuse is decided against stored content hashes so re-enqueued rows never re-embed.Query:
searchSemanticis a cosine vector-table query with account/mailbox/thread/from/after/before pushed down as Lance SQL predicates (onlytohydrates through SQLite). Search against a table built by a different embedder fails loudly with a rebuild hint.Rebuild guard: identity mismatch (provider/model/dimension/prefixes) on
mailcrawl indexdiscards the table and re-embeds everything — never silent reuse.repair --semanticforces a full rebuild;doctor/statusreporthealthy/interrupted/never-completed/rebuild-required/corrupt/missingfrom the store identity plus the queue.Verified on this machine (M4 Max, macOS arm64)
sync→index(2 embedded) →status/doctorhealthy →search:semantic/--mode hybridranked hits → mailbox filter → resume (reused: 2) →repair --semantic(rebuilt: true) → mismatch guard exits 1 with the rebuild hint.npx tsc --noEmitclean,npm run buildclean, 100/100 vitest tests pass (newtests/issue-39.test.ts: 15 tests over the real LanceDB binding — store roundtrip, filters, dimension guard, crash-resume batching, rebuild-on-identity-change, legacy import/discard, sweep, summaries),npm run validate:skillvalid.Notes