Skip to content

feat: LanceDB semantic vectors + in-process native Qwen3 embedder (MinSync shape) - #40

Merged
vkehfdl1 merged 3 commits into
mainfrom
feat/issue-39-lancedb-native-embedder
Sep 20, 2026
Merged

vkehfdl1 merged 3 commits into
mainfrom
feat/issue-39-lancedb-native-embedder

Conversation

@vkehfdl1

Copy link
Copy Markdown
Member

Fixes #39 (also supersedes the closed #38 — batching alone was not enough).

What changed

Storage: semantic_vectors (JSON number[] text in SQLite) is replaced by a LanceDB table at <data-dir>/semantic.lance — a typed FixedSizeList<Float32> vector column plus account/mailbox/thread/date predicate columns. SQLite remains the source of truth for messages, chunks, and FTS; the Lance table is a rebuildable semantic index (same layout MinSync produces: _versions, _transactions, data). <data-dir>/semantic.identity.json persists the embedder identity next to the table.

Embedder: the zero-config default is now the in-process native profile — native:Qwen/Qwen3-Embedding-0.6B (1024-d) through ONNX Runtime's native Node binding (WebGPU → Metal on Apple Silicon, automatic CPU fallback), last-token pooling, no HTTP sidecar. EmbeddingGemma stays as the legacy-onnx opt-in (and imports compatible legacy JSON vectors on first index); loopback-http remains the explicit #31 override.

Indexing: mandatory configurable batching (--batch-size / MAILCRAWL_EMBED_BATCH_SIZE, default 4 for the native profile, mirroring MinSync). Each batch commits the Lance upsert and the queue completion; a crash resumes from the last committed batch. Reuse is decided against stored content hashes so re-enqueued rows never re-embed.

Query: searchSemantic is a cosine vector-table query with account/mailbox/thread/from/after/before pushed down as Lance SQL predicates (only to hydrates through SQLite). Search against a table built by a different embedder fails loudly with a rebuild hint.

Rebuild guard: identity mismatch (provider/model/dimension/prefixes) on mailcrawl index discards the table and re-embeds everything — never silent reuse. repair --semantic forces a full rebuild; doctor/status report healthy / interrupted / never-completed / rebuild-required / corrupt / missing from the store identity plus the queue.

Verified on this machine (M4 Max, macOS arm64)

  • Real Qwen3-Embedding-0.6B on WebGPU/Metal: load ~1s, batch-of-4 ~30ms, dim 1024, sequential calls stable, sensible cosine rankings (related 0.91 vs unrelated 0.42).
  • Full CLI e2e with the default native path: sync → index (2 embedded) → status/doctor healthy → search:semantic / --mode hybrid ranked hits → mailbox filter → resume (reused: 2) → repair --semantic (rebuilt: true) → mismatch guard exits 1 with the rebuild hint.
  • npx tsc --noEmit clean, npm run build clean, 100/100 vitest tests pass (new tests/issue-39.test.ts: 15 tests over the real LanceDB binding — store roundtrip, filters, dimension guard, crash-resume batching, rebuild-on-identity-change, legacy import/discard, sweep, summaries), npm run validate:skill valid.

Notes

  • PR fix: batch semantic embedding to prevent OOM kill #38 was closed unmerged; its batching behavior is included here as part of the redesign.
  • Version bumped to 0.2.0 (semantic storage format change).
  • Docs updated: README, docs/architecture.md, docs/multilingual-installation.md, skills/mailcrawl/SKILL.md, THIRD_PARTY_NOTICES.md (Qwen3 Apache-2.0, LanceDB Apache-2.0).

…ative embedder

Fixes #39. The semantic arm now matches the MinSync contract:

- Vector store: LanceDB table at <data-dir>/semantic.lance (typed
  FixedSizeList<Float32> column) instead of JSON text in semantic_vectors.
  SQLite stays the source of truth for messages/chunks/FTS.
- Embedder: in-process native default, native:Qwen/Qwen3-Embedding-0.6B
  (1024-d) through the ONNX Runtime native binding, WebGPU (Metal) on Apple
  Silicon with CPU fallback, batched. No HTTP sidecar by default.
  EmbeddingGemma remains as the legacy-onnx opt-in profile; loopback-http
  stays the explicit #31 override.
- Batching is mandatory and configurable (--batch-size, default 4 for the
  native profile); each batch commits vector upsert + queue completion so a
  crash resumes from the queue.
- Identity is data: provider/model/dimension/prefixes/runtimeBuild persisted
  at <data-dir>/semantic.identity.json. Mismatch forces a full rebuild and
  search against a mismatched table fails loudly. Compatible legacy JSON
  vectors are imported when the legacy profile is explicitly active.
- Query is a vector-table query with account/mailbox/thread/from/date
  predicates pushed down; only the to filter hydrates through SQLite.
- doctor/status report semantic health from the store identity and queue;
  repair --semantic forces a full rebuild.
Windows CI spends ~1s per CLI spawn loading native addons, so five
sequential sync/status/embed/repair/search invocations exceeded
vitest's default 5s budget.
A Lance table without a readable identity is an unknown vector space.
Index reused those rows by content hash and wrote a new identity.
Drop and re-embed instead; doctor reports rebuild-required.
@vkehfdl1
vkehfdl1 merged commit 375e3d5 into main Sep 20, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace JSON-in-SQLite 'vectors' with LanceDB + in-process native embedder (MinSync shape, no HTTP sidecar)

1 participant