Skip to content

feat(embedding): built-in local backend with on-demand model download - #54

Merged
mrsibe merged 1 commit into
mainfrom
feat/local-embedding-backend
Sep 24, 2026
Merged

mrsibe merged 1 commit into
mainfrom
feat/local-embedding-backend

Conversation

@mrsibe

@mrsibe mrsibe commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

What does this PR do?

Implements the built-in local embedding backend (#52) and its download transport (#53).

With no remote embedding connection configured, KnowNote now indexes and searches
with a bundled runtime and an on-demand-downloaded model, instead of failing
with No embedding model configured. The first time a user actually uses the
knowledge base (upload / create knowledge base / search), the model is downloaded
once to userData/models and then works fully offline. A remote connection, when
configured, still takes precedence and keeps the existing AI SDK path.

Why?

The RAG path required the user to understand protocol / base URL / API key /
model id / dimensions before uploading a first PDF. Public issues (#33, #35, #31,
#14, #10, #12) all trace back to this. Embedding should be infrastructure, not a
configuration step: configure chat -> upload a PDF.

Issue #53 covers making the weights arrive reliably: the default downloader in
transformers.js does not verify, resume, or retry, and huggingface.co is not
always reachable.

Related issue

Fixes #52
Fixes #53
Fixes #33
Fixes #37

#37's report is brief, but upload is what triggers indexing, so the zero-config
local default plus the mirror fallback should remove the failure it describes.

Already closed before this PR, listed as context only (no closing keyword):
#35, #31, #14, #10, #12. The dimension-mismatch family (#14 / #31) was fixed by
the per-notebook vector table (#47); this PR adds the zero-config local default
and the EmbeddingSpace identity on top of it.

What changed?

  • EmbeddingBackend abstraction: EmbeddingService only says "I want an
    embedding"; LocalEmbeddingBackend (ONNX) and RemoteEmbeddingBackend
    (existing ModelClient) implement it. Remote is used when configured,
    otherwise local.
  • Local backend: pinned Xenova/multilingual-e5-small (q8, pinned revision,
    384-dim), E5 query: / passage: prefixes applied by purpose, truncation
    at 512 tokens, normalize. Inference sets env.allowRemoteModels = false.
  • Index identity: EmbeddingSpace (model + revision + dtype + pooling +
    normalize + prefixes + dimensions) persisted per notebook in a new
    notebook_embedding_spaces table. Search rejects vectors from a different
    space; the indexing path rebuilds/clears and marks documents pending.
  • Download transport: checked-in manifest (sizes + SHA256), self-written
    downloader with SHA256 verification, atomic rename, HTTP Range resume,
    per-file retry/backoff, disk-space pre-check, classified actionable errors,
    source reachability probing (huggingface.co then hf-mirror.com), and
    manual import from a local folder. Download progress is broadcast to the UI.
  • Settings UI: Embedding section with Built-in Local / Custom provider
    and Not installed / Downloading / Installed states, model path, size, delete,
    import and source probing.
  • Packaging: @huggingface/transformers, onnxruntime-node and sharp are
    production dependencies with asarUnpack entries; the packaged smoke test
    loads them.

How was this tested?

  • npm run typecheck (node + web + test) — passes.
  • npm run lint — 0 errors.
  • npm test — 49 tests pass (22 pre-existing + 27 new).
  • npx electron-vite build — main/preload/renderer bundle builds; the
    @huggingface/transformers dynamic import stays external as intended.
  • Real end-to-end run in Node against the real mirror: downloaded all 7
    manifest files (140,461,908 bytes, every SHA256 verified), then loaded the
    pinned model with allowRemoteModels = false and embedded text offline:
    dimensions === 384, all components finite, L2 norm 1.0000.
  • Semantic sanity check: a Chinese query scored 0.904 against its matching
    Chinese document vs 0.804/0.670 for others, and an English query scored 0.870
    against the Eiffel Tower document — confirming the query/document prefixes and
    cross-lingual behavior.
  • Tests cover space identity, manifest/buildResolveUrl, the downloader (resume,
    checksum mismatch cleanup, retry, cancel, probing, error classification),
    model registry/aliases and local-backend prefixes.

Not run: npm run smoke:packaged / build:unpack (needs the full electron-builder
matrix locally). The packaged smoke check was extended, but should be exercised in CI.

Notes / deferred

  • transformers.js v4.3.0 does not forward the pinned revision during its file
    discovery step, which breaks offline (allowRemoteModels=false) loading.
    This PR works around it by hardlinking the manifest files at the revision-less
    cache path; the revision directory stays canonical so a revision bump still
    forces a fresh download.
  • A user-configurable custom download base URL is not exposed in the UI yet;
    huggingface.co -> hf-mirror.com probing plus manual import is shipped. It
    is a small follow-up if enterprise/internal mirrors are needed.

Screenshots / recordings

UI change: the Embedding section in Settings now offers Built-in Local vs Custom
provider with install state. No screenshot captured in this environment.

Checklist

  • I have reviewed my own changes.
  • npm run typecheck passes.
  • npm run build passes (electron-vite build).
  • I have tested the affected user workflow (real model download + inference).
  • I have not included unrelated changes.
  • I have updated documentation when necessary.

Desktop / build changes

  • Not applicable
  • npm run build:unpack passes.
  • npm run smoke:packaged passes.

Embedding is infrastructure, not a configuration step: uploading a PDF must not
require the user to understand protocol / base URL / API key / model id /
dimensions. With no remote embedding connection configured, KnowNote now uses a
built-in local model, so the path collapses to "configure chat -> upload a PDF".

Local inference (no bundled weights):
- LocalEmbeddingBackend runs multilingual-e5-small (q8, pinned revision,
  384-dim) through ONNX runtime; weights are downloaded on first RAG use and
  cached under userData/models. Inference sets env.allowRemoteModels=false, so
  transformers.js never touches the network after install.
- E5 requires distinct query/document prefixes; indexing passes 'document' and
  search passes 'query'. Previously both used the same input.

Index identity:
- EmbeddingSpace (model + revision + dtype + pooling + normalize + prefixes +
  dimensions) is persisted per notebook. Comparing dimensions alone missed
  same-width model changes, whose vectors are not comparable. Search refuses a
  mismatched space and asks for a re-index; the indexing path rebuilds/clears
  the vector table and marks documents pending.

Download transport (issue #53):
- Checked-in manifest with real sizes and SHA256; downloader verifies every file
  and only then atomically renames it into place.
- HTTP Range resume, per-file retry with backoff, disk-space pre-check,
  classified, actionable errors (DNS / refused / timeout / HTTP / checksum /
  disk / cancelled), and reachability probing across huggingface.co and
  hf-mirror.com. Manual import from a local folder is the offline fallback.
- transformers.js v4.3.0 does not forward the pinned revision during its file
  discovery step, so revision-less hardlink aliases are created for the cache to
  stay readable with allowRemoteModels=false.

Settings: Built-in Local vs Custom provider, with Not installed / Downloading /
Installed states plus model path, size and delete.

Packaging: @huggingface/transformers, onnxruntime-node and sharp are production
dependencies with asarUnpack entries; the packaged smoke test loads them.

Tests cover space identity, the manifest, the downloader (resume, checksum,
retry, cancel, probing), error classification, model registry/aliases and the
local backend prefixes.

Closes #52
Closes #53
@github-actions github-actions Bot added the enhancement New feature or request label Sep 24, 2026
@mrsibe
mrsibe merged commit f48cb45 into main Sep 24, 2026
6 checks passed
@mrsibe
mrsibe deleted the feat/local-embedding-backend branch September 24, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

1 participant