Skip to content

fix(vector): one vector table per notebook with its own embedding width - #47

Merged
mrsibe merged 1 commit into
mainfrom
fix/per-notebook-vector-dimensions
Sep 24, 2026
Merged

mrsibe merged 1 commit into
mainfrom
fix/per-notebook-vector-dimensions

Conversation

@mrsibe

@mrsibe mrsibe commented Sep 24, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

Moves vectors from one global vec_embeddings table (fixed FLOAT[1024]) to one
vec0 table per notebook (vec_<notebookId>), with each notebook's embedding
width recorded in vec_metadata. The width now comes from the embedding model
instead of being assumed to be 1024.

Why?

A vec0 table's FLOAT[...] width is fixed at creation. KnowNote asked for
dimensions: 1024 on every embedBatch() call and created vec_embeddings as
FLOAT[1024], so any embedding model that does not return 1024 dimensions fails
at insert time - after the embedding call has already been paid for. #33 reports
exactly this: every embedding model except a 1024-dimensional one cannot index.
VectorStoreManager also carried a mutable defaultDimensions, so whichever
notebook was indexed last decided the width for the next one.

Related issue

Related to #33 (fixes the dimension half of the report; the "only the first of
several selected PDFs indexes" half is untouched).
Design follows #21 by @jimmyken - see the note at the bottom.

What changed?

  • src/main/db/index.ts: vec_metadata (notebook_id → table_name, dimensions),
    plus getNotebookVectorTable / createNotebookVectorTable /
    rebuildNotebookVectorTable / dropNotebookVectorTable. Creating a table whose
    stored width differs from the requested one throws - it never drops rows.
    initVectorStore() creates vec_metadata and migrates the old global
    vec_embeddings (vec0 → vec0 INSERT..SELECT, per notebook, then DROP),
    skipping orphan vectors whose notebook no longer exists.
  • src/main/vectorstore/SQLiteVectorStore.ts: each store resolves its own table.
    No width → read it from vec_metadata; no table yet → query/count answer
    empty instead of creating a table from a guess. upsert throws on a width
    mismatch instead of letting sqlite-vec fail cryptically.
  • src/main/vectorstore/VectorStoreManager.ts: defaultDimensions and
    setDefaultDimensions are gone; the store, not the manager, knows the width.
  • src/main/services/KnowledgeService.ts: embeddings are requested with the
    model's native width; the measured width is fed back via
    reconcileVectorDimensions(), which rebuilds the table and marks the
    notebook's documents pending only when the width actually changed.
  • src/main/db/queries.ts: deleteNotebook() drops the notebook's vector table;
    cascade deletes do not reach vec0 shadow tables. Also tightens
    createMessage(metadata) from Record<string, any> to
    Record<string, unknown>, matching the column's own $type<...>().
  • src/main/vectorstore/vectorTableSql.ts (new): every statement that must
    interpolate a table name - a SQL identifier cannot be a bind parameter - behind
    a single safeIdentifier() allowlist.
  • src/main/smokeTest.ts: seeds a legacy vec_embeddings, asserts it migrates
    and stays searchable, and asserts the data-loss case directly (mismatch is
    refused, existing vectors survive, only an explicit rebuild replaces a table).

How was this tested?

  • npm run typecheck
  • npm test - 20 passing, including the new test/vectorTableSql.test.ts
    (identifier allowlist, per-table statements, placeholder counts)
  • npx eslint / npx prettier --check on the touched files (one pre-existing
    no-explicit-any warning at queries.ts:200, unchanged)
  • npm run build:unpack && npm run smoke:packaged - PASS, 13 checks, on
    Linux x64. The output shows the legacy table being migrated, the 768- and
    1536-dimensional rebuilds, and the table being dropped with its notebook.

Not tested: a real end-to-end index against a non-1024 provider (OpenAI
text-embedding-3-small at 1536, Ollama nomic-embed-text at 768). The smoke
test covers the storage layer against real sqlite-vec, not a live API call -
worth doing before release.

Screenshots / recordings

Not applicable.

Checklist

  • I have reviewed my own changes.
  • npm run typecheck passes.
  • npm run build passes.
  • I have tested the affected user workflow.
  • I have not included unrelated changes.
    The Record<string, any> → unknown line is the one exception (it sits in
    a file this PR already touches for deleteNotebook).
  • I have updated documentation when necessary.

Desktop / build changes

  • Not applicable
  • npm run build:unpack passes.
  • npm run smoke:packaged passes.

Note on #21

#21 (@jimmyken, d37edfa) is where this design comes from: one vector table per
notebook, a vec_metadata table, widths detected from the model. It could not be
merged as-is - it dropped and recreated a notebook's table whenever a caller
asked for a different width, and the search/reindex call sites had no width to
give, so one wrong default silently destroyed that notebook's vectors; the
vec_embeddings migration was also missing (flagged in review on #21). This PR
keeps the design and changes the rule to "the stored width decides, and only the
indexing path may rebuild", plus the migration. @jimmyken is credited on the
commit as co-author; #21 is closed in favour of this.

Embedding width was hardcoded to 1024 in three places at once: the vec0 DDL
(FLOAT[1024]), the `dimensions: 1024` passed to embedBatch(), and the
VectorStoreManager default. A vec0 table's width is fixed at creation, so any
model that does not return 1024 dimensions could not index at all - the insert
fails after the embedding call has already been paid for. That is #33: every
embedding model except a 1024-dimensional one fails to index.

Vectors now live in one vec0 table per notebook (`vec_<notebookId>`) with the
width recorded in `vec_metadata`. That table is the single authority on the
width:

- The embedding width comes from the model (`embeddings[0].dimensions`) instead
  of being requested as 1024.
- Search, delete and single-document reindex read the width from `vec_metadata`.
  They no longer pass a guessed default, so they can never create or drop a table
  based on a wrong number.
- A width change is only possible through `rebuildNotebookVectorTable()`, called
  from the indexing path, which also marks the notebook's documents `pending` so
  they get re-indexed. A mismatch anywhere else throws.
- Existing installs are migrated in `initVectorStore()`: the old global
  `vec_embeddings` is copied notebook by notebook (`INSERT..SELECT` between vec0
  tables) and then dropped. Without this, upgrading users would keep a table the
  new code never reads.
- Deleting a notebook drops its table and metadata row; cascade deletes do not
  reach vec0 shadow tables.

`vectorTableSql.ts` collects every statement that has to interpolate a table
name - a SQL identifier cannot be a bind parameter - behind an allowlist, and is
covered by `test/vectorTableSql.test.ts`.

The packaged smoke test now seeds a legacy 1024-dimensional `vec_embeddings`
table, asserts it is migrated and still searchable, and asserts the data-loss
case explicitly: a width mismatch must be refused, must leave existing vectors
untouched, and only an explicit rebuild may replace them.

Design follows #21 by @jimmyken (d37edfa), which introduced per-notebook tables
and dynamic widths. That version dropped and recreated a notebook's table
whenever a caller asked for a different width, and several call sites (search,
reindex) had no width to give, so a single wrong default silently destroyed a
notebook's vectors; the migration was also missing. This reworks it so the
stored width, not a default, decides.

Co-authored-by: jimmyken <four498@gmail.com>
@mrsibe
mrsibe merged commit f961f03 into main Sep 24, 2026
3 checks passed
@mrsibe
mrsibe deleted the fix/per-notebook-vector-dimensions branch September 24, 2026 08:08
@mrsibe mrsibe added the bug Something isn't working label Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant