fix: add embedding regeneration for stale/missing vectors - #61
Open
johnathanneals-dev wants to merge 1 commit into
Open
fix: add embedding regeneration for stale/missing vectors#61johnathanneals-dev wants to merge 1 commit into
johnathanneals-dev wants to merge 1 commit into
Conversation
…ctors
When memory_store fails to generate an embedding (provider unreachable,
context window exceeded), the memory is stored with a NULL embedding and
becomes permanently invisible to semantic search. There is also no
remediation path after switching embedding providers or models.
Add memory_regenerate_embedding (MCP tool) and
POST /memories/{id}/regenerate-embedding (REST endpoint) that re-embed
an existing memory's content via the configured provider. By default
only NULL embeddings are updated; a force flag allows overwriting valid
embeddings for model-migration scenarios.
Includes:
- db layer: regenerate_embedding() in attribution.py
- REST: POST /memories/{id}/regenerate-embedding with 409 protection
- MCP: memory_regenerate_embedding tool
- tests: 8 unit tests matching existing test conventions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #60. Adds the ability to regenerate embeddings for memories that were stored with
NULLembeddings (provider failure at store time) or that need re-embedding after a model/dimension change.I flagged this gap in my review of PR #15 — the underlying issue (no remediation path for failed or stale embeddings) is still present in the current codebase.
Changes
DB layer (
src/db/attribution.py):regenerate_embedding()— reads memory, checks embedding state, UPDATEs theembeddingcolumn. Returns None for missing ID, raises ValueError if embedding exists withoutforce=True.REST endpoint (
src/api/main.py):POST /memories/{id}/regenerate-embeddingwith JSON body{"force": false}. Status codes: 400 (bad UUID), 404 (not found), 502 (embedder failure), 409 (has embedding, no force).MCP tool (
src/main.py):memory_regenerate_embeddingwithmemory_id(required) andforce(optional boolean, default false).Tests (
tests/test_embedding_regeneration.py): 8 unit tests covering all status paths (missing memory, existing embedding without force, null embedding update, force overwrite) plus MCP tool schema and handler tests. Follows existing mock-cursor test conventions fromtest_agent_attribution.py.Design notes
regenerate_embeddingis not concurrency-guarded, matching existing conventions in the codebase (no locking patterns elsewhere).force=Falsedefault protects against accidental re-embedding of valid vectors.force=Trueis the explicit operator gesture for model-migration scenarios.Test plan
psycopg2which is not available locally but is in the CI/Docker environment)