Skip to content

fix: add embedding regeneration for stale/missing vectors - #61

Open
johnathanneals-dev wants to merge 1 commit into
benclawbot:masterfrom
johnathanneals-dev:fix/stale-embedding-regeneration
Open

fix: add embedding regeneration for stale/missing vectors#61
johnathanneals-dev wants to merge 1 commit into
benclawbot:masterfrom
johnathanneals-dev:fix/stale-embedding-regeneration

Conversation

@johnathanneals-dev

Copy link
Copy Markdown

Summary

Closes #60. Adds the ability to regenerate embeddings for memories that were stored with NULL embeddings (provider failure at store time) or that need re-embedding after a model/dimension change.

I flagged this gap in my review of PR #15 — the underlying issue (no remediation path for failed or stale embeddings) is still present in the current codebase.

Changes

  • DB layer (src/db/attribution.py): regenerate_embedding() — reads memory, checks embedding state, UPDATEs the embedding column. Returns None for missing ID, raises ValueError if embedding exists without force=True.

  • REST endpoint (src/api/main.py): POST /memories/{id}/regenerate-embedding with JSON body {"force": false}. Status codes: 400 (bad UUID), 404 (not found), 502 (embedder failure), 409 (has embedding, no force).

  • MCP tool (src/main.py): memory_regenerate_embedding with memory_id (required) and force (optional boolean, default false).

  • Tests (tests/test_embedding_regeneration.py): 8 unit tests covering all status paths (missing memory, existing embedding without force, null embedding update, force overwrite) plus MCP tool schema and handler tests. Follows existing mock-cursor test conventions from test_agent_attribution.py.

Design notes

  • The SELECT-then-UPDATE in regenerate_embedding is not concurrency-guarded, matching existing conventions in the codebase (no locking patterns elsewhere).
  • force=False default protects against accidental re-embedding of valid vectors. force=True is the explicit operator gesture for model-migration scenarios.

Test plan

  • Verify tests pass in CI environment (tests require psycopg2 which is not available locally but is in the CI/Docker environment)
  • Manual test: store a memory with embedder stopped, verify NULL embedding, call regenerate, verify embedding populated and memory appears in search results
  • Manual test: call regenerate without force on a memory with valid embedding, verify 409 response

…ctors

When memory_store fails to generate an embedding (provider unreachable,
context window exceeded), the memory is stored with a NULL embedding and
becomes permanently invisible to semantic search.  There is also no
remediation path after switching embedding providers or models.

Add memory_regenerate_embedding (MCP tool) and
POST /memories/{id}/regenerate-embedding (REST endpoint) that re-embed
an existing memory's content via the configured provider.  By default
only NULL embeddings are updated; a force flag allows overwriting valid
embeddings for model-migration scenarios.

Includes:
- db layer: regenerate_embedding() in attribution.py
- REST: POST /memories/{id}/regenerate-embedding with 409 protection
- MCP: memory_regenerate_embedding tool
- tests: 8 unit tests matching existing test conventions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memories with failed embeddings are permanently invisible to semantic search

1 participant