Skip to content

Memories with failed embeddings are permanently invisible to semantic search #60

Description

@johnathanneals-dev

Bug

When memory_store (MCP) or POST /memories (REST) fails to generate an embedding — e.g. the embedding provider is temporarily unreachable or the input exceeds the model's context window — the memory is stored with embedding = NULL. These memories are permanently excluded from memory_search semantic results because the search query filters on embedding IS NOT NULL.

There is currently no mechanism to regenerate the embedding after the fact, so the only remediation is to delete and re-store the memory (which changes the ID and loses the created_at timestamp).

A related scenario occurs when switching embedding providers or models (e.g. from text-embedding-3-small at 1536 dimensions to nomic-embed-text at 768 dimensions, as migration 015 supports). Existing embeddings become semantically mismatched with new query embeddings, degrading search quality. Again, there is no built-in way to re-embed existing memories with the new model.

Reproduction

  1. Start Open Brain with an embedding provider configured
  2. Stop the embedding provider (e.g. ollama stop)
  3. Store a memory via MCP or REST — it succeeds with a warning to stderr
  4. Restart the embedding provider
  5. Search for the memory by its content — it is not returned

Suggested fix

Add a memory_regenerate_embedding tool (MCP) and a POST /memories/{id}/regenerate-embedding endpoint (REST) that:

  1. Reads the memory's existing content
  2. Generates a fresh embedding via the configured provider
  3. Updates the embedding column

By default, only memories with NULL embeddings should be updated. A force flag allows overwriting valid embeddings (for model migration scenarios).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions