Python: callable embedder (PyCallableEmbedder via EmbedderChoice::InProcess)

[coreyt]

Summary

Allow Python callers to register a custom embedder (any object with embed(text) -> list[float], identity() -> dict, max_tokens() -> int) with the engine. Today only EmbedderChoice::Builtin is reachable from Python — anyone wanting a non-builtin embedder (OpenAI, Cohere, local Ollama, deterministic test stub, etc.) is blocked.

Context

The Rust engine already supports EmbedderChoice::InProcess(Arc<dyn QueryEmbedder>). The pyo3 path at crates/fathomdb/src/python.rs::parse_embedder_choice only handles "none" and "builtin". Python build wheels without the default-embedder feature (Candle + bge-small ~100MB) have no way to drive the vector projection drain.

Design doc: dev/notes/2026-04-22-python-callable-embedder.md (to be added — outline in .claude/plans/ during the managed vector projection release, April 2026).

Parent release: managed per-kind vector projection (branch design-db-wide-embedding-per-kind-vec, ~47 commits, adds configure_embedding, configure_vec_kind, drain_vector_projection, semantic_search, raw_vector_search, schema v24).

Scope

New pyo3 types

• PyEmbedderHandle (#[pyclass]) wrapping Arc<PyCallableEmbedder>.
• PyCallableEmbedder implementing Rust QueryEmbedder:
  • embed_query(&self, text) -> Result<Vec<f32>, EmbedderError> calls back via Python::with_gil(|py| callable.call_method1(py, "embed", (text,))).
  • identity(&self) -> QueryEmbedderIdentity returns a value cached at registration time from the Python object's identity() method. The Rust side never re-queries Python for identity after registration.
  • max_tokens(&self) -> usize from the Python object's max_tokens(), default 512.
• BatchEmbedder impl reuses the existing QueryEmbedderBatchAdapter pattern (sequential loop over embed_query).

Registration API

• New #[pyfunction] make_embedder_choice_from_callable(py_embedder) -> PyEmbedderHandle.
• Extend EngineCore::open with an optional embedder_handle: Option<&PyAny> kwarg alongside the existing embedder: Option<&str> string. When the handle is set, the string path is ignored (or returns an error if both are supplied).
• parse_embedder_choice gains a branch: if embedder_handle is Some, extract PyEmbedderHandle and build EmbedderChoice::InProcess(shim.inner.clone() as Arc<dyn QueryEmbedder>).

Python protocol (documented, not enforced via ABC)

class Embedder(Protocol):
    def identity(self) -> dict[str, object]:
        # {"model_identity": str, "model_version": str | None,
        #  "dimension": int, "normalization_policy": str | None}
    def embed(self, text: str) -> list[float]: ...
    def max_tokens(self) -> int: ...  # optional; defaults to 512

Identity invariant

Identity flows FROM the embedder — the caller cannot override identity via an admin call. The pyo3 shim reads identity() once at registration, caches it, and ignores any later mutation on the Python object. AdminService::configure_embedding continues to read identity via QueryEmbedder::identity(); no code on the admin side changes.

GIL safety

• Drain thread releases the GIL between calls (with_gil scope ends), so Python progress isn't blocked.
• Auto-drain path from submit_write(auto_drain_vector=true) on a Python-caller thread: confirm submit_write releases the GIL (py.detach) before the drain runs, mirroring the pattern already established in EngineCore::open.
• Sequential, GIL-serialized embedding is acceptable for v1; document the perf characteristic on PyCallableEmbedder.

Error translation

Python exceptions in embed(text) → EmbedderError::Backend(format!("python embed: {e}")). Do not attempt to preserve traceback across the boundary.

Test plan

New python/fathomdb/tests/test_callable_embedder.py:

class DeterministicEmbedder:
    def identity(self):
        return {"model_identity": "test-det", "model_version": "v1",
                "dimension": 4, "normalization_policy": "l2"}
    def embed(self, text):
        h = hash(text) & 0xFFFFFFFF
        return [float((h >> i) & 0xFF) / 255.0 for i in (0, 8, 16, 24)]
    def max_tokens(self):
        return 128

def test_configure_and_semantic_search(tmp_path):
    handle = make_embedder_choice_from_callable(DeterministicEmbedder())
    engine = Engine.open(tmp_path / "db", embedder_handle=handle, auto_drain_vector=True)
    engine.admin.configure_embedding({...matching identity fields...})
    engine.admin.configure_vec_kind({"kind": "chunk", "source": "chunks"})
    engine.submit_write({...chunk with text...})
    results = engine.semantic_search("chunk", "hello")
    assert len(results) > 0

Coverage required:

1. Happy path end-to-end (above).
2. Identity mismatch: configure_embedding with fields that don't match the embedder's identity() → EmbeddingIdentityMismatch.
3. Dimension mismatch at embed time: embedder returns a vector of the wrong length → EmbedderError::Backend surfaced at drain.
4. Python exception in embed propagates as FathomError with the message.
5. Callable embedder works on a build without the default-embedder feature (explicit feature-disabled CI lane, or a comment noting this is the whole point).

Scope guardrails

• Do NOT add a new EmbedderChoice variant; reuse InProcess.
• Do NOT support mutating the Python embedder's identity after registration.
• Do NOT attempt concurrent embed_query from Rust.
• Do NOT touch crates/fathomdb/src/node.rs in this issue (the Node-callable embedder is a separate issue).

Followups / open questions

• Batched Python embed (single call with list[str]) — trivial extension once this lands.
• async def embed — needs a pyo3 async runtime bridge; not now.
• Free-threaded pyo3 (gil_used=false) intersects this design — validate that Python::with_gil remains correct before that migration.

Memex impact

Optional — Memex's current path (FathomStore.open(embedder="builtin")) is unblocked by the 0.5.x managed vector projection release and does not need this. This issue unlocks alternative embedders (non-Candle wheels, deterministic test embedders in Memex's own pytest suite, OpenAI/Cohere/Ollama backends).