Gemini multimodal embeddings for image/PDF memory

[Paul-Kyle]

Test gemini-embedding-2-preview for indexing screenshots, diagrams, and PDFs alongside text memories.

[Isha-Zaka]

Hi Paul, I am interested in picking this up. I looked through the current embedding/indexing path and the existing PDF ingestion flow.

I was thinking of starting with a focused spike using Gemini Embedding 2: test text + image/PDF embeddings in the same embedding space, verify cross-modal retrieval with the existing SQLite-vec setup, and document what changes would be needed to keep the current file-first architecture intact.

This is marked as a future, would you be open to me working on this?

[Paul-Kyle]

Hi — yes, and thank you for asking before starting; that's exactly the right instinct on a future issue, and it's saved you from a wall I should have documented on this thread long ago.

Your reading of the ingestion path is sound, but there's something in the way that isn't visible from here. Palinode can already embed against a non-local host in principle: embeddings.primary.dialect: "openai" posts to an OpenAI-compatible embeddings endpoint, with the same retry, circuit-breaker and per-input error handling as the Ollama path. The Palinode examples exercise local, unauthenticated llama.cpp, vLLM and LM Studio deployments, so credential support was never built. And the request path is hard-coded as /v1/embeddings appended to the configured URL, which doesn't match every provider's root: against Google's it produces …/v1beta/openai/v1/embeddings, where the real endpoint is …/v1beta/openai/embeddings.

I've opened #203 for the provider class: a credential read from the environment plus a configurable endpoint path, both off by default and with no vendor-specific code. OpenAI and Voyage need the credential; Google needs both changes, and either one alone still fails there. It's yours if you want it — comment there and I'll assign it. That's the piece that makes these hosted text embedding providers a matter of configuration, and it's genuinely useful independent of this issue.

Now the part I want to be straight with you about, because it changes what #4 is worth doing. That work does not hand you multimodal. Google documents image, audio, and document embedding through its native embedContent API, not through the OpenAI-compatibility layer, whose embeddings endpoint is documented with text input. So the compat seam gets us hosted text and stops there; images and PDFs mean a second request contract, which is real design work rather than configuration. I'd rather you know that before you start than discover it halfway through a spike.

There's a second wall behind that one, and it's the more interesting problem. The vector table is created at a fixed width from embeddings.primary.dimensions, so a store built for BGE-M3 can't take vectors of another width without a reindex, and nothing currently guards the mismatch. Text and images sharing one searchable space is exactly where that bites. That is the substance of #4, it stays future and stays open, and it's a design question before it's an implementation.

If you do #203 and still want to write up the multimodal path afterwards — what contract, what happens to the existing store, what a migration looks like — I'd read that with real interest. Just don't spend the evening on it expecting the compat layer to carry it.

Either way: welcome, and thanks for a first message that did the homework.

[Isha-Zaka]

Hi Paul — #203 is merged now, so I would like to come back to this one.

Since PDF ingestion currently extracts the text into a research Markdown file, and the indexer/reconcile path only sends section text through embed() / embed_many(), I'm thinking the multimodal path would need a separate native Gemini embedContent contract using gemini-embedding-2, rather than trying to push image/PDF inputs through the OpenAI-compatible path.

I'd keep the existing text path as-is and add the native Gemini path for multimodal inputs, while keeping the resulting vectors as derived index data so the file-first model stays intact.

The other part is the embedding space itself. chunks_vec and triggers_vec are created using embeddings.primary.dimensions, and right now an unchanged chunk with an existing vector is treated as already indexed. So switching embedding models wouldn't automatically cause those vectors to be rebuilt, even if the configured dimensions happened to stay the same.

My thought is to record enough information to identify the embedding space used by the current index, then check that against the active embedding configuration. If the model/space or dimensions change, we'd rebuild the affected vec tables at the configured width and re-embed from the file-backed sources rather than allowing vectors from different embedding spaces to coexist.

For PDFs, the extracted research Markdown can still remain the durable memory record and provenance, but if we're embedding the PDF itself we'd also need a stable reference back to the original source asset so the multimodal index can be rebuilt later. We'd need the same kind of source reference for screenshots or diagrams.

Does that line up with the direction you had in mind for this issue?