Vector Search Integration Research

[KinyaElGrande]

This issue is going to track the research for integrating vector-search with Corteza

Research questions

1. How are we going to generate the Embeddings?- Expanded to include:

• What embedding models will you use? (OpenAI, Cohere, etc.)
• Are we going to support both Local and Cloud based embedding
• Will we need multiple embedding types (text, image, multimodal)?
• Are we going to fine-tune the embeddings?

2. Which indexing strategy are we going to adopt?
3. How are we going to store the emebeddings?
4. Which type of searching are we going to support? (semantic search, hybrid-search, filtered Vector Search, etc)
5. What is the overall architecture with Corteza going to look like?

[KinyaElGrande]

Vector databases to consider for Corteza intergration which provide options for both Cloud and Self host (for projects with strict data privacy requirements), therefore giving Corteza users control over infrastructure and costs.

Feature/Capability	Weaviate	Qdrant	Milvus
Opensource licence	BSD 3-Clause	Apache License 2.0	Apache License 2.0
Embedding Integration	Native support for a variety of self-hosted and API-based models https://docs.weaviate.io/weaviate/model-providers	Manual embedding management https://qdrant.tech/documentation/embeddings/	Manual embedding management https://milvus.io/docs/embeddings.md
Local Embedding Support	Yes (HuggingFace transformers, GPT4All, KubeAI Ollama and Meta ImageBind)	Yes (bring your own)	Yes (bring your own)
Multi-modal Support	Text, image, multi-modal	Primarily text-focused	Text, image, multi-modal
Supported Search Types	Semantic, hybrid, filtered, Full-text	Semantic, hybrid, filtered, Full-text	Semantic, filtered, full-text, advanced
Go Client Support	Excellent	Excellent	Excellent
Resource Requirements	Moderate	Low to Moderate	High
Ease of Integration	High	High	Moderate
Community Support	Strong with both forum https://forum.weaviate.io/latest and Slack channels	Good with a Discord community support	Good with Discord community support
RBAC support	Native support https://docs.weaviate.io/weaviate/configuration/rbac	?	Native support https://milvus.io/docs/rbac.md

I think Weaviate would be a natural addition to Corteza.
Weaviate adds hybrid search (combining vector and keyword-based queries) and built-in ML model integrations, which can simplify workflows if we need to generate embeddings on the fly.
Weaviate also have an active community and really extensive documentation.
Additionally Weaviate have an official Go SDK, both local and API emebeddings support and an RBAC feature which can be mapped to Corteza's.
It is production-ready, has been battle-tested and have a wide-variety of use-cases both in small and large companies

[KinyaElGrande]

Default ML models

When choosing a default Corteza model, the following are the criteria which should evaluate if a model matches Corteza's vector search needs.

1. Does it support multilingual content?
2. Does it have good semantic understanding?
3. Does it offer good performance in terms of speed and quality?
4. Is it reliable in production enviroments?

After evaluating multiple models against Corteza's specific requirements, sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 emerged as the strongest candidate for default embedding model as it meets the goals outlined above.

Why this model:

• Multilingual: Supports 50+ languages with consistent quality
• Semantic Understanding: Optimized for paraphrase detection and similarity and excels at understanding different expressions of the same concept
• Performance: 384D vectors = efficient storage + fast retrieval
• Production Ready: Pre-built Weaviate Docker image available semitechnologies/transformers-inference:sentence-transformers-paraphrase-multilingual-MiniLM-L12-v2

Technical Advantages for Corteza

Storage Efficiency: The 384-dimensional output reduces storage requirements making it cost effective.

Memory Optimization: Smaller vector dimensions result in more compact HNSW indexes, allowing for better memory utilization and faster search operations.

Deployment Simplicity: Pre-built Weaviate integration eliminates custom containerization complexity, reducing deployment overhead.

Proof of Concept

SImple working implementation available: https://gist.github.com/KinyaElGrande/8b9569b643451250d1f5a862813df118

Demonstrates:

• Schema creation and data indexing
• Multilingual querying (everything is hardcoded)

Support for Custom Models

While the default model covers most Corteza use cases, support for other models can be added through either pre-build or custom Docker-built images

Pre-built Options: Weaviate offers multiple pre-built transformer images for different use cases (available at: https://github.com/weaviate/t2v-transformers-models#pre-built-images)

Custom/Private Model Support: Organizations can deploy private or specialized models using Weaviate's local image creation tutorial (https://github.com/weaviate/t2v-transformers-models?tab=readme-ov-file#pre-built-images)

[KinyaElGrande]

The current implementation will be on OpenSearch, as we still support it on Corteza discovery but the plan is to include other Vector databases, as a plug and play option.

Below is the implementation plan:

Important Bits

• Vector search misconfiguration must never break existing search functionality. The application will gracefully fall back to traditional search when vector search fails.
• Invalid vector configurations will be rejected during startup with clear error messages.
• Configuring vector search can be enabled/disabled per index or globally without service restart.
• Embedding service failures must not prevent document indexing or search operations.
• How are we going to introduce RAG
• How are we going to handle migration of existing documents by adding embeddings to them??

Back-End Implementation

Vector Configuration Management

• Configuration CRUD
  • Create Vector Config
    • Validate embedding API URL accessibility
    • Verify model compatibility and dimensions
    • Validate OpenSearch cluster vector support
    • Include ability to pass some configurations using the environment variables

Index Management

• Dynamic Index Mapping
  • Mapping Generation
    • Generate vector field mappings based on model dimensions
    • Support multiple vector fields per document??
    • Set default similarity metrics (cosine, l2, innerproduct) and HNSW parameters (M, EF_construction)
    • Include ability to pass vector similarity and HNSW parameters
  • Mapping Updates
    • Add vector fields to existing indices
    • Validate mapping compatibility
    • Handle mapping conflicts gracefully

Document Operations

• Indexing Enhancement

  • Single Document Indexing

    • Generate embeddings during document indexing
    • Extract text content for embedding generation
    • Handle embedding failures gracefully
    • Intelligently Concatenate discovery accessible record fields into a single value for vector embeddings

  • Update Operations

    • Regenerate embeddings on content changes
    • Selective embedding updates
    • Handle embedding versioning

Search Operations

• Query Processing
  • Vector Query Generation
    • Build query embedding pipeline that takes user search text, generates embeddings using the same model and settings as indexed documents, handles embedding generation failures by falling back to traditional search.
  • Hybrid Search
    • Create result merging system that combines vector search results with traditional keyword search results, implements configurable scoring weights for vector vs keyword relevance, and provides tunable parameters for balancing different search approaches.
  • Search Modes
    • Implement vector-only search mode that uses only semantic similarity for matching, provides fastest search when documents have complete embedding coverage, handles similarity threshold filtering, and returns results ranked purely by vector similarity scores.
    • Activate automatically traditional keyword fallback when vector search is unavailable, maintains exact same functionality as existing search.
    • Build hybrid search as default mode that combines benefits of both approaches.

Future considerations

• Have the option for choosing which model should be used to generate which record fields or potentially resource.
• Have the vector search switch settings on other resources as well, module and eventually record fields.

[github-actions]

Stale issue message