Skip to content

feat(index): add optional Milvus derived index backend#332

Open
zc277584121 wants to merge 3 commits into
EverMind-AI:mainfrom
zc277584121:feat/milvus-derived-index
Open

feat(index): add optional Milvus derived index backend#332
zc277584121 wants to merge 3 commits into
EverMind-AI:mainfrom
zc277584121:feat/milvus-derived-index

Conversation

@zc277584121

Copy link
Copy Markdown

Summary

  • Add a backend-neutral derived index facade while keeping LanceDB as the default backend.
  • Add an optional Milvus backend behind everos[milvus] and [index] backend = "milvus".
  • Support Milvus Lite by default under <root>/.index/milvus/milvus.db, plus Milvus server or Zilliz Cloud via [milvus] uri, token, and db_name.
  • Use MilvusClient, explicit schemas, AUTOINDEX, native BM25 functions, dense COSINE search, and backend-specific filter rendering.

Design Notes

  • Markdown remains the source of truth.
  • SQLite remains responsible for system state, queues, and coordination.
  • Milvus collections are rebuildable derived indexes and can be dropped/rebuilt without losing source data.
  • subject_vector recall falls back to the main episode vector on Milvus because the Milvus backend stores one dense vector field per row.

Docs

  • Update README, quickstart, configuration, architecture, storage layout, cascade/search docs, and example config.
  • Document the opt-in [index] / [milvus] configuration and the local Lite default.

Tests

  • uv run python -m compileall -q src/everos
  • uv run ruff check src/everos/infra/persistence/milvus/repository.py src/everos/memory/search/filters.py tests/unit/test_memory/test_search/test_recall_episode.py tests/unit/test_memory/test_search/test_recall_knowledge_topic.py tests/unit/test_memory/test_search/test_manager.py tests/unit/test_memory/test_get/test_manager.py
  • uv run lint-imports
  • git diff --check
  • uv run pytest tests/unit/test_config/test_settings.py tests/unit/test_core/test_persistence/test_memory_root.py tests/unit/test_memory/test_search/test_filters.py tests/unit/test_infra/test_milvus/test_repo.py tests/unit/test_memory/test_search/test_recall_episode.py tests/unit/test_memory/test_search/test_recall_knowledge_topic.py tests/unit/test_memory/test_search/test_recall_profile.py tests/unit/test_memory/test_search/test_recall_atomic_fact.py tests/unit/test_memory/test_search/test_recall_agent_skill.py tests/unit/test_memory/test_get/test_manager.py tests/unit/test_memory/test_search/test_manager.py tests/unit/test_scripts/test_check_datetime_discipline.py::test_real_repo_passes_discipline_gate
  • Milvus Lite smoke: upsert/search/filter/delete through the optional backend test.
  • Full uv run pytest tests/unit: 1483 passed, 20 failed due to Python 3.14 rejecting invalid escape sequences in the installed jieba package during tokenizer import.

Related to #326.

@zc277584121 zc277584121 force-pushed the feat/milvus-derived-index branch from 70476b6 to c7db5bd Compare July 9, 2026 06:59
@zc277584121

Copy link
Copy Markdown
Author

Pushed a small follow-up that normalizes Milvus COSINE and BM25 score semantics across Milvus Lite and Zilliz Cloud. Milvus Lite reports COSINE as distance while Zilliz Cloud reports similarity through the same field, so the backend now normalizes both to the LanceDB-style distance contract expected by the recall layer. BM25 scores are also normalized to positive higher-is-better values.

Additional validation:

  • real Milvus Lite smoke with OpenAI text-embedding-3-small using 1024 dimensions
  • real Zilliz Cloud smoke with the same data and query
  • both returned the same dense ordering, the same normalized dense distances, the same sparse top hit, and cleaned up temporary collections
  • focused Milvus and recall tests pass

@zc277584121 zc277584121 force-pushed the feat/milvus-derived-index branch from c7db5bd to b184fad Compare July 9, 2026 07:25
Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com>
@zc277584121 zc277584121 force-pushed the feat/milvus-derived-index branch from b184fad to 9758ba2 Compare July 9, 2026 09:25
@zc277584121

Copy link
Copy Markdown
Author

Pushed a follow-up to narrow the COSINE score workaround to Milvus Lite 3.0 / 3.0.0 only, with a link to the upstream Milvus Lite tracking issue: milvus-io/milvus-lite#343

For Milvus server, Zilliz Cloud, and future Milvus Lite versions, the backend now treats COSINE raw distance as similarity and converts it to the LanceDB-style distance contract expected by the recall layer. Re-ran the Lite + Zilliz smoke and the focused test suite after the change.

@zc277584121

Copy link
Copy Markdown
Author

Pushed one follow-up fix commit after running a larger Milvus smoke matrix.

Changes:

  • Bound hierarchical atomic-fact recall to Milvus/Zilliz's 1024 search topK limit.
  • Switched Milvus count() to an exact count(*) query instead of collection stats, which can lag immediately after writes on cloud deployments.

Validation run:

  • python -m pytest tests/unit/test_memory/test_search/test_recall_atomic_fact.py -q
  • python -m pytest tests/unit/test_infra/test_milvus/test_repo.py -q
  • python -m ruff check src/everos/memory/search/recall/atomic_fact.py src/everos/infra/persistence/milvus/repository.py tests/unit/test_memory/test_search/test_recall_atomic_fact.py tests/unit/test_infra/test_milvus/test_repo.py
  • Manual large E2E smoke on Milvus Lite and Zilliz Cloud with real OpenAI embeddings: 121 episodes + 121 atomic facts, covering keyword/vector/hybrid search, nested atomic facts, owner/project/session isolation, deprecated filtering, and cleanup.

@zc277584121 zc277584121 changed the title Add optional Milvus derived index backend feat(index): add optional Milvus derived index backend Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant