Proposal Draft: Split Control Plane Metadata from Vector Indexing in KB

[sqhyz55]

Proposal Draft: Split Control Plane Metadata from Vector Indexing in KB

Abstract

This proposal introduces a clear storage boundary for KB:

• MetadataStore for control-plane metadata (collection_metadata, ingestion_runs, prompt_templates, main_pointers)
• VectorIndexStore for vector indexing and retrieval (documents/parses/chunks/embeddings and search/index operations)

The goal is to remove current coupling, reduce migration risk, and make multi-backend vector support practical.
This proposal follows the direction discussed in Issue #90.

What We Are Proposing

1. Define a strict storage boundary:
   • Control-plane -> MetadataStore
   • Data-plane -> VectorIndexStore
2. Migrate control-plane metadata from LanceDB to RDB in a phased way.
3. Keep vector path provider-agnostic through VectorIndexStore SPI.
4. Use single-primary vector write/read by default, while reserving interfaces for future multi-store read.

Current State

Today LanceDB serves both:

• control-plane metadata
• vector indexing and retrieval

This leads to cross-layer coupling in API/service code and makes backend expansion expensive.

Current (As-Is) Core Flow

sequenceDiagram
    participant API as API/Service
    participant Biz as KB Business Modules
    participant LDB as LanceDB (mixed responsibilities)

    API->>Biz: ingest/search/delete
    Biz->>LDB: write metadata tables
    Biz->>LDB: write vector tables
    Biz->>LDB: read/search/index ops

Loading

Problems and Motivation

Problems

• Boundary ambiguity: control-plane and vector-plane responsibilities are mixed.
• High migration cost: adding a new vector backend risks copying existing coupling.
• Consistency complexity: metadata/vector updates are difficult to reason about and recover.
• Review friction: architecture intent is hidden behind implementation details.

Motivation

• Make responsibilities explicit and auditable.
• Enable safe phased migration with rollback controls.
• Prepare clean SPI for pluggable vector providers.
• Improve long-term maintainability for open-source contributors.

Proposed Architecture

Target (To-Be) Core Flow

sequenceDiagram
    participant API as API/Service
    participant Coord as KBWriteCoordinator
    participant MS as MetadataStore (RDB)
    participant OP as Operation/Outbox
    participant VS as VectorIndexStore (VDB)

    API->>Coord: submit write/delete
    Coord->>MS: write control-plane state
    Coord->>OP: enqueue operation
    OP->>VS: apply vector-side changes
    VS-->>MS: update status/stats via coordinator

Loading

Design Principles

• Domain contracts first (no direct DB semantics in business/API layer)
• Explicit consistency model (operation/outbox/reconcile)
• Configurable and rollback-friendly migration
• Provider-agnostic vector API with explicit capability fallback
• 2PC is treated as a reference option, not the default path

Phased Plan

Phase 1A: Interface Decoupling (no physical DB split yet)

• Introduce MetadataStore / VectorIndexStore / coordinator contracts.
• Remove direct control-plane LanceDB semantics from service/API paths.

Phase 1B: Control Plane Migration to RDB

• Add RDB MetadataStore.
• Run dual-write + backfill.
• Switch control-plane reads to RDB.
• Stop VDB control-plane writes after stability window.

Phase 2: Pluggable Vector Providers

• Finalize vector SPI.
• Incrementally add providers (for example, Milvus/Chroma).
• Define and test explicit degrade/fallback behavior.

Phase 3: Stabilization and Decommission

• Remove legacy paths.
• Harden operations (reconcile, alerts, runbooks, SLO checks).
• Finalize archival/cleanup and compatibility policy.

Compatibility and Migration Principles

• Read compatibility first
• Idempotent backfill/dual-write
• Independent read/write feature gates for rollback
• Continuous reconcile checks during migration window
• Prefer Outbox + idempotency + reconcile as the default consistency path; keep 2PC as optional future mode only.

Non-Goals (for this proposal)

• No Memory subsystem refactor in this workstream.
• No frontend progress subsystem redesign.
• No immediate default enablement of multi-store read.

Open Questions

1. What are acceptance thresholds for reconcile mismatch before read switch?
2. Which provider capability matrix is required before Phase 2 exits?
3. What retention/archival policy should be enforced in Phase 3?

Suggested Baseline (for discussion)

To make review concrete, here is a proposed starting baseline for the three open questions:

1. Reconcile thresholds before read switch

   • Record-count mismatch rate: <= 0.1% over at least 3 consecutive runs.
   • Critical-key mismatch (collection/doc_id/step_type/model_tag): 0 unresolved in final pre-switch run.
   • Operational health gates: no sustained outbox backlog breach and no high-severity reconcile errors for a full stability window (for example, 7 days).

2. Minimum provider capability matrix for Phase 2 exit

   • Required capabilities (must pass): upsert, delete, dense search, metadata filter, index health checks.
   • Optional with explicit fallback (must be documented): sparse/FTS, hybrid search, advanced index tuning controls.
   • Exit gate: at least one non-LanceDB provider passes required capabilities in CI + integration environment, with fallback behavior verified.

3. Phase 3 retention and archival policy

   • Operations/outbox records: hot retention 30 days, archive retention 180 days (configurable by deployment).
   • Legacy VDB control-plane artifacts: keep read-only for one release cycle after cutover, then archive and remove.
   • Reconcile/audit evidence: keep summaries for at least one full release cycle to support post-cutover incident analysis.

[AldenWangExis]

Agree with the overall direction.

Small suggestion: LanceDB also has a documents table (doc registry metadata like source_path/file_type/content_hash/title/language/user_id).
Maybe include documents in Phase 1B RDB migration scope too.

Refs:

• src/xagent/core/tools/core/RAG_tools/file/register_document.py
• src/xagent/web/api/kb.py
• src/xagent/core/tools/core/RAG_tools/LanceDB/schema_manager.py

[sqhyz55]

Great suggestion, and we agree with the direction.
For now, since the unified file_id-based file management refactor is still in progress, we plan to keep the current documents table in scope for Phase 1B migration (same as other control-plane tables), to avoid introducing migration risk or partial semantics during the transition.
Our plan is:
Short term (Phase 1B): migrate and maintain documents with the current framework for compatibility and stability.
After file_id refactor is complete and stable: move read/write dependencies to the new file registry model.
Then deprecate/remove documents: only after dependency removal + reconcile stability window + rollback-safe cutover checks.
So we fully agree with the end goal, but prefer a staged transition to keep rollout risk controlled.

[qinxuye]

Yeah, I think file should be managed by file management, that means each uploaded file will have a file id which can be used for reading files, KB, etc.

[sqhyz55]

Agree with the overall direction.

Small suggestion: LanceDB also has a documents table (doc registry metadata like source_path/file_type/content_hash/title/language/user_id). Maybe include documents in Phase 1B RDB migration scope too.

Refs:

• src/xagent/core/tools/core/RAG_tools/file/register_document.py
• src/xagent/web/api/kb.py
• src/xagent/core/tools/core/RAG_tools/LanceDB/schema_manager.py

If you’re willing to continue with this proposal, should we divide the tasks, or let whoever has bandwidth pick up the implementation?

[AldenWangExis]

Phase 1A/1B metadata migration to RDB is challenging and interesting, and I’m happy to help. But for collaboration efficiency, communication cost, and test coverage, it may be better handled within your team. After the abstraction layer is done and validated, I can contribute VDB provider extensions in Phase 2. Happy to discuss anytime.

[sqhyz55]

Staged Plan for doc_id vs file_id
Phase 1A: Keep doc_id as the KB/RAG primary key, and focus only on interface decoupling (MetadataStore / VectorIndexStore) without changing identity semantics.
Phase 1B: Add optional linkage (external_file_id / source_file_id) in RDB metadata and backfill where possible, while keeping doc_id as the primary key and preserving full backward compatibility.
Not in default 1A/1B scope: No full re-key from doc_id to file_id, and no mandatory file_id requirement for all ingestion sources.
Fast-follow option (1B+): If priority is high, start file_id-primary migration immediately after 1B stabilization (short stability window), guarded by feature flags, dual-read/write compatibility, and tested rollback.
Gate to proceed: linkage coverage/reconcile quality is acceptable, critical APIs/tools support both IDs during transition, and core workflows remain stable.

[sqhyz55]

KB delete: permissions & rollback (current vs. post control/data-plane split)

Current
Non-admin delete uses a best-effort LanceDB preflight (no foreign documents).
A second check runs under the per-collection file lock before trash + vector delete to reduce TOCTTOU risk.
Ingest/delete are serialized via the same lock.
Rollback is SQL-only: db.rollback() applies to ORM state (e.g., UploadedFile), but does not revert LanceDB deletes or filesystem changes.
No cross-store transaction (no 2PC across Postgres, LanceDB, FS).
Post-split (planned)
Permissions and lifecycle state move to a relational MetadataStore.
A KBWriteCoordinator manages writes via transactional commits + outbox.
Vector/physical deletes are applied asynchronously by idempotent workers to VectorIndexStore.
Consistency model
Explicit consistency:
Control plane → transactional (SQL + operation records)
Data plane → async + idempotent + reconcilable
No true atomic rollback across SQL/vector/FS.
“Rollback” = abort/compensate control-plane transaction + reconcile data plane (Saga-style), with repair runbooks for edge cases.

[sqhyz55]

The following changes were made to proposal based on team discussion:

1. Phase 3 Enhancement - KB Delete Permissions & Rollback

Added explicit details for KB delete permissions and rollback implementation:

• Permissions and lifecycle state fully managed by relational MetadataStore
• KBWriteCoordinator manages writes via transactional commits + outbox pattern
• Vector/physical deletes applied asynchronously by idempotent workers to VectorIndexStore
• Control plane → transactional (SQL + operation records)
• Data plane → async + idempotent + reconcilable
• "Rollback" = abort/compensate control-plane transaction + reconcile data plane (Saga-style)
• No true atomic rollback across SQL/vector/FS - limitations acknowledged and documented

2. Phase 1B - file_id Integration as Core Scope

Updated Phase 1B description to explicitly include file_id integration:

Before: Phase 1B was described only as "Control Plane Migration to RDB"

After: Phase 1B is now "Control Plane Migration to RDB + file_id Integration"

Added bullet point: "Core scope (not optional): Integrate file_id linkage as described in Document Identity Strategy below."

3. Document Identity Strategy - Phased Approach Decision

Changed from "keep doc_id primary with optional file_id linkage" to "phased migration to file_id-primary":

Before:

• Keep doc_id as canonical KB document key
• Introduce optional cross-domain linkage as metadata
• Treat key unification as later optional evolution

After:

• Agreed approach: Phased migration from doc_id-primary to file_id-primary semantics
• Keep doc_id as canonical during Phase 1A and 1B for stability
• Introduce cross-domain linkage in Phase 1B
• Begin progressive migration to file_id-primary after Phase 1B stabilization (~1-2 weeks)
• Maintain dual-read/dual-write compatibility during migration window

4. Scope Clarification Updates

Refined scope boundaries for clarity:

Out of scope for Phase 1A: No changes to doc_id vs file_id semantics

In scope for Phase 1B: Add nullable external_file_id linkage field, backfill linkage, establish cross-domain mapping

Progressive migration (post-1B): Re-keying RAG tables, making file_id primary identity - begins after 1B stabilization window

5. Acceptance Gates Updates

Enhanced acceptance criteria to reflect phased approach:

• 1A exit: No functional regression with doc_id as primary
• 1B exit: Linkage field populated, optional, does not break doc_id workflows, dual-write stable
• Post-1B (migration start): Linkage quality verified, feature gates exist, rollback path tested

6. Restructured Migration Timeline

Replaced "Optional Fast-Follow after 1B (1B+)" section with "Post-Phase 1B: Progressive file_id Migration"

Rationale: This is now a planned part of the roadmap, not an optional add-on. The migration approach remains the same but with clearer commitment to execute.