Agent workspace indexing: hybrid roadmap (manifest.json → FTS5 → vector/RAG)

[up-guillaume]

Context

Agents accumulate hundreds of files in their workspace over time (current observed: 48 dirs / 1,145 files for a single agent). Navigation and retrieval become a token cost issue as workspaces grow.

Problem

There is no structured indexing layer for agent workspace files today. Agents rely on list_files + search_files (regex) + read_file — which works at small scale but degrades as:

• Workspace file count exceeds ~200 files
• Domain folders exceed 10 subdirectories
• Agents need to find files by topic/tag, not just filename

Proposed 3-Phase Roadmap

Phase 1: Flat index + frontmatter (current, agent-level)

• workspace/manifest.json — flat hash→path mapping, updated on write/move
• workspace/tree.txt — visual directory tree, max depth 3
• YAML frontmatter on all generated files (id, project, tags, status, updated)
• Agents use search_files(pattern="status: active") for scoping
• Value: ~80% of retrieval value at ~10% of complexity
• Implementation: Agent-level skill, no platform changes needed

Phase 2: Full-text search (platform-level, medium-term)

• SQLite FTS5 index built from workspace file contents
• Platform API endpoint: search_workspace(agent_id, query, filters)
• Hybrid: regex (search_files) + FTS5 for keyword search
• Trigger: Phase 1 proves insufficient in production
• Reference: obsidian-agent-brain uses FTS5 for keyword layer (0.3 weight)

Phase 3: Vector/RAG (platform-level, long-term)

• Embedding generation for workspace files (FastEmbed or LiteLLM embeddings)
• Vector store: sqlite-vec or ChromaDB
• Hybrid retrieval: 0.7 vector + 0.3 keyword (obsidian-agent-brain pattern)
• Trigger: Retrieval bottleneck observed, workspace >200 files per agent
• Reference: obsidian-agent-brain architecture (coltonfitzgerald/obsidian-agent-brain)

Related Issues

• [Feature] Workspace: Cloud-Drive Style File Management with Search, Folders, Bulk Operations, and Agent-Powered Organization #466 — Workspace: Cloud-Drive Style File Management with Search (partially overlaps Phase 2)
• [Feature Request] Agent long-term memory: persist tool calling history and enable task resumption #454 — Agent long-term memory: persist tool calling history (adjacent territory)
• [Improvement] Workspace file organization and deduplication - implement code-level optimizations #697 — Workspace file organization and deduplication (code-level optimization)
• 没有openclaw embedding 配置功能 #68 — embedding config (closed wontfix, OpenClaw-specific context)

Reference Implementations

• obsidian-agent-brain — FastEmbed + sqlite-vec + FTS5, hybrid RAG for Obsidian vaults
• Ralph Loop context engineering — disk as memory, fresh context per iteration

Current Status

Phase 1 is being implemented as an agent-level file-management skill across 4 core agents. This issue tracks the platform-level roadmap for Phases 2-3, which require Clawith platform changes.

This issue should NOT be implemented until Phase 1 has been validated in production (estimated 4-6 weeks from 2026-08-02).