[Roadmap] Hybrid workspace file indexing — manifest.json + frontmatter + future vector/RAG layer

[up-guillaume]

Context

This issue documents a phased roadmap for agent workspace file indexing, derived from a file system audit of 4 production agents (Moiria tenant) and industry research across Hermes, Obsidian, and Ralph Loop patterns.

Related issues:

• [Feature] Workspace: Cloud-Drive Style File Management with Search, Folders, Bulk Operations, and Agent-Powered Organization #466 — Workspace cloud-drive style management (UI-level search, bulk operations)
• [Feature Request] Agent long-term memory: persist tool calling history and enable task resumption #454 — Agent long-term memory / tool call persistence
• [Improvement] Workspace file organization and deduplication - implement code-level optimizations #697 — Workspace file organization and deduplication
• 没有openclaw embedding 配置功能 #68 — Embedding config feature (closed: wontfix, OpenClaw-specific)

Problem

As agents accumulate workspace files over weeks/months (current state: 48 dirs / 1,145 files for one agent), they face two problems:

1. Navigation cost — Agents spend tokens traversing directory trees to find files they created days/weeks ago
2. Retrieval gaps — search_files (regex) works for exact matches but misses semantic relationships (e.g., searching "firewall rules" should find a file titled "Hetzner security config")

Proposed Phased Approach

Phase 1: Structural Indexing (no dependencies, agent-level)

Already feasible today using existing Clawith tools:

Component	File	Purpose
Global Manifest	workspace/manifest.json	Flat key-value map: {file_hash: relative_path}. Updated after every write_file / move_file.
Breadcrumb Tree	workspace/tree.txt	Visual directory structure (max depth 3). Updated after structural changes.
YAML Frontmatter	(per-file header)	Standardized metadata: id, project, tags, status, updated. Parsed via regex without reading full body.
MOC (Map of Content)	workspace/MOC.md	Hub page linking all domain folders. Required when workspace >10 subdirs.

Effort: Zero platform changes. Agents maintain these files via a file-management skill (4 routines: Structure Blueprint, Clean & Prune, Structural Audit, Backup & Staging).

Value: ~80% of retrieval needs covered. Agents can search_files(pattern="status: active") or search_files(pattern="tags:.*security.*") to scope results.

Phase 2: Full-Text Search Enhancement (platform-level)

• SQLite FTS5 index over all workspace Markdown files
• Updated on write_file / delete_file events
• Exposed as a tool: search_workspace(query="firewall rules", scope="all") returning ranked results
• Hermes uses this pattern for tier-2 memory search

Phase 3: Vector/Semantic Layer (platform-level, future)

• Embed workspace files using LiteLLM embedding models (already available in the stack)
• sqlite-vec or pgvector for storage (aligns with existing PostgreSQL)
• Hybrid scoring: 0.7 vector similarity + 0.3 keyword match (obsidian-agent-brain reference architecture)
• Triggers: agent explicitly searches and structural index (Phase 1) returns no results

Reference implementation: obsidian-agent-brain — FastEmbed + sqlite-vec + FTS5

Why Phased

Phase 1 is actionable now at the agent level — no platform code changes needed. It solves the immediate navigation problem. Phases 2-3 require platform-level work and should be prioritized based on observed retrieval bottlenecks after Phase 1 rollout.

Current Status

• Phase 1: Being implemented as part of Moiria agent file system cleanup plan (2026-08-02)
• Phase 2: Not started
• Phase 3: Not started, deferred until retrieval bottleneck is observed

Sources

• Hermes Agent official docs — bounded memory model, auto-consolidation
• Obsidian agent patterns (Luna-chan, obsidian-agent-brain) — MOCs, frontmatter, hybrid RAG
• Google AI synthesis — dual-layer indexing, file-management as explicit skill
• Clawith DeepWiki — workspace revision tracking, edit locks, path traversal protection