Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ internal/
├── embedding/ — Embedding provider interfaces, mock provider, and ONNX provider implementation
├── hybrid/ — Reciprocal Rank Fusion helpers for merged lexical/vector retrieval
├── sequences/ — F4 PrefixSpan mining (deterministic discovery of frequent tool-call sequences per session)
└── storage/ — SQLite adapter (dual FTS5 indexes: tool_fts + messages_fts, BM25, WAL mode, migrations v1–v13, search_items, session_tags, tool_events, message_templates, template_matches, correction_signals, annotations, AggregateCommands, AggregateFailures, AggregateTemplates, AggregateCorrections, UpsertAnnotation, LoadToolSequences)
└── storage/ — SQLite adapter (dual FTS5 indexes: tool_fts + messages_fts, BM25, WAL mode, migrations v1–v14, search_items, session_tags, tool_events, message_templates, template_matches, correction_signals, annotations, AggregateCommands, AggregateFailures, AggregateTemplates, AggregateCorrections, UpsertAnnotation, LoadToolSequences)
```

Ten v2 CLI commands: `list [--project] [--all-projects] [--recent N] [--order timestamp:desc|asc] [--limit] [--offset] [--json] [--robot]`, `search [--text <query>] [--project] [--all-projects] [--source] [--source-path] [--after] [--before] [--role] [--content-type] [--tag] [--limit] [--offset] [--fields minimal|full] [--max-tokens N] [--lexical-only] [--similarity-threshold F] [--json] [--robot]`, `patterns --kind commands|failures|templates|sequences|corrections [--pending] [--batch N] [--project] [--all-projects] [--tag] [--trend] [--after] [--before] [--min-support N] [--min-confidence F] [--min-length N] [--max-length N] [--limit] [--offset] [--json] [--robot]`, `annotate --uuid <u> --kind <k> --label <l> [--path <p> --ordinal <n>]`, `recover --from <path> [--dry-run]`, `status [--json]`, `validate [--json]`, `rebuild`, `purge --before <date>`, `config [--json]`.
Expand Down Expand Up @@ -150,6 +150,7 @@ External knowledge sources are configured with active `*.inputs.toml` manifests
- **Cross-host project identity**: `projects.Identify()` normalizes session cwd against registry roots by matching root tails (≥2 components, case-insensitive), so `/home/shared/<proj>` sessions resolve against `/Users/Shared/<proj>` roots on a synced index. Registry roots should keep distinct suffixes — two projects whose roots share the same trailing components could misbucket.
- **Recall eval-set**: `docs/eval/queries.toml` (~20 real mined queries with `expected_match` ground truth) + `scripts/eval.sh` compute recall@5; a query counts only if its expected target appears in the top 5. Local regression gate, not a required CI step.
- **Single catalog source of truth (fix #52)**: The lineage catalog lives in `internal/compat/manifest.json` (data, regenerable via `REGEN_MANIFEST=1`). `internal/storage/recovery_records.go` queries `compat.Catalog.IsKnownSignature()` to check if a schema is recognized, rather than carrying a stale hardcoded switch. This eliminates duplication and the risk of signature drift after normalization changes. Whitespace normalization collapsed 76 fixtures into 17 distinct signatures; collision consistency is verified by a test that ensures all entries with the same signature agree on `AppliedVersion` and `HasSourceMetadata`.
- **V14 file metadata prefilter with racy-clean guard**: (migration v14) adds `file_size INTEGER` and `file_mtime TEXT` columns to `indexed_files`, populated during sync for every processed file. During startup sync, `maybeAutoSync` collects current file metadata (size via `os.Stat()`, mtime in RFC3339 format) BEFORE calling `reader.Hash()` on each discovered file. Freshness guarantee: skipped files have not changed since last index time; any modification (content, size, timestamp) re-triggers hashing. Conservative design: NULL metadata for pre-v14 rows means always re-hash. Racy-clean guard (git-style): files whose mtime is not strictly older than `last_indexed` (within 2-second granularity margin) are treated as potentially modified in the same timestamp tick and are always re-hashed, preventing silent data loss from same-length edits. Unchanged files (size + mtime both match, and mtime strictly older than `last_indexed`) skip re-reading entirely, reducing startup time on stable corpora. Truncations, replacements, and same-size-different-content edits are all reliably detected. The `isRacyCleanFile()` function accepts both SQLite CURRENT_TIMESTAMP raw format (`2006-01-02 15:04:05`) and RFC3339 (actual driver conversion): the indexed_files.last_indexed column stores raw text but modernc.org/sqlite converts DATETIME on read, so Go receives RFC3339; the sqlite3 CLI shows raw format, a trap that has fooled multiple reviewers, so verify through the driver.

## Dependencies

Expand Down Expand Up @@ -235,7 +236,7 @@ github.com/pablontiv/backscroll/internal/chunking — Token-aware text chun
github.com/pablontiv/backscroll/internal/embedding — Embedding provider interface, mock provider, and ONNX provider implementation
github.com/pablontiv/backscroll/internal/hybrid — Reciprocal Rank Fusion helpers for merged lexical/vector retrieval
github.com/pablontiv/backscroll/internal/sequences — F4 PrefixSpan mining (deterministic pattern discovery per session)
github.com/pablontiv/backscroll/internal/storage — Database schema, migrations v1–v13, FTS5 indexes
github.com/pablontiv/backscroll/internal/storage — Database schema, migrations v1–v14, FTS5 indexes
github.com/pablontiv/backscroll/internal/projects — Project identity registry
github.com/pablontiv/backscroll/internal/readers — SessionReader interface, Registry, ClaudeReader (text+tool_use+tool_result), PiReader (text+toolCall+custom results), OpenCodeReader (text+tool state.input+state.output), MarkdownDocumentReader (`markdown_document`), MarkdownSectionsReader (`markdown_sections`); toolfmt serializer
github.com/pablontiv/backscroll/internal/recovery — Stranded database recovery orchestration, durable backup, atomic replacement, and post-install sync
Expand Down
Loading
Loading