feat(indexing): partial re-index — skip unchanged files, add --force flag

[f2r]

Problem

Running dotmd index on an existing knowledgebase reprocesses every file from scratch on every run, even when nothing has changed. The dominant cost is the GLiNER NER model and the graph population loop, both of which run on all
chunks regardless of whether the underlying file was modified. On a corpus of a few dozen files this can take several minutes for a simple refresh.

Solution

This PR introduces a checksum-based partial re-index strategy while keeping the existing dotmd index interface unchanged.

How it works

1. After each successful index run, the MD5 checksum of every indexed file is persisted in a new files table in the SQLite metadata store.
2. On the next run, each file's current checksum is compared against the stored value. Files are classified as new, changed, or unchanged.
3. Only new/changed files go through the expensive pipeline steps: reading, chunking, structural extraction, GLiNER NER, key-term extraction, and graph population.
4. Files that no longer exist on disk have their chunks and checksum records cleaned up.
5. The BM25 index and LanceDB vector store are still rebuilt from the full merged corpus (unchanged chunks reloaded from SQLite + new chunks) so search quality is preserved.

--force flag

A --force / -f flag is added to dotmd index to bypass the checksum check and reprocess all files from scratch. Useful after changing --extract-depth, --entity-types, or the embedding model.

  dotmd index ./notes/            # partial re-index (default)
  dotmd index ./notes/ --force    # full reindex

Changes

• storage/metadata.py: New files table + save_file_checksums, get_file_checksums, delete_chunks_by_file methods
• ingestion/pipeline.py: Checksum diff logic; extraction/graph steps scoped to changed files only; force parameter
• api/service.py: Propagate force parameter
• cli.py: --force / -f option on dotmd index

Backward compatibility

• Fresh index (no stored checksums) → all files treated as new, identical behaviour to before.
• Existing indexes created before this change → fully reprocessed on first run, then partial re-index on subsequent runs.
• dotmd clear clears the new files table along with all other stores.

What is not changed

The LanceDB vector store still uses mode="overwrite" (full rebuild from the merged corpus). A future optimisation could use merge_insert() to update only changed vectors, but the embedding step is fast compared to NER and the
added complexity is not warranted here.

[f2r]

I want to be transparent: I did not test this against the full CLI before opening the PR, so I went back and added a test suite covering the changes.

Since the project has no existing test infrastructure and the ML dependencies (sentence-transformers, GLiNER, LanceDB) make end-to-end CLI testing impractical without a full environment, I split the validation into two layers:

1. SQLiteMetadataStore (stdlib only, no mocks)

Verified directly against a real SQLite database:

• files table is created on initialization
• save_file_checksums / get_file_checksums round-trip correctly
• Upserting a checksum after a file change stores the new value
• delete_chunks_by_file removes both the chunk rows and the file record
• delete_all also clears the files table

2. Pipeline diff logic (ML layers mocked)

Using unittest.mock to stub out SemanticSearchEngine, BM25SearchEngine, LanceDBVectorStore, LadybugDBGraphStore, and the extractors, with a real SQLiteMetadataStore wired in so checksum state is genuine:

• First run — all 3 files treated as new, 3 checksum records persisted
• No changes — extractor not called at all (zero new chunks)
• One file modified — extractor called exactly once, checksum updated
• --force flag — all files reprocessed regardless of checksums
• File deleted — its entry removed from the checksum store

One thing worth noting: FileInfo.checksum is a @computed_field that reads the file from disk on every access. This means the checksum comparison in the pipeline always reflects the current file content, which is the correct
behaviour — but it also means each dotmd index run reads every file once (for the checksum) before deciding whether to skip it. For very large knowledgebases this could be optimised by comparing last_modified + size_bytes first as
a fast pre-filter, falling back to MD5 only on a mismatch. I have not included that optimisation here to keep the change focused.