Skip to content

feat(watch): a watched folder keeps itself current, and never deletes - #173

Merged
mrsibe merged 1 commit into
mainfrom
feat/158-folder-watch
Sep 28, 2026
Merged

mrsibe merged 1 commit into
mainfrom
feat/158-folder-watch

Conversation

@mrsibe

@mrsibe mrsibe commented Sep 28, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

Watched folders (#158): mark a folder as a live source. New files are imported, changed files are re-indexed, and deleted files are marked missing — without deleting anything.

Why?

A folder import (#98) is a snapshot: a paper collection that keeps growing falls out of date the moment it changes. This is the "keep it current" counterpart, and it is deliberately a separate code path — the snapshot never runs on a timer, and the watcher never re-implements parsing.

Related issue

Fixes #158
Related to #98, #95, #154

What changed

  • Schema (migration 0021): documents.source_state (available / missing / changed) and source_mtime_ms; plus a folder_watches table (unique per notebook + path). source_state is separate from status on purpose: a deleted file is missing while its index, notes, excerpts and citations all stay.
  • diffFolder() (folderDiff.ts, pure): decides added / changed / restored / missing. Being pure, the decision table is unit-tested without a filesystem or a database — and "changed" is an mtime comparison, not a guess. A row with no recorded mtime counts as changed, so the first reconcile after this upgrade does not treat old rows as current.
  • FolderWatchService: reconciles on launch (a change made while the app was closed is not missed) and live via fs.watch with an 800 ms debounce. New files go through the same batch import as [Feat] Batch and folder import #98; a changed file is re-parsed into the same documentId, because a plain reindex would reuse the stored content — exactly the thing that changed.
  • UI: the source list shows the missing state; the Add menu gains "Watch folder".

How was this tested?

  • npm run typecheck — passes.
  • npm test — 415 pass, including 8 diffFolder cases (added / changed / restored / missing, no-mtime rows, and all four computed together).
  • npm run check:design — no violations.
  • npm run build — passes.
  • Eval baseline unchanged.
  • electron . --smoke-test — PASS (29 checks), including a real watched folder that imports a newly added file, then marks a deleted file missing while asserting its chunks survive.

Not verified

  • The live fs.watch path was not observed firing in a running app; the reconcile it triggers is what the smoke test exercises. If a platform refuses recursive: true, the watcher falls back to reconcile-on-launch (the catch is explicit in FolderWatchService).

Checklist

  • I have reviewed my own changes.
  • npm run typecheck passes.
  • npm run build passes.
  • I have tested the affected user workflow.
  • I have not included unrelated changes.
  • I have updated documentation when necessary.

Desktop / build changes

  • Database schema changed (additive: two documents columns + folder_watches; migration 0021).

#158. A folder import is a snapshot; this is what keeps it current.

- documents.source_state + source_mtime_ms (migration 0021) separate the SOURCE
  file's state from the index state: a deleted file becomes `missing` while its
  index, notes, excerpts and citations all stay.
- folder_watches persists the watch (one row per notebook + folder).
- diffFolder() decides added / changed / restored / missing. It is pure, so the
  decision table is unit-tested without a filesystem or a database — and "changed"
  and "missing" are determined, not guessed.
- FolderWatchService reconciles on launch (so a change made while the app was closed
  is picked up) and live via fs.watch with an 800ms debounce. New files go through
  the same batch import as #98; a changed file is re-parsed into the SAME documentId
  (a plain reindex would reuse the stored content — exactly the thing that changed).
- The source list shows the `missing` state; the Add menu gains "Watch folder".

Verified: npm run typecheck; npm test (409 pass, incl. 8 diffFolder tests);
npm run check:design; npm run build; eval baseline unchanged; electron . --smoke-test
PASS (29 checks), incl. a watched folder that imports a newly added file and marks a
deleted one missing without dropping its chunks.
@mrsibe mrsibe added enhancement New feature or request area:foundation Module boundaries, architecture, tech debt area:provenance Source location, citations, document structure labels Sep 28, 2026
@mrsibe
mrsibe merged commit f5ec15f into main Sep 28, 2026
4 checks passed
@mrsibe
mrsibe deleted the feat/158-folder-watch branch September 28, 2026 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:foundation Module boundaries, architecture, tech debt area:provenance Source location, citations, document structure enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feat] Folder as a live source: watch, auto re-index, and mark stale

1 participant