Skip to content

feat(import): batch and folder import, as a snapshot with honest counts - #172

Merged
mrsibe merged 1 commit into
mainfrom
feat/98-folder-import
Sep 28, 2026
Merged

mrsibe merged 1 commit into
mainfrom
feat/98-folder-import

Conversation

@mrsibe

@mrsibe mrsibe commented Sep 28, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

Batch and folder import (#98): add a folder recursively, import many files at once, and report honestly what happened to each.

Why?

Adding sources was one file at a time. Research material arrives as directories — a course folder, a paper collection, a project's docs — and importing 40 PDFs one dialog at a time is where a user stops importing. The folder import is explicitly a snapshot, not a watch; watching is #158.

Related issue

Fixes #98
Related to #95 (ingestion runs), #154

What changed

  • scanFolder() (src/main/services/ingestion/folderScan.ts): recursive, supported extensions only, hidden directories skipped, deterministic order, symlinks not followed. The extension list comes from FileParserService.supportedExtensions(), so the scanner and the loaders cannot drift apart.
  • KnowledgeService.addDocumentsFromPaths(): a batch where one file's failure does not abort the rest — each file still goes through addDocumentFromFile, so each gets its own ingestion run and failure record ([Feat] Ingestion runs: persistent pipeline state, retry, and non-destructive re-index #95). A path already in the notebook is skipped and reported, not imported twice.
  • addFolder() = scan + batch. Files added to the folder later are not picked up.
  • IPC: knowledge:add-folder, knowledge:add-files, knowledge:select-folder. The renderer's multi-select and drag-and-drop both go through the same batch path, so dedup and skip reporting are one implementation. Electron 39 removed File.path, so the drop handler resolves paths through webUtils in the preload.
  • UI: an "Add folder" item, a drop target on the source panel, and a summary — Imported N · skipped M (already imported) · failed K — instead of a single total.

How was this tested?

  • npm run typecheck — passes.
  • npm test — 401 pass, including folder-scan fixture tests (recursion, hidden dirs skipped, unsupported extensions skipped, case-insensitive, deterministic order).
  • npm run check:design — no violations.
  • npm run build — passes.
  • Eval baseline unchanged.
  • electron . --smoke-test — PASS (28 checks), including a real folder import that asserts unsupported files are skipped and that re-importing the same folder adds nothing (snapshot semantics).

Not verified

  • Drag-and-drop was not exercised in a running app (it needs a real drag from the OS); the batch path it calls is the same one the smoke test exercises, and webUtils.getPathForFile is the documented Electron 39 API. The markup is covered by typecheck and the design guard only.

Checklist

  • I have reviewed my own changes.
  • npm run typecheck passes.
  • npm run build passes.
  • I have tested the affected user workflow.
  • I have not included unrelated changes.
  • I have updated documentation when necessary.

Desktop / build changes

  • Not applicable (no schema or packaging change).

Adding sources was one file at a time, and research material arrives as folders
(#98).

- scanFolder(): recursive, supported extensions only, hidden directories skipped,
  deterministic order, symlinks not followed.
- KnowledgeService.addDocumentsFromPaths(): a batch where one file's failure does
  not abort the rest (each file still gets its own ingestion run, #95), and a path
  already in the notebook is skipped and reported rather than imported twice.
- addFolder() = scan + batch. A snapshot, NOT a watch: files added to the folder
  later are not picked up — that is #158.
- IPC knowledge:add-folder / add-files / select-folder. The renderer's multi-select
  and drag-and-drop both go through the same batch path, so dedup and skip reporting
  are one implementation. Electron 39 removed File.path, so the drop handler resolves
  paths through webUtils in preload.
- The source list reports "Imported N · skipped M (already imported) · failed K"
  instead of a single total, and gains an Add folder item and a drop target.

Verified: npm run typecheck; npm test (401 pass, incl. folder-scan fixture tests);
npm run check:design; npm run build; eval baseline unchanged; electron . --smoke-test
PASS (28 checks), including a real folder import that asserts unsupported files are
skipped and that re-importing the same folder adds nothing.
@mrsibe mrsibe added enhancement New feature or request area:ux Interface, workflow, information architecture labels Sep 28, 2026
@mrsibe
mrsibe merged commit 9143ba7 into main Sep 28, 2026
4 checks passed
@mrsibe
mrsibe deleted the feat/98-folder-import branch September 28, 2026 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ux Interface, workflow, information architecture enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feat] Batch and folder import

1 participant