Skip to content

feat: configurable max concurrency for document ingest#467

Open
472869879 wants to merge 2 commits into
nashsu:mainfrom
472869879:main
Open

feat: configurable max concurrency for document ingest#467
472869879 wants to merge 2 commits into
nashsu:mainfrom
472869879:main

Conversation

@472869879

Copy link
Copy Markdown

Summary

Replace the single-threaded ingest queue with a configurable semaphore allowing up to N documents to be processed in parallel.

Changes

  • src/lib/ingest-queue.ts: Global processing boolean replaced with activeCount semaphore. Each concurrent task gets its own AbortController (Map<taskId, AbortController>). cancelTask() targets a specific in-flight task. cancelAllTasks() and pauseQueue() abort all active controllers.
  • src/stores/wiki-store.ts + src/lib/project-store.ts: ingestConcurrency added to GeneralConfig. Default: 1 (backward compatible). Persisted to app-state.json with 1-16 validation.
  • src/components/settings/sections/general-section.tsx: Number input (1-16) under General settings.
  • src/components/settings/settings-types.ts + settings-view.tsx: Draft/save flow for the new field.

Design

  • Default = 1: Zero behavior change for existing users.
  • Per-task abort: Each concurrent ingest gets an independent AbortController.
  • Sweep-on-drain: onQueueDrained only fires when all active tasks complete and no pending tasks remain.
  • Stale-context guard: All existing project-switch protections are preserved.

Testing

  • 33/33 unit tests pass (ingest-queue.test.ts)
  • 12/12 integration tests pass (ingest-queue.integration.test.ts)
  • TypeScript typecheck passes
  • Vite frontend build succeeds
  • Tauri/Rust release build succeeds (Windows MSVC)

472869879 and others added 2 commits June 23, 2026 23:59
Replace single-threaded ingestion queue with a configurable semaphore (1-16). Each concurrent task gets its own AbortController and file-tracking. Defaults to 1 for backward compatibility. Settings UI added under General section.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant