Repository navigation
feat!: libscope 2.0 — one operation layer for CLI, MCP, REST and SDK - #515
Merged
Merged
Conversation
- Add confirmOrCancel() to cli/confirm.ts and use it at all 13 confirm sites - Extract addBulkFilterOptions/addBulkRunOptions/selectorFromOpts/runBulk for bulk delete/retag/move - Use initializeAppWithEmbedding() in connect obsidian - Use static connector-config imports in schedule set/remove - Add splitCsv() for identical comma-split option parsing
- mcp: add runMaybeAsync() so each async-capable tool formats its output once - mcp: add internal textResult() helper and use ToolResult instead of inline types - mcp: collapse duplicate cancelled branches in startAsyncTask; drop dead fallbacks - api: replace hand-written match*/is*Path/dispatch* helpers with a pattern route table - api: remove dead placeholder/never-read state in spider handler
- confluence: compute html.toLowerCase() once per helper call - obsidian: build the vault file map once per sync instead of per file - obsidian/onenote: call createTopic directly (ON CONFLICT returns existing) - slack/spider/repo/onenote: use node:timers/promises setTimeout for delays - url-fetcher: fetchAndConvert reuses fetchRaw; share getInsecureAgent with http-utils - graph/docs: has-only Maps become Sets - rag: extract shared retrieval helper - config: truthy() and LLM_PROVIDERS helpers in getEnvOverrides - lite/normalize: file and buffer inputs share one path - parsers: MarkdownParser extends PlainTextParser
- search: shared doc column list, rating join helpers, base zod row schema with .extend, toResult/vectorRowToResult mappers, runFtsQuery for the FTS5 AND/OR-fallback queries; hoist zod schemas to module scope; drop unused vectorSearch offset param - documents/topics/tags: DOC_COLUMNS + rowToDocument, TOPIC_COLUMNS + rowToTopic - add deleteChunkEmbeddings(db, docIds) and use it in documents, indexing, watcher, packs and ttl with each site's existing error handling - registry/publish: reuse readPackFile from core/packs - registry/resolve: remove unreachable manifest alt-path fallback - packs: drop redundant batchInstalled counter; tags: dedupe STOPWORDS
search-docs results now show a Document ID / Chunk ID line, link-documents prints the created link's ID, and get-document-links shows each link's ID, so get-related, rate-document, and delete-link are usable from tool output. The link-documents enum now comes from LINK_TYPES in core/links.ts and accepts "references".
pino defaults to stdout, so the MCP stdio server interleaved JSON log lines (DB open, migrations) with protocol messages. initLogger now takes a destination option (default stderr); MCP main requests stderr explicitly. The CLI configures logging before loading config so validation warnings honour the CLI log level.
…ry path fireWebhooks had no callers, so webhooks never fired. Add a small core/events module; indexDocument, updateDocument, deleteDocument, rateDocument and searchDocuments emit document.created/updated/deleted, document.rated and search.executed. Webhooks subscribe to it. Emission never throws into the caller and never waits on delivery. Add deliverWebhook(webhook, event, data): decrypts the stored secret, signs, re-runs the SSRF check and POSTs with redirect: "error". Used by fireWebhooks, the CLI 'webhooks test' command and the REST test route, which previously signed with the encrypted secret (and the CLI skipped the SSRF check and followed redirects).
The FTS backfill ran after every migration, so each schema upgrade re-inserted the whole corpus into chunks_fts and doubled keyword hits. Backfill now runs only when chunks_fts has no rows. runMigrations takes an optional target version so tests can build older schemas.
getEnvOverrides spread DEFAULT_CONFIG sections into the override layer, so setting any LIBSCOPE_* variable reset sibling keys from config files to defaults (e.g. file provider=openai + LIBSCOPE_OPENAI_API_KEY gave provider=local). The env layer now contains only variables that are set.
…t 50 syncDocSite built its existing-URL set from listDocuments, which applies a default LIMIT 50 and loads full content. Sites with more than 50 pages had unchanged and changed pages miscounted as new. Query the url column directly with no limit.
The root program's .version() registers a global --version option, so `add/import/import-batch/docs update/pack install/pack create/registry publish --version X` printed the CLI version and exited, and `registry unpublish` (required --version) could never run. - Document metadata flag: --version -> --lib-version (add, import, import-batch, docs update) - Pack/registry flag: --version -> --pack-version (pack install, pack create, registry publish, registry unpublish) - registry unpublish accepts <name>@<version>; --pack-version is now optional and takes precedence over the suffix - Only call program.parse() when the module is the process entry point, and export the program so tests can parse arguments without running it
The FTS delete and update triggers matched rows on the UNINDEXED chunk_id column, so deleting one chunk scanned the whole FTS table. Migration 18 recreates chunks_fts with rowid = chunks.rowid, makes the triggers delete by rowid, and repopulates the index once from chunks, which also removes duplicate rows left by earlier backfills. Column names are unchanged, so search queries are unchanged. Deleting a 40-chunk document at 20k chunks: 94 ms -> 0.15 ms.
indexDocument checks for an existing document with WHERE title = ? AND LENGTH(content) = ?, which scanned the whole documents table on every index call. Migration 18 adds idx_documents_title.
- Add signal?: AbortSignal to ReindexOptions, InstallOptions, and SpiderOptions; each checks throwIfAborted() between batches/pages. - MCP submit-document (single and spider), reindex-documents, and install-pack pass the task signal instead of throwing from onProgress. - installPack no longer hangs (with an unhandled rejection) when a batch callback throws inside its scheduler; it now rejects and rolls back the partially installed pack. - TaskRegistry.run records completed when work resolves (even after a late cancel), cancelled only when it rejects because of the abort, else failed. - Rename the reindex task type from reindex_library to reindex_documents.
Keys now resolve in loadConfig as LIBSCOPE_<P>_API_KEY > <P>_API_KEY > config file, for both embedding and LLM (OPENAI_API_KEY was accepted by validation but never read by providers). LIBSCOPE_ANTHROPIC_API_KEY no longer requires LIBSCOPE_LLM_PROVIDER, LIBSCOPE_OLLAMA_MODEL is read, validation checks the resolved config (incl. anthropic), the config-file key warning names the file and the real env vars, and config show masks keys. Docs: env var tables, .env.example (nothing loads .env), REST API auth header.
deleteTopic({ deleteDocuments: true }) deleted the documents (and, by
cascade, their chunks) but left their vectors in chunk_embeddings,
a vec0 virtual table with no foreign key. Delete the vectors first with
deleteChunkEmbeddings, in batches of 500 documents.
config set accepted only 3 keys while docs and error messages told users to set others, and saveUserConfig rewrote config.json from 5 known sections: it dropped the registries array, wrote every default, and did not restrict permissions. - config set <key> <value> accepts every key of LibScopeConfig (typed key table; booleans/integers coerced, enums validated). API keys are refused with the env var to use. - New config get, config unset, config path; show/get mask API keys. - saveUserConfig is read-modify-write: only passed keys are written, unknown keys (registries, hand-written values) are kept. - One raw reader/writer of ~/.libscope/config.json (readRawUserConfig / writeRawUserConfig, 0600 file, 0700 dir), used by registry/config.ts.
Fresh indexDocument embedded a title/library/version prefix plus the chunk and wrote chunk_embedding_metadata, but updateDocument, reindex, and pack install embedded the raw chunk and skipped the metadata row, so a chunk's vector depended on how its document was written. Add buildEmbeddingText, embedChunks, splitIntoChunks, and createChunkWriter in core/indexing.ts and use them on all four paths. updateDocument now also re-extracts document links when content changes, and re-embeds existing chunks in place when title, library, or version change.
…stry import-batch (core/batch.ts) and watch (core/watcher.ts plus the CLI's initial scan) read every file as raw UTF-8, so PDF/DOCX/EPUB/PPTX files were indexed as binary garbage. - batchImport now indexes through indexFile (same parser path as `add` and `import`) and honours its sourceType option, which was ignored. Unsupported extensions are skipped with a warning and counted in a new `skipped` field instead of failing. - FileWatcher and the CLI watch scan parse files with the matching parser via readWatchedFile(); .rst stays indexed as plain text, and unsupported extensions are skipped with a warning. - Default extension sets for import, import-batch (--filter) and watch now come from getSupportedExtensions() instead of hand-written lists. - indexFile accepts an optional sourceType override.
import-batch called node:fs globSync, which only exists in Node 22+, so the
command crashed on Node 20. Replace it with recursive readdirSync plus a
shared globToRegExp helper (supports **, *, ?, {a,b}), and use the same helper
for pack-create exclude patterns and the repo indexer's include/exclude
globs instead of two separate copies.
The REST API and the CLI link command kept their own link-type lists that omitted "references", which core accepts. Use the exported LINK_TYPES.
database.path always had a default (~/.libscope/libscope.db), so the SDK opened that file while CLI/MCP opened the workspace database, the SDK workspace option did nothing, init/stats reported the wrong file, and connect/disconnect onenote built join(wsPath, absolutePath). - resolveDatabasePath(): explicit path (SDK dbPath, or database.path set in a config file, ~ expanded) > workspace (option, else active). - database.path has no default; used by CLI initializeApp, MCP main, LibScope.create (new dbPath option; workspace option now works) and the onenote commands (now use initializeApp*). - init prints db.name; getStats defaults to the open file (CLI stats, REST /health and /stats, SDK stats()). - If the workspace DB does not exist and ~/.libscope/libscope.db does, a single warning names it and how to keep it (config set database.path). Nothing is moved or deleted.
…er run Only 'connect slack' saved a named config, so 'schedule set notion' (the command's own help example) always failed, and scheduled runs wrote two connector_syncs rows (scheduler + connector) with drifted stats mappings. - Every 'connect <type>' saves a named config (connectors/<name>.json, 0600, default name = type, --name to override) with the credentials a later run needs and connectorType; an existing schedule is kept. - --sync on every connect command re-runs the saved config (obsidian's vault path is now optional with --sync). - Connectors record the run (new trackSync helper, optional syncName); the scheduler no longer tracks, so each run writes exactly one row under the configured name with the connector's own stats. A config that cannot be loaded records one failed row. - OneNote token refresh moved into connectors/onenote.ts (ensureOneNoteAccessToken), so scheduled runs refresh expired tokens and the new tokens are saved. Old connectors.json OneNote credentials are still read for --sync. - Schedule entries resolve the type from connectorType, so Confluence's cloud/server 'type' field no longer hides the connector type. - Slack: send lastSync as Unix seconds in 'oldest' (saved configs store ISO-8601, which Slack does not accept). - disconnect <type> also deletes the saved config (--name). Connector function signatures only gain an optional options argument. The DB connector_configs functions stay: core/index.ts exports them.
libscope/lite in 1.x indexed source files by path with the tree-sitter chunker; 2.0 rejected them as an unsupported format. indexFile now reads such files as UTF-8 when a chunker is set and indexes them if the chunker returns chunks.
Directory ingest replaces batchImport; searchBatch had no callers after the root exports shrank.
…provider plugin registry registerProvider was not exported in 2.0 (custom providers are passed as an instance), so the open-enum config path for plugin provider names is removed.
Titles taken from <title> or <h1> kept references such as —, so a URL add, a spider crawl and the docs connector stored titles like "textwrap — Python documentation".
…n indexed disconnect of a docs connection deleted every document whose URL started with the site URL, including pages added with `libscope add <url>`. The URL was also matched with LIKE, so "_" and "%" in it acted as wildcards. Match the prefix exactly and only for pages the docs connector stored (source type library, added by the crawler).
…default export
Under ESM, import() of the CommonJS grammar packages returns module.exports as
the default export, so TreeSitterChunker passed { typescript, tsx } to
setLanguage and every TypeScript chunk failed with "Invalid language object".
The docs and the install hints now pin tree-sitter@0.21 and the matching
grammar versions: the latest grammars need a newer tree-sitter and npm refuses
them next to libscope's peer range. The code-indexing example no longer nests a
template literal inside its template-literal source string.
When no chunk contains every query word, the keyword search retried with any word. "how to authenticate" then matched every chunk containing "to", and the fused ranking put those chunks above the semantic match. The OR retry now skips stopwords (shared with tag suggestions in core/stopwords.ts) unless the query has nothing else.
…s the clone publish --submit created feature/add-<pack> in the local registry clone and then ran a plain `git push`, which fails for a new branch without an upstream (git's default settings). The cleanup then ran `git checkout main`, which fails for a registry whose branch is master, so the clone stayed on the feature branch with a dirty tree; after a successful --submit it also stayed there, so the next normal publish went to the feature branch. Push the commit explicitly to refs/heads/feature/add-<pack>, always check out the clone's previous branch again and delete the local feature branch, and use the published version (not the pack file's) in the default commit message.
…-submit `registry create` committed on git's default branch (often master), so the documented `git push -u origin main` failed and the remote stayed empty. `registry publish --submit` ran a plain `git push` on the new feature branch, which has no upstream, and failed before its own `push -u`. commitAndPush now pushes HEAD with -u, and the clone returns to its base branch after a submit.
Titles taken from <title> and <h1> kept their HTML character references, so https://docs.python.org/3/tutorial/ was indexed as "The Python Tutorial — Python 3.14 documentation". URL ingestion, the spider and the docs connector now decode numeric and common named references.
- Python SDK quick start: create the "python" topic before add_text uses it (the server answers 404 for an unknown topic); note that ask needs an LLM. - README SDK example: note that ask needs an LLM. - Programmatic usage: askStream throws a ConfigError without an LLM, including with passthrough. - Connectors: a --schedule-only connect works on a saved connection; a new one also needs its settings. - REST reference: a task result string starts with "kind". - Architecture: renameTopic exists in core/topics.ts; only the operation is hypothetical.
…design Both verification branches fixed HTML entities in titles and publish --submit; keep the first (already merged) implementation, keep this branch's registry create on main, and make commitAndPush push new branches with -u origin HEAD.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
…e addons Lint/typecheck, the npm publish job and the Docker builder stage need no native addon. The test job rebuilds better-sqlite3 and sharp (loaded by @xenova/transformers in the neural retrieval suite); the build job rebuilds better-sqlite3 for docs:check. Sort the apt package names in the Dockerfile.
Format the blocks of a file and the four doc files with Promise.all instead of awaiting in a loop, and use String.raw for the escaped pipe. Output, problems and exit code are unchanged.
AsyncLibscopeClient.wait_for_task no longer takes timeout; wrap it in asyncio.wait_for or asyncio.timeout. The synchronous client keeps timeout. Use a constant for the /tags path.
- Replace await-in-loop (S9382) with shared helpers in src/utils/async.ts (forEachSequential, mapSequential, runConcurrent) or recursion for pagination, retries, polling and crawl loops; order is kept everywhere. - Rewrite super-linear regexes (S8786) as linear scanners with the same matches: Obsidian wikilinks, first markdown heading, trailing slashes and embedded credentials in registry URLs. - Split connectSettings, syncConfluenceSpace and globToRegExp (S3776). - Remove nested ternaries and nested template literals (S3358, S4624). - Use top-level await in the CLI and MCP entry points (S7785); the CLI still runs only when it is the process entry point. - Combine Array#push calls, use Number directly, a Set for watch extensions, replaceAll, a readonly scheduler map and an async wrapper instead of a trivial promise.
New test files for the async helpers, the wikilink and heading scanners, registry URL credential checks, glob brace edge cases, connect settings per connector type, paged document views and OneNote device-code polling.
Escape the Unicode line separators in markdown.ts, pass promises to Promise.all in gen-docs, and keep one call inside pytest.raises.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the libscope 2.0 redesign. Every action is defined once, in a shared operation layer (
src/core/operations/, 61 operations, each with one zod input schema). The CLI, MCP server, REST API, web dashboard server and Node SDK are thin adapters over that layer, so their names, defaults and validation can no longer drift apart. Backward compatibility was traded for simpler code: old names are removed without aliases, anddocs/migration-v2.mdlists every old name and its replacement.Diff against
main: 276 files changed, +26,843 / −19,558 lines, in 95 commits. None of the commits has aCo-authored-bytrailer.What changed
config setkeeps theregistrieslist.--versionflags were renamed to--lib-versionand--pack-version.reindex --rebuild.config get/set/unset, theLIBSCOPE_<SECTION>_<KEY>env vars and the generated docs.embedding.{provider,model,url,dimensions}andllm.{provider,model,url}.llm.providerdefaults toauto: passthrough under MCP, otherwise whichever API key is set.~/.libscope/secrets.json(mode 0600).src/cli/index.tswent from 3,194 lines to 129.addhandles files, folders, URLs, crawls and repos.searchwith no query opens interactive search.doctorreplacesinit.admin: reindex, dedupe, backup, restore, stats and prune.connect <type>,sync,disconnectandconnections.--jsonand--verbose. Errors print✗ messageplus a hint for the next step.mcp.toolsets).createMcpServer()has no side effects on import.LibScopeis the only public entry point, with namespaces such asscope.docsandscope.packs.libscope/liteis now a preset ofLibScopeinstead of a separate class.serve api, stores its data in the volume, and usesnode:22-bookworm-slim, because the local embedding model does not load on Alpine.npm run docs:gen). CI runsdocs:check. The README and guides were rewritten for 2.0.Verification
test:coverage(2,035 tests; 91% statements, 83% branches), build anddocs:checkall pass. Node 22 was run on an earlier tip of this branch.--branch/--pathand tree URLs);registry publish --submitagainst a local bare repo;lite-api/registry/rest-api, and the Python and Go READMEs was run or compiled. Examples that need credentials were checked against--helpand the generated references.Bugs found during verification were fixed with tests in this PR:
disconnectdeleting pages it did not index.registry publish --submitfailing onmasterand leaving the clone on the feature branch.registry createusing git's default branch.Not verified
Known issues not fixed here
add <github-url>needs--branchwhen the default branch is notmain.registry publish --pack-versiondoes not rewrite the version inside the pack file.logging.level.connector_configstable is kept; dropping it is pending a decision.package.jsonis still 1.8.0. Release-please should produce 2.0.0 from the breaking-change commits.