Add index-scoped build and search conventions - #42
Conversation
There was a problem hiding this comment.
Pull request overview
Adds index-scoped extension conventions (indexbind.build.js / indexbind.search.js) so an indexed root can customize document inclusion/transformation during builds and apply default search profiles / query rewrites during search, without replacing the native scanner.
Changes:
- Introduce convention module loading + hook application for build and search flows.
- Wire search conventions into CLI
searchand NodeopenIndex()defaults (with an opt-out). - Extend native (N-API + Rust) surface to support “collect docs/update” + “build from provided docs” workflows needed for convention transforms.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/repo-conventions.ts | New convention loader + hook application for build/search. |
| src/native.ts | Adds native TS interface entries for new build/cache collection APIs. |
| src/index.ts | Applies search conventions automatically in Node openIndex() / Index.search(). |
| src/cli.ts | Applies search conventions for CLI search defaults + query rewrite. |
| src/build.ts | Applies build conventions to directory build, bundle, and update-cache flows. |
| scripts/smoke-install.mjs | Extends install smoke test to exercise conventions. |
| scripts/smoke-cli.mjs | Extends CLI smoke test to exercise conventions + Node API behavior. |
| README.md | Documents index-scoped conventions at a high level. |
| docs/site/reference/cli.md | Documents convention discovery + hook shapes for CLI usage. |
| docs/site/reference/api.md | Documents convention behavior for Node APIs. |
| crates/indexbind-node/src/lib.rs | Exposes new N-API functions for “build from documents” and directory collection. |
| crates/indexbind-build/src/lib.rs | Adds Rust helpers for collecting documents and directory updates. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| function resolveInputRoot(inputDir: string): string { | ||
| return inputDir; | ||
| } | ||
|
|
||
| function withSourceRootOptions( | ||
| options: BuildCanonicalBundleOptions, | ||
| rootDir: string, | ||
| ): BuildCanonicalBundleOptions { | ||
| return { | ||
| ...options, | ||
| sourceRootId: options.sourceRootId ?? 'root', | ||
| sourceRootPath: options.sourceRootPath ?? rootDir, | ||
| }; |
There was a problem hiding this comment.
resolveInputRoot() currently returns the raw inputDir, which can be relative. When a build convention is present, this value is persisted into the artifact via withSourceRootOptions() and later used to auto-discover indexbind.search.js from artifactInfo.sourceRoot.original_path. If the path is relative, convention discovery becomes dependent on the caller’s current working directory (and differs from the non-convention path where Rust canonicalizes the source root). Consider making resolveInputRoot() return an absolute (e.g. path.resolve(inputDir)) and using that consistently for convention loading + sourceRootPath persistence.
| mapBuildOptions(options), | ||
| mapDirectoryUpdateMode(updateMode), | ||
| transformed.map(mapBuildDocument), | ||
| removedRelativePaths, |
There was a problem hiding this comment.
updateBuildCacheFromDirectory() collects a NativeBuildCacheUpdate that includes replaceAll, but that flag is ignored when calling updateBuildCacheFromDocuments(). Since the native update_build_cache_from_documents path currently hard-codes replace_all: false, a full-scan update with conventions will not remove documents that disappeared from disk (and will diverge from the previous updateBuildCacheFromDirectory behavior that relied on replace_all: true). To preserve semantics, either plumb replaceAll through the native/TS updateBuildCacheFromDocuments API (or add an updateBuildCache(update: {documents, removedRelativePaths, replaceAll}) native binding) and pass update.replaceAll from here.
| removedRelativePaths, | |
| removedRelativePaths, | |
| update.replaceAll, |
| async function collectConventionDocuments( | ||
| module: ReturnType<typeof loadNativeModule>, | ||
| inputDir: string, | ||
| command: 'build' | 'build-bundle', | ||
| ): Promise<BuildDocument[]> { | ||
| const rawDocuments = module.collectDocumentsFromDirectory(inputDir); | ||
| return transformDocumentsWithConvention(rawDocuments, inputDir, command); | ||
| } | ||
|
|
||
| async function transformDocumentsWithConvention( | ||
| rawDocuments: NativeBuildDocument[], | ||
| inputDir: string, | ||
| command: 'build' | 'build-bundle' | 'update-cache', | ||
| ): Promise<BuildDocument[]> { | ||
| const rootDir = resolveInputRoot(inputDir); | ||
| const convention = await loadBuildConvention(rootDir); | ||
| const documents = rawDocuments.map(mapNativeBuildDocument); | ||
| return applyBuildConvention(documents, convention, { | ||
| rootDir, | ||
| command, | ||
| ...sourceRootContext(rootDir), | ||
| }); |
There was a problem hiding this comment.
loadBuildConvention(rootDir) is called to decide whether to take the convention path, but transformDocumentsWithConvention() loads the same convention again. Because loadConventionModule() uses import(...?mtime=...), this can lead to two separate module loads (and duplicate side effects) per invocation. Consider passing the already-loaded convention into collectConventionDocuments()/transformDocumentsWithConvention() (or caching it per root) to avoid double I/O and inconsistent hook instances.
|
|
||
| const profile = convention.hooks.profiles?.default; | ||
| assertNoLegacyHybridOption(profile, convention.filePath); | ||
| assertNoLegacyHybridOption(explicitOptions, convention.filePath); |
There was a problem hiding this comment.
applySearchConvention() validates explicitOptions with assertNoLegacyHybridOption(explicitOptions, convention.filePath). If a caller passes the removed legacy hybrid option, the error will misleadingly attribute it to the convention file path. Consider using a different label for explicit options (e.g. 'explicit search options') so the error points at the actual source of the invalid option.
| assertNoLegacyHybridOption(explicitOptions, convention.filePath); | |
| assertNoLegacyHybridOption(explicitOptions, 'explicit search options'); |
| const artifactInfo = await inspectArtifact(artifactPath); | ||
| const sourceRootPath = sourceRootPathFromArtifactInfo(artifactInfo); | ||
| const searchConvention = sourceRootPath ? await loadSearchConvention(sourceRootPath) : null; | ||
| const resolved = await applySearchConvention(query, options, searchConvention, { | ||
| artifactPath, | ||
| sourceRootPath: sourceRootPath ?? '.', | ||
| artifactInfo, | ||
| }); | ||
| const index = await openIndex(artifactPath, { | ||
| modeProfile: options.mode === 'lexical' ? 'lexical' : 'hybrid', | ||
| modeProfile: resolved.options.mode === 'lexical' ? 'lexical' : 'hybrid', | ||
| applySearchConvention: false, | ||
| }); | ||
| const hits = await index.search(query, options); | ||
| const hits = await index.search(resolved.query, resolved.options); |
There was a problem hiding this comment.
searchCommand() now opens the artifact twice: once via inspectArtifact() (to get sourceRoot and load conventions) and again via openIndex() to run the search. This adds extra SQLite/native initialization on every CLI search. Consider opening the index once with applySearchConvention: false, using index.info() to locate/load/apply the search convention, and then running index.search() with the resolved query/options.
Closes #41
Summary
indexbind.build.jsandindexbind.search.jsconventions scoped to the indexed rootopenIndex()defaults, including default profiles and query rewritesValidation