fix: Bun SQLite boot and first-search auto-index - #33
Merged
Conversation
Pi was crashing at import time because Bun has no node:sqlite. Search now builds an empty index in-process so agents get hits instead of exit 2.
Keep only the four clone-required scripts. Move the cpu-limit test under tests/. Drop campaign docs and process gates a clean clone does not need.
Delete the behavioral suite, fuzz tree, and testkit crate. CI and contributor gates now compile the shipped libs, bins, and Pi package.
Keep the tests that prove those features work. Leave campaign fuzz, benches, keep-gates, and process suites deleted.
Remove crate-source #[cfg(test)] path stubs and tests/unit. Search, index, and Pi intent suites stay in tests/ and use ast-sgrep-testkit.
`--lang ts` compared against stored `typescript` and returned ok-empty. Map aliases at Searcher/Indexer construction and in the SQL bind.
Owner
Author
|
Follow-up on this branch: |
Search now incrementally indexes on query unless --no-auto-index, so edits are visible without a separate index run. --lang aliases use the same extension table as detect_language, including h/hpp and the rest.
Stop the CI workflow from starting on every pull_request push. Dispatch it from the Actions tab when a GitHub matrix is actually needed.
Owner
Author
|
Pushed two follow-ups:
|
Warm distinct-query p50 on the self corpus drops ~28% (3.8 -> 2.7 ms single-term, serve surface); identical-repeat cache hits stay at ~0.1 ms. Two levers, each measured in isolation: - literal_trigram: the SQL ORDER BY forced SQLite to materialize the whole trigram doclist into a TEMP B-TREE before yielding row 1, defeating the lazy budget break (EXPLAIN QUERY PLAN: USE TEMP B-TREE FOR ORDER BY). Drop the ORDER BY, stream candidates in posting order, stop at the retained budget, and restore (path, line_no) ordering in Rust over the small candidate set. Ordering contract preserved by construction; only >=budget overflow subsets shift (same class as the pre-existing lazy cut). - lexical_from_field: rank inside the FTS table (it already stores file_id/line_no/content), then resolve identities for the <=limit surviving rows via one bounded files IN-list lookup instead of two per-row joins over every candidate line. 35-contract golden battery (literals, word, regex incl. metachars, defs/callers/imports, pattern:, unicode, CRLF, no-EOL, lang/file filters, no-hit): all fixture cases byte-identical; the four diffs are >=16-hit overflow subsets whose membership shifts by posting order.
Two failure-first pairs (red proven on pre-fix code, green from production-only change, assertions frozen): - word: queries post-filter whole-word boundaries AFTER the SQL window; substring-only rows consumed every window slot, silently dropping real matches deeper in path order. Over-fetch a bounded 16x multiple in word mode so the boundary filter has candidates. Test: literal_word_limit_window (150 'alphabetic' lines before the lone standalone 'alpha' at line 151; pre-fix returned 0 hits). - required_literal harvested character-class CONTENT as a trigram prefilter literal ([]abc] -> "abc", [a\]bcd]efg -> "]efg"), dropping lines the regex genuinely matched. Classes are alternatives, not required text; bail conservatively (None) instead. Also removes the same false-negative class from the codemod memchr prefilter via required_pattern_literal. Tests: regex_class_literal. Targeted neighbors green: literal_glob, literal_diff, regex_budget, pattern_diff, code_prose_fields (14 tests).
…rd two measured perf closes
split_once(" AND ") fired on the first separator even when it sat inside
a quoted channel payload. word:helper AND literal:"cats AND dogs" then
left rhs containing " AND ", tripping the v1 two-channel bail and
silently falling through to hybrid search (empty results, no error).
The separator scan now skips double-quoted spans; a quoted " AND " is
payload bytes, not a channel boundary. Regression tests live in the
conjunction_queries suite (br-9kb).
SemanticCache validated only local meta counters (max_id, index/ semantic data_version, lang_filter, embed_backend). A FOREIGN raw-SQL mutation through a separate connection bumps SQLite's PRAGMA data_version while moving none of those counters, so search_semantic kept serving vectors for deleted chunks (br-yp1). The identity check now requires the connection's PRAGMA data_version to match the one captured at load; an unreadable pragma fails closed (the context is simply not cached). Regression test drives a targeted lower-id chunk delete through a raw second connection — counter-neutral by construction — and asserts the deleted chunk vanishes while the surviving chunk still resolves.
ebfaace rewrote the lexical FTS fallback SQL with a table alias (FROM lines_fts t WHERE t MATCH ?1). FTS5 resolves MATCH through the table-name pseudo-column, so the alias form fails with 'no such column: t' — the whole lexical FTS arm errored out whenever trigram did not answer first (exposed by tests/core/freshness_identity.rs, red on the pre-fix tree). Keep the join-free projection but always qualify MATCH with the real table name; projection columns drop the alias prefix.
run_serve answered every post-budget request with its own identical budget error forever — a flood that hides the outage instead of reporting it once, loudly, and stopping (br-r49). bump_call now returns typed CallError::BudgetExhausted, the session exposes exhausted(), and both serve arms (Call and Batch) answer the offending request exactly once and terminate run_serve with Err so the CLI process fails visibly. Regression test drives select past the 10k sticky budget: exactly one budget response, then Err.
Lever built exactly as the campaign note prescribed ((key,hit) pairs, never index-keyed arrays): 35-contract golden battery byte-identical. Measured across 8 interleaved codemode-serve rounds at two operating points (limit=8, limit=25): p50/p10 deltas within run-to-run noise. The prune branch rarely engages at real traffic shapes, so the comparator recomputes were never a measurable share of warm-path time. Reverted before commit; retry predicate is profiler-gated (>=5% of search_process_request in excerpt_term_coverage frames).
Lever 2 (SQL LIMIT on postings in literal_trigram_scan) measured a no-op: wall p50, trigram span share, and per-scan averages identical to base across interleaved A/B; goldens byte-identical. Postings probe shows why — the ebfaace hit-count break already bounds dense terms at ~100 rows and the corpus has no long-sparse doclists (max 4,874), so the population the cap would trim is empty. Reverted before commit; retry predicate: non-empty long-sparse tail on target corpus, or a two-phase deferred-join prototype measuring >=15% span reduction.
…-4 premise Lever 3 (lexical FTS fallback): measured 0.34ms avg per fired fallback, but lexical_pass is unreachable from codemode tools entirely — it serves MCP Keyword mode and one CLI path. Reclassified as surface-scoped MCP work with a profiler-gated retry predicate, not a campaign-metric lever. Lever 4 (symbol/caller batching): premise stale on HEAD — terms are already OR-batched into one LIKE statement per stage; direct SQL timings sit below timer resolution.
…S5 phrase machinery Three SQL-level prototypes measured: posting-cap (prior entry), deferred rowid-join past LIMIT (-3%: joins are effectively free), and subset-trigram MATCH (~15% total only with lucky rare-trigram picks; blind picks regress up to +8ms on a single term). Scan cost is flat in doclist size and scales with term length — the cost center is FTS5 phrase intersection itself. Recorded as an Open pointer gated on trigram df-metadata availability.
Zero-posting miss queries complete the full pipeline in ~0.156ms median: fixed overhead is already far below the sub-1ms budget, so remaining gains must come from volume-dependent cost. Cross-links the trigram-scan attribution entry.
Response-finishing audit: no high/medium defects; one LOW determinism gap in cmp_ranked_hits tie-breaks (br-23f) against the MCP byte-stability contract. Codemod edit-path audit: four MEDIUM crash/race-window defects (TOCTOU concurrent-writer overwrite, empty-path crash window with no recovery, rollback remove-then-rename destruction, final- component symlink swap) plus four LOW findings; all tracked as beads, fixes gated on the fault-injection harness (br-d77) so they can be proven failure-first. Full ruled-out checklists included.
br-hbd: plan reads are O_NOFOLLOW but apply verification followed final-component in-root symlinks, so a file swapped for a relative symlink between plan and apply passed verification, was renamed into the backup slot, and deleted by success cleanup - lying success, lost leaf. Apply now fails closed (symlink_metadata check before each swap). br-1xx: an apply killed between rename(source->backup) and rename(staged->source) left the canonical path missing with no way back; re-runs failed verification with ENOENT forever. plan_codemod now heals first: restores the newest orphaned backup for any planned path whose canonical file is gone, then sweeps stale stage/backup sidecars. br-bci: rollback no longer remove_file()s the new content before renaming the backup back; POSIX rename replaces atomically, removing the destroy-on-fault window inside rollback itself. RED->GREEN: new codemod_crash_windows suite proves both named failures on pre-fix code (symlink destroyed with Ok; canonical path still ENOENT after re-run) and passes post-fix; cli_smoke 14/14 stays green.
The literal trigram scan's cost grows with term length because FTS5 intersects every needle trigram. Pick the single rarest trigram and MATCH only its posting list; the existing content_matches_literal reverify keeps output exact (any candidate's posting list is a superset of true matches since candidates are always needle-derived trigrams). df source: an ephemeral temp fts5vocab virtual table over the live lines_trigram index - no persisted sidecar to drift from any writer path; memoized per store keyed on index_data_version, invalidated on generation bump, fail-safe to full-phrase MATCH on every uncertain outcome (non-ASCII fold ambiguity, lookup failure, vocab unavailable). Failure-first regressions in tests/core/trigram_shortcut.rs: - c1: shortcut hit-set equals a filesystem contains-oracle - c2: same-named decoy temp table with forged dfs is not trusted (RED-proven: forged dfs picked phantom trigrams -> silent empty) - c2b: post-warm forgery claiming a required trigram absent (df=0) must not flip output (RED-proven: [] vs [src/mod_0.py]) - c3: foreign raw-SQL delete/addition flips results despite warm memo Measured (benchmarks/results/speed.md::2026-08-23 trigram df): warm distinct p50 ~21% lower across interleaved A/B rounds (2.30-2.73 -> 1.96-2.23 ms), p10 -13%, p90 -8%, mixed-batch throughput +28% real calls at -22% avg/call, 35/35 golden battery byte-identical. Threshold tuning: 256 netted negative (+0.3 ms; gate above corpus p75), 4096 engaged everywhere and won; shipped 2048 as the bounded choice. Ledger: closes trigram-scan-cost-attribution predicate; records trigram-df-gate-too-tight-256 negative pointer.
This reverts commit 762df53.
Phase 1 lists depth-1 subroots (skipping ignored dirs); phase 2 walks each subtree on its own rayon thread with a per-thread IgnoreMatcher. The file set is partitioned by subtree, so it is identical to the serial walk by construction; oracle A/B on four declaration patterns returned identical hit sets (253-hit struct pattern byte-equal in (file,start,end)). Distinct structural pattern first-touch: ~80ms -> ~43ms steady. Warm distinct literal/hybrid p50 1.58-1.74ms, p90 7.4-7.6ms.
Phase-1 listing now enumerates to depth 2 and pushes grandchild directories as additional parallel work units (skipping any whose depth-1 ancestor was already pushed, preventing double coverage). Balances the rayon workload across more threads on wide subtrees. Verified: serial-vs-parallel oracle identical on healthy index; golden battery 35/35 byte-identical; pattern_routing/prefilter, trigram_shortcut, cli_smoke green.
Replace the two-phase depth-2 partitioning with a breadth-first traversal: each frontier directory is expanded on a dedicated 4-thread walk pool (ASGREP_WALK_THREADS to tune), files claimed exactly once per parent dir, child dirs form the next level. Coverage exact by construction; serial-vs-BFS oracle identical on four declaration patterns (253-hit struct set equal in (file,start,end)). Distinct structural pattern first-touch: ~43-48ms -> ~35-42ms (8 workers: 26-39ms; ASGREP_WALK_THREADS tunes the latency/CPU trade). Sustained load unchanged: 31,515 real calls/120s, zero errors. Golden battery 35/35 byte-identical; pattern_routing/prefilter, trigram_shortcut, finish_determinism, cli_smoke (14), codemod_crash_windows all green.
Three profiled levers targeting the cold-needle tail of warm distinct hybrid search (p99 14-20ms, max 22-26ms; pipeline floor 0.156ms): 1. store/sql.rs: read-path page cache 16MB -> 70MB so a serve session holds the whole ~58MB index resident; tail max drops to ~21ms and late-run needles flatten. 2. trigram_df.rs: bulk-preload the fts5vocab term->df map once per generation. fts5vocab point probes walk the full term index (~ms each, x3+ per needle); preload makes every probe a HashMap hit. 3. symbol.rs: quantize allowed-files IN-list placeholder count up to power-of-two buckets by repeating the last path (membership- equivalent), stabilizing prepare_cached statement text that per-count variance was thrashing. Measured on the 299-distinct-needle battery: p99 20.5 -> 18.9 ms, max 26.1 -> 20.7 ms across matched interleaved runs. Sustained load 28,562 real calls/120s, zero errors. Golden battery 35/35 byte-identical; trigram_df(5)/trigram_shortcut(4)/pattern_routing/ cli_smoke green. Honest scope note: first-touch high-df needles remain ~10-25ms — that cost is candidate-volume work bounded below by data volume, not removable overhead. Sub-1ms p99 for every cold needle would require an answer cache across sessions (rejected here as semantics-changing) or literal:-direct mode, which is already sub-ms. AGENTS.md has unrelated uncommitted working-tree changes from another session; deliberately excluded from this commit.
… candidate The hybrid cold-needle tail (13-25ms) was dominated by per-hit excerpt SQL inside the structural passes: every def/caller/anchor candidate fetched its indexed excerpt before fusion, then fusion discarded most of them. Move attachment out of the channel passes into finish, where it runs once on post-dedup hits before the first excerpt-dependent prune (cmp_ranked_hits' final excerpt tie-break requires excerpts to exist by then; dedup/margins/confidence/best_definition never read them). Byte-identity: golden battery 35/35 byte-identical on a freshly rebuilt index (an earlier DIFF reading was stale-index drift — both binaries agreed pairwise). High-df cold needles 24.9 -> 12.4 ms avg; tail battery p99 19.9 -> 17.3-18.8 ms, max ~23 ms; sustained load unchanged (27.2k calls/120s, 0 errors). pattern_routing(5)/trigram_shortcut(4)/ finish_determinism/cli_smoke(14) green.
calls_matching used WHERE lower(c.callee) = lower(?1), a full scan of every caller row (~20 ms) because the raw-column indexes cannot serve lower() expressions. Add idx_callers_callee_lower / idx_callers_caller_lower and bump user_version 12 → 13 so existing stores rebuild the DDL. Planner-only: same query text, same results. Incoming lookup 20.2 ms → 0.01 ms on the populated corpus. Chain JSON byte-identical on a migrated index. Never reuse SCHEMA_VERSION 13.
…reverify Round-11 keeps on the warm hybrid path, plus a latent IN-list bind bug found while probing them: - S1: generation-keyed snapshot_stamp memo (−7–10% p50 embed ON). - E1: skip embed-pass file loops when both semantic sources are empty. - B1: batched IN-list for semantic file fetch (statement-count scaling). - T1: push case-sensitive non-word trigram reverify into SQL GLOB (dense-scan tail p90 −16%). - Bugfix: IN-list buckets round UP with n.next_power_of_two().max(8) so 2^k+1 file sets (9, 17, 33…) bind instead of "Wrong number of parameters". Empty allow-lists become AND 0 = 1. Lexical/structural 35-contract goldens stay byte-identical. rustfmt-only touch-ups on pattern.rs / trigram_df.rs ride along.
A one-file edit paid a full 12-iter k-means rebuild (~46 s of a 48–58 s dir-mode delta) because reassign_all called build_from_flat and mark_semantic_ivf_stale deleted semantic.ivf first. Keep the sidecar on delta upsert/remove. reassign_all now nearest-centroid assigns every current vector onto the existing centroids and rewrites postings. Chunk-count drift is expected; bail only on dim mismatch, a missing sidecar, or empty centroids. Full wipe and asgrep reindex still drop the sidecar so k-means runs. Recall@10 on the 2048-vector fixture stays 0.998 after +1/+10/+50 appends (SLO 0.99); centroids byte-identical. 54k-chunk wall-time is not claimed here.
Close the measured round-11 rows (E1, S1, B1, T1, schema-13 callers indexes, IN-list bucket bugfix) and the scale retests that failed their retry predicates. Door A is landed as centroid-preserving reassign with recall numbers; 54k wall-time is still unmeasured. Door C (zero-weight field-fetch skip) is closed until the why contract drops those blobs.
Dir-mode one-function append on the 54k-chunk corpus: wall 48.1 s → 2.47 s; IVF span semantic_ivf_build 46.47 s → semantic_ivf_reassign 50 ms. No-op stays ~1.0–1.6 s. Cold asgrep reindex still runs k-means (50.2 s), as designed.
Unique-query semantic-only on the 54k-chunk corpus was ~202 ms because search fetched ~90% of concat+field blobs from SQLite. Rank probed members from the cached IVF mmap, SQLite-fetch only the top-N survivors (hit_limit.max(64)), and SELECT only intent-weighted field columns. Default nprobe stays 90% at n<=10_000 (recall@10 0.998 on the 2048 fixture); above that, cap at sqrt(k) in 16..=48. Sequential SIMD-dot member scoring, generation-keyed chunk-id+dim memo, header-only fingerprint peek, one-row dim instead of MAX(length(vector)). Literal intent keeps concat score and omits embed_field:* why terms (form 8). Hashed embed identity is preserved (alloc-free hash). Unique semantic-only p50 0.677 ms (was 201.9 ms); hybrid distinct still ~29 ms.
54k unique semantic-only p50 201.9 ms -> 0.677 ms. Close embed-channel-rescoring-fetch-scale as KEEP. Document mmap rank, weighted-field why, and large-n nprobe cap.
Keep docs/progress on disk for local campaign notes, but ignore it so the curated product docs stay the published surface.
Unique semantic on the 54k-chunk corpus is now p50 0.51 ms / p90 0.74 ms (n=85). IVF prefaults on first load and caps nprobe at 8 above 10k vectors. Hybrid scores only mmap rows in cascade files. Default hybrid (Pi asgrep.search / natural mode) was 25-48 ms unique because the cascade prefilter ran literal_sql LIKE '%0%' for 1-2 character tokens and then 500-row def/caller LIKE over 100 files. Prefilter now ignores short tokens, widens conceptual discovery with offline concept groups (credential -> auth/token), and skips def/caller LIKE for conceptual NL. Identifier queries keep the full structural pass. Unique hybrid is p50 1.27 ms / p90 8.4 ms on the same 54k shape; high-df conceptual tails remain (encode payload). Recall@10 on the 2048-vector fixture stays 0.998437. Cascade planner tests pass. AGENTS.md stays local-only.
.gitignore already ignores /AGENTS.md; the file was still tracked so local rewrites showed as dirty. Keep the on-disk copy untracked.
Schema 14 adds pattern_nodes(file_id, signature). Identifier hybrid seeks that index instead of INDEXED BY idx_pattern_nodes_file (a full-node scan of the 100 cascade files). Conceptual NL skips the whole structural stage: generic AST tokens (query/graph/render) owned the unique-hybrid p99 shortlist. Empty structural still falls through to lexical + embed. Unique hybrid on the 54k-chunk corpus: p50 1.08 ms / p99 3.9 ms (n=85; was 1.90 / 7.24). Cascade planner tests pass.
Keep PUSH-PROMPT.md and other internal notes on disk, untracked, same as docs/progress.
Drop the Worker sandbox isolate. Code Mode programs run in-process in node:vm; asgrep/console are built inside the context from a JSON host bridge so host Function cannot leak. Worker spawn was the activation serial wall. Model surface is search/find/read/edit. find is lexical (word:); blast:Symbol reverse-walks callers and blast:path uses imports. read batches indexed windows; edit is unique replace then targeted reindex. Promise.all still rides one warm CodeModeSession. ParallelMode::Auto stays serial-warm. Rebuild pi-ast-sgrep dist and drop sandbox-worker from the packed inventory. CLI sticky-serve (worker.ts) remains the degraded fallback.
Code Mode Searcher skips snapshot stamps, query expansions, and response-cache PRAGMA probes that capsule JSON never reads. CLI Searcher still stamps and still samples index generation before compute so concurrent reindex cannot mix snapshots. Warm unique hybrid p50 on the 54k corpus is ~0.56ms (HEAD ~1.26ms). Hit identity on the campaign smoke queries is unchanged. Also: ASCII case-insensitive literal match, IVF allowed-file invert, chunk-index fingerprint memo, AND of two rarest needle trigrams, generation-keyed line-count probe, combined IVF survivor fetch, and a unique-string edit that stops at the second match.
Lang-filtered IVF returns before refreshing the process-global chunk memo. Brute-force embed then reused the previous generation's backend/model after reindex in a long-lived process.
Owner
Author
|
Landed on this branch:
Unique hybrid (54k
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pi-ast-sgrepnow loads SQLite throughnode:sqliteon Node andbun:sqliteon Bun, so Pi/OMP/ZMP no longer crash withNo such built-in module: node:sqlite.asgrep search(and keyword/semantic/chain/call-path) incrementally indexes an empty checkout in-process on first use, then returns hits.--no-auto-index/ASGREP_NO_AUTO_INDEX=1keeps the old fail-closed exit 2.Agents often run default hybrid (
asgrep.search/mode: natural), not--semantic-only. This PR makes that path share the same coreSearcheras CLI, and makes unique semantic-only sub-1 ms on a 54k-chunk corpus (p50/p90). Unique hybrid is ~1 ms p50 / ~4 ms p99 on the same corpus.AGENTS.mdis local-only and is not in this PR. Campaign ledgers underdocs/progress/and internal notes underdocs/internal/are gitignored so curated docs stay the product surface.Schema, index, and query-path keeps
feat(store): schema 13 callers lower() indexes.calls_matching20 ms → 0.01 ms. Existing indexes migrate on next open.pattern_nodes(file_id, signature)for cascade structural seeks. Existing indexes migrate on next open. This mutates.asgrep/index.dbuser_version; pre-v14 binaries refuse it.2^k+1bind fix. Lexical/structural 35-contract goldens stay byte-identical.perf(index): keep IVF centroids on delta reassign. One-file edits no longer pay a full k-means rebuild.asgrep reindexstill rebuilds centroids. 54k one-file delta 48 s → 2.47 s.perf(search): rank IVF from mmap, fetch only top-N field columns. Search ranks concat vectors from the IVF mmap, then SQLite-fetches only top-N survivors and only intent-weighted field columns.Unique-query latency (54,732 chunks, hashed embed,
codemode-serve, limit 8)asgrep.search)Earlier on this line: unique semantic-only was 201.9 ms (Door A binary); unique hybrid was 48 ms (double FTS) then 25 ms (short-token
LIKE '%0%'), then ~7 ms p99 after file-scoped pattern fetch. Conceptual NL used to return 8×patternhits on generic AST tokens (query/graph/render).How:
LIKEscan). Conceptual discovery widens with offline concept groups (credential→auth/token). Gibberish still returns empty (cascade_stops_when_a_stage_has_no_survivors).pattern_nodes(file_id, signature)for cascade files instead of scanning every node in those files.process_request) keep it. Empty structural still falls through to lexical + embed. Form 8 hybrid JSON moves for conceptual queries (pattern shortlist no longer owns the keep-set).Recall@10 on the 2048-vector fixture stays 0.998437 (SLO 0.99) after reassign +1/+10/+50.
Not sub-1 ms: unique hybrid p50 is 1.08 ms; p99 is ~4 ms, the same cluster as unique semantic p99. IVF mmap search is tens of microseconds after warmup. First hybrid query still pays vocab preload + IVF prefault (~150–220 ms). Use
asgrep.semantic/asgrep semanticwhen the query has no token overlap and you want the 0.73 ms p90 path.pi-ast-sgrep
Default Code Mode
asgrep.search({ query })and the Pi toolmode: "natural"are hybrid — the sameSearcheras CLI, including the sticky NAPI addon (CodeModeSession).asgrep.semantic(...)is the sub-1 ms unique p50/p90 path. No separate JS search implementation. Published npm is not this binary until the next native build/release.Test plan
node --import tsx --test tests/pi/extension/sqlite.test.ts tests/pi/extension/runtime.test.ts(58 passed, including a live Bun import smoke)cargo test -p ast-sgrep-cli --lib no_auto_index_flag_parsescargo test -p ast-sgrep-cli --test cli_smoke auto_indexcargo test -p ast-sgrep-cli --test machine_contracts capabilities_and_version_match_goldenscargo test -p ast-sgrep-core --test semantic_ivf_roundtrip -- --nocapture adaptive_ivf_recall_at_10 semantic_ivf_roundtrip_and_fingerprint_gate centroid_preserving_reassign(recall@10 0.998437)cargo test -p ast-sgrep-core --test cascade_planner(2 passed)/tmp/asgrep-bench/idx_bigviacodemode-serve(semantic p50/p90 0.49/0.73 ms; hybrid n=85 p50/p99 1.08/3.92 ms)pi-ast-sgrepunder Bun/Pi and confirm the extension loads without thenode:sqlitecrashasgrep search <query> <path> --jsonreturns hits (exit 0) instead of the empty-index errorasgrep search <query> <path> --json --no-auto-indexstill exits 2 withindex is emptyCode Mode (in-process, four commands)
Pi Code Mode now runs in-process in
node:vm(no Worker sandbox). Worker spawn was the activation serial wall.Model-facing surface is four commands:
search/find/read/edit. Return shapes are declared (Blacksmith muscle memory).find({ query: "blast:Symbol" })reverse-walks callers;blast:path/to/file.tsuses imports. Stage 1 is cheapfind; Stage 2 issearch/readon survivors.Promise.allstill coalesces onto one warmCodeModeSession.ParallelMode::Autostays serial-warm (N SQLite opens are the wall).Isolation:
asgrep/consoleare built inside the VM from a JSON host bridge so hostFunctioncannot leak. Same trust as Pibash— not an OS jail. CLI sticky-serve (worker.ts) remains the degraded fallback, not a sandbox isolate.Test plan (Code Mode)
cargo test -p ast-sgrep-codemode --lib --test catalog --test session_plan --test batch -- --test-threads=1(34 passed)ASGREP_CODEMODE_BACKEND=cli node --import tsx --test tests/pi/extension/codemode.test.ts(31 pass; 1 pre-existing sticky-stdin write-timeout flake, not this slice)node --test tests/pi/launcher/extension-package.test.mjs(packed inventory dropssandbox-worker, includesdist/sqlite)