feat: in-process critic, causal planner, and channel conjunction

[AdityaVG13]

make the engine behave like a second model checking its own work, in-process, deterministic, no sidecar. Stacked on #30 (feat/golden-assert-testkit, base 7a004f1).

What ships

1. Deterministic post-fusion critic (a2dd668)

crates/ast-sgrep-core/src/search/critic.rs, wired after fusion in Searcher:

• Corroboration gate: embed-only hits whose parent has no lexical or structural corroboration are dropped, except on the conceptual/empty-structural path (annotated, not silent).
• Agreement boosts: def + caller + embed agreeing on the same span outrank a merely similar chunk; semantic + structural agreement is boosted.
• Collision penalty: identifier-fragment collisions (refresh in CSS vs auth_refresh) are penalized.
• why on agent JSON: critic notes surface through hit_why and the agent envelope's why array, built from field scores plus the contributor set.

2. Causal follow-up queries (035e7fc)

crates/ast-sgrep-core/src/search/planner.rs:

• follow_up_queries are computed from the actual top hit: kind, symbol, contributor gaps, critic notes, and margin. A weak-margin semantic hit on auth_refresh emits defs:auth_refresh / callers:auth_refresh as executable commands the engine would run.
• suggested_next derives from the actual top hit; settled hits get no follow-ups.
• Replaces the previous static templates.

3. Two-channel conjunction queries (c209752)

crates/ast-sgrep-core/src/search/conjunction.rs:

asgrep 'callers:process_request AND pattern:fn $NAME($$$)'
asgrep 'imports: rusqlite AND semantic:"parameterized query"'
asgrep 'defs:handle AND NOT callers:test_'

• Exactly two prefixed channels (defs:/callers:/imports:/pattern:/literal:/regex:/word:/semantic:), AND / AND NOT.
• Left channel is result identity. Pattern/caller pairs use contained-span scope; other pairs use file scope. AND NOT subtracts at the same scope, and matching right evidence merges into the contributor set.
• Unprefixed text, lowercase and, or multiple operators fall through to ordinary hybrid search, so plain English keeps its meaning (QG-023/QG-024 hold).
• semantic: exists only in this conjunction layer; ParsedQuery is unchanged.
• Documented in docs/QUERY_GRAMMAR.md (normative section + "not supported" updated).

4. Honesty docs (2021084)

• docs/comparison.md: adds a "Campaign goal (not current status)" section. No "replaces X" claim; the section names the keep-gates that must go green first.
• docs/progress/surface-deferrals.md: pattern-rewrites-not-in-product reopened on purpose, linked to bead ast-sgrep-2t4q (blocked on nested patterns), row stays Open.

Validation (RCH, all green)

cargo test -p ast-sgrep-core --lib                       170 passed
cargo test -p ast-sgrep-core --lib conjunction            11 passed
cargo test -p ast-sgrep-core --test conjunction_queries    5 passed
cargo test -p ast-sgrep-core --test evidence_merge         5 passed
cargo test -p ast-sgrep-core --test search_correctness_epics  10 passed
cargo test -p ast-sgrep-core --test pattern_diff           3 passed, 1 ignored (env-gated)
cargo test -p ast-sgrep-embed --lib                       green
cargo test -p ast-sgrep-plugins                           19 passed
cargo test -p ast-sgrep-cli --test machine_contracts      27 passed (2 filtered: known bench_json/chain_expand skips)

Agent golden tests/cli/fixtures/search_agent_hits.json regenerated for the new why / follow-up shapes.

Contracts kept

• No cloud/Ollama resurrection; Auto stays hashed-256; ONNX untouched (P4 bead exists, blocked on eval).
• No invented MRR/latency; parity_score.json untouched (certified=false, lower_bound=0); no release_certificate.json.
• Pattern never shells out to ast-grep; MCP stays non-fusing; compact still drops provenance.
• No .beads/, no .bench-history/, no incidental keep_gate.rs / sqlite fmt noise in the diff.

Tracker

Closed: ast-sgrep-3xym (critic), ast-sgrep-z741 (causal follow-ups), ast-sgrep-1rgt (conjunction).
Created for the remaining campaigns: yira (nested patterns), o8ng (pattern-diff keep-gate), k1ns (pattern-graph join), zxd9 (literal-vs-rg fixture), o6i3 (watch freshness), nusg (taint-lite), y7lj (nl fanout), 2t4q (codemod dry-run, blocked on yira), ozcq (ONNX, blocked on 7d5x.4). Dependency graph acyclic.

Not in this PR by design (the brief says do not boil the ocean in one PR): 7d5x.4 multi-field conceptual eval stays an open bead; P1+ campaigns are tracked, not started.

Adversarial review remediation

• Made conjunction retrieval exhaustive within its bounded channels, kept semantic retrieval exact, localized critic evidence, rejected malformed patterns, and preserved field-rescoring evidence.
• Corrected the pattern/caller contract above from file scope to contained-span scope.
• Updated the embed helper regression fixture after its API gained an explicit hit limit; the focused RCH regression passes.
• Review fixes: f1e0ae6c, f4f70d8e; stacked sync head eae486ea. CI was not awaited.

[AdityaVG13]

Extended this PR with the last open P0 item, 7d5x.4 (multi-field conceptual eval), plus a real bug it flushed out.

70c360a feat: add concat-embed eval arm and conceptual gold for 7d5x.4

• SearchOptions.use_field_rescoring (default true): when false, embed hits keep the concatenated-chunk similarity instead of the intent-weighted per-field mix. Included in cache_identity. Both field-vector fetch sites in passes/embed.rs are gated, so the concat arm also skips the DB reads.
• asgrep eval --ab concat-embed: third A/B arm beside no-embed / semantic-only. run_eval.sh now emits self-ab-concat-embed.json.
• 5 new conceptual gold queries in benchmarks/gold/self.json (17 total), each pointing at the verified authoritative file (critic, planner, conjunction, fusion, query-embed cache).
• tests/core/concat_embed_ab.rs: proves the toggle is live — default arm attaches embed_fields per-field scores, concat arm never does, same files either way on the fixture.

d18725e fix: excerpt query must survive blank lines inside a hit span

First eval attempt crashed: defs:to_compact_json died with Invalid column type Null from indexed_excerpt_in_range. Root cause: SQLite substr() over an empty BLOB (any blank line) yields NULL. Pre-existing since 00bdcff; any defs: hit whose span contains a blank line was affected. Fixed with COALESCE(..., x'') / COALESCE(length(...), 0) and a regression test (defs_excerpt_survives_blank_lines_inside_the_span). Bead ast-sgrep-5vur filed and closed.

ddc2a38 docs: record 7d5x.4 A/B result in baselines.md

Measured at frozen clean commit d18725e, hashed-256, both arms through the identical fused pipeline including the critic:

comparison	delta MRR	delta nDCG	delta Recall@20
hybrid (multi-field) vs --ab concat-embed	0.000	0.000	0.000
hybrid vs --ab no-embed	+0.008	+0.020	+0.059

Null result, recorded as such: on this gold, per-field rescoring changes nothing fusion had not already decided. The embed channel as a whole earns its seat (one conceptual query found only with embed on). Rows self-ab-concat-7d5x4-d18725e / self-ab-noembed-7d5x4-d18725e in baselines.md, tagged reproducible-in-tree with exact commands; explicitly not canonical MRR fingerprints. ONNX gate stays closed — no measured hole; a judged paraphrase eval on a foreign corpus remains the missing evidence and is named as such.

Validation: core lib 170, concat A/B 3, conjunction 5+11, e2e_smoke 8, evidence_merge 5, correctness epics 10, pattern_diff 3, plugins 19, embed lib, machine contracts 27 — all green. Tracker: 7d5x.4 and 5vur closed with evidence. All four P0 campaign items from the brief are now in this PR.

[AdityaVG13]

Adds bead ast-sgrep-yira: native in-process nested structural templates.

• fn $N($$$) { $STMT }, fn $N($$$) {}, if ($COND) { $BODY }, if $COND { $BODY }, if $COND: $BODY all match natively with exact statement-count semantics (single $STMT/$BODY = exactly one named statement; comments are trivia; concrete conditions fail closed).
• Nested templates bypass pattern_nodes (signatures cannot encode statement counts); per-thread parser reuse and global compiled query caching.
• $OBJ.$METHOD($$$) now handles keyword receivers (self.helper(), this.draw()).
• Differential gate vs ast-grep 0.45.1: 5/5 match-set equality (harness corrected: exit 1 = valid zero matches).
• Workspace clippy clean on 1.97, fmt clean, 60/60 test suites green across lang/core/cli/mcp.
• Real CLI E2E on a mixed Rust/Python fixture verified native matching without external fallback.