diff --git a/dev/plans/runs/0.8.11-memex-handoff-prompt.md b/dev/plans/runs/0.8.11-memex-handoff-prompt.md new file mode 100644 index 00000000..5b57995b --- /dev/null +++ b/dev/plans/runs/0.8.11-memex-handoff-prompt.md @@ -0,0 +1,133 @@ +# Hand-off prompt — discuss 0.8.11 planner-router results + the Slice-36 iteration seam with Memex + +> **Paste this to spin up a FathomDB-side liaison session.** It is scoped to **0.8.11** (do NOT drift +> into building 0.8.15) — its job is to bring the 0.8.11 *experimental results* and the *Slice-36 +> iteration seam* to **Memex** (FathomDB's #1 consumer) and converge on positions in the shared +> leverage ledger. Nothing here is a contract; everything stays provisional until `AGREED` + HITL-signed. + +--- + +## Your role + +You are the **FathomDB-side agent** in the Memex⇄FathomDB collaboration. 0.8.11 is **complete and +merged** (PR #122, `abfc8b24` on `main`; F-11 discharged). Your task is **not** to build anything — +it is to **discuss the 0.8.11 planner-router screening results and the Slice-36 seam with the +Memex-side agent**, advance the relevant items in the leverage ledger, and surface any product +decisions to HITL. Speak for *what the engine measured and can expose and its cost*; do **not** +dictate Memex's LLM/product choices. + +**The cohesion seam governs every proposal** (the ledger's Guiding Principle): *FathomDB owns +structure & mechanism; Memex owns semantics & judgment; the LLM lives entirely on Memex's side, +reached through narrow provider protocols.* Reshape anything that violates it. + +## Read first (all on `main`) + +- **The caveat doc / hand-off:** `dev/plans/runs/0.8.11-handoff-to-0.8.15.md` — §1 confidence ledger, + §2 V-1…V-7 validation gate, §2a distilled recommendations, **§2b at-power experiment registry**, + §3 corrected findings, §5 function-contribution ledger. **This is load-bearing — read it fully.** +- **Contracts:** `dev/plans/0.8.11-implementation.md` (§3 per-intent tuple format, §4 L2 API, §5 F-8b). +- **Results of record:** `dev/experiments-ledger.md` → the `## 0.8.11` section (8 Track-E rows). +- **The prototype (what Memex would consume):** `dev/prototypes/l2-router/` — `router.py`, + `registry.json` (measured-vs-provisional tuples + confidence header), `README.md`. +- **Competitor landscape:** `dev/research/planner-router-{survey,secondary-review,competitors}.md`. +- **The design:** `dev/design/planner-router-psd-0.8.x.md` (§II.A planner/intent, §II.B config-carrying + router, §II.C closed-loop/VoI). +- **The Memex side (where you'll converge):** `~/projects/memex/dev/fathomdb/LEVERAGE-OPPORTUNITIES-LEDGER.md` + — especially **OPP-10** (intent-routed retrieval; FATHOM-proposed, `PROPOSED`) and the FATHOM Thread + comments on **OPP-3** (cascade/VoI signal), **OPP-7** (CE α/pool_n per-intent), **OPP-9** + (`record_feedback` / EXP-AF). Honor its editing protocol: append dated `[YYYY-MM-DD][FATHOM]` + Thread bullets; `AGREED` only on convergence; `NEEDS-HITL` for product/contract calls; cite paths. + +## The experimental results to discuss (frame as PROVISIONAL SCREENING — not a contract) + +State plainly to Memex: these are **single-corpus (mostly LME), small-N screening** results, gated +behind the **Pre-0.8.15 Validation Gate (V-1…V-7)**. Do not let them be consumed as validated config. + +1. **Config-carrying per-intent tuning is the lever — not arm-switching.** Gate-2: static per-class + arm selection buys ≈0 (RRF already absorbs BM25+vector on LME); the per-*query* oracle is still + unmeasured (V-2). EXP-B′: per-intent `(candidate_k, pool_n, α, …)` optima **diverge** (3 measured, + 2 provisional) and cross-regress (multi_session→needle −0.147). → the router's value to Memex is + "Fathom carries the right *config* for the intent Memex names," not "Fathom picks the arm." +2. **Memex is intent-source #1.** The internal classifier is only ~0.768 (a lexical lower-bound); the + design's intent order is **agent-label → provider-callback → internal-fallback**. **Memex supplying + the intent is the single highest-leverage connection** (OPP-10 Q1). Ask: does Memex's + phase-graph/goal-graph already classify into `{needle, multi_session, temporal, global, multi_hop}`? +3. **The binding constraint is recall/substrate, not routing-cleverness or agent-feedback.** Embedder + ceiling ~0.571 IR relevance; ANN fidelity 0.937 ≫ relevance (tuning ANN buys ≈0); keep bge-small. +4. **EXP-AF KILL is current-substrate provisional, NOT permanent.** Even a strong agent (claude-sonnet) + doesn't beat internal `ce_score` net of round-trip — but it was **recall-bound** (agent realized + ~6% of headroom) and **never tested `multi_hop`**. So `record_feedback` stays instrumentation (F-8b), + and the L2 prototype ships `feedback_arm=False` — *for now* (V-3 can flip it). + +## The Slice-36 seam to discuss + +> **UPDATE (2026-06-29) — Memex has already narrowed this; FATHOM concedes.** After this prompt was +> written, Memex posted two `[2026-06-29][MEMEX]` ledger entries that partly pre-empt this section: +> **OPP-10 (line ~819)** — *"single-shot **per FathomDB read**, NOT single-shot end-to-end"*: the +> EXP-AF KILL is scoped to the **feedback/re-rank loop (a)**, not a blanket rejection of iteration; +> **"keep FathomDB single-shot per read and do NOT prebuild the Slice-36 feedback seam"** (abstraction +> before evidence) — the engine would need it only if FathomDB owned loop policy/budgets/LLM-calls/ +> stopping, which are Memex's; if a measured bottleneck appears, add a **narrow [OPP-5] fan-out/merge +> verb**, not a generic escalation hook. It also downgrades the "two independent confirmations" claim +> (EXP-AF + Memex cascade are convergent only in the *dense* regime; the *sparse* multi-hop regime — +> where (b) is hypothesized to pay — is unsettled). **OPP-9 (line ~681)** — capture must bind **all +> reads of a turn** to one replayable eval case (decomposition issues N reads/turn). +> **HITL-agreed FATHOM stance (2026-06-29):** concede it. The engine stays single-shot per read; the +> Slice-36 `_maybe_escalate()` hook is **caller-side prototype scaffolding (`router.py`), not an +> engine commitment** and will not be hardened into the engine. FATHOM's durable value is +> **recommend-only**: Fathom *recommends* the per-intent `iteration_policy` in the config tuple; Memex +> hosts the loop. The at-power EXP-ITER-D/P/POLICY (V-3) experiments are how it would ever be earned — +> "who drives them" is the live HITL question below. Read the section below as the *original* framing, +> superseded by this note where they conflict. + +The L2 prototype exposes a **default-off seam** (`L2Router(feedback_arm=False)` + a no-op +`_maybe_escalate()` hook) so the agent-loop can be wired without re-architecting. The richer framing +(hand-off §2b, agreed with HITL): generalize it to a **per-intent `iteration_policy ∈ {single_shot, +parallel_decompose, sequential_iterate}`**. The seam can host **two distinct loop kinds**, and the +distinction is load-bearing: + +- **(a) feedback/re-rank** — reshuffles a *capped* pool (EXP-AF killed this for needle: recall-bound). +- **(b) decomposition/multi-hop** — forms *new* sub-queries to **manufacture recall** (IRCoT/Self-Ask/ + Iter-RetGen family) — **untested**, and it attacks EXP-AF's own root cause. Plus **parallel fan-out** + ("multiple single-shot": decompose up front, one round) for *independent* sub-Qs vs **sequential + iterate** for *dependent* hops. + +HITL's intuition (carry it to Memex): *it will probably be "both," for different needs* — i.e. +**iteration depth is a per-intent routing knob** (needle=single_shot; multi_hop=test (b)/parallel; +global=map-reduce fan-out, shipped). This is exactly the planner-router thesis applied to iteration. +**Cost honesty:** (b)/parallel need a caller-side (Memex) LLM → a higher compute tier, **per-intent +opt-in, not default-on**. The registered at-power experiments are **EXP-ITER-D / EXP-ITER-P / +EXP-ITER-POLICY / EXP-AF-MH** (hand-off §2b) — run at the representative paper's corpus+N, survive +further trials, KILL only at-power. + +## What to bring to Memex (drive these to a position in the ledger) + +1. **OPP-10 Q1 (intent source):** does Memex's phase-graph map to the 5 classes? If yes, Memex is the + high-accuracy intent source that retires reliance on the 0.768 classifier. (Highest leverage.) +2. **OPP-10 Q2 (locus):** recommend-only API (intent→config; Memex executes via governed verbs) vs + Fathom executes the routed plan end-to-end? Cohesion favors recommend-only. +3. **Iteration (Slice-36 / §2b):** does Memex want to **own/drive the multi_hop iteration experiments** + (it supplies the decompose/reason LLM)? Which intents does Memex actually need iteration for? Is + "both, per-intent" the right shape for Memex's workloads? +4. **OPP-10 Q4 / OPP-5:** does Memex's world-model filter set map onto the #17 unified `Filter` + `constraints` surface (now shipped)? +5. **OPP-9 / F-8b:** `record_feedback` stays instrumentation unless V-3 (EXP-AF on multi_hop/improved + recall) flips to GO — confirm Memex agrees the feedback loop is *gated on evidence*, not built now. + +## Guardrails + +- **Provisional, not a contract.** Repeat the screening caveat; point to the V-gate. Do not let any + tuple/verdict be treated as validated config. +- **Cohesion seam** on every proposal; **LLM stays Memex-side**. +- **Ledger protocol:** append to Threads; converge → `AGREED`; product/contract → `NEEDS-HITL` (don't + unilaterally contract across the boundary). Cite paths/lines. Update the ledger Changelog. +- **Stay in 0.8.11 scope** — you are discussing results + the seam, not building the 0.8.15 dispatcher. +- **Honesty on fairness:** competitor wins often used frontier LLMs/GPU; flag whether they transfer to + the CPU-default/local-first posture rather than assuming the paper number holds. + +## Deliverable + +Updated **OPP-10** (and threaded OPP-3/7/9) in the Memex leverage ledger with converged FATHOM⇄MEMEX +positions, any items moved to `AGREED` or `NEEDS-HITL`, and a short readback to HITL of: (a) whether +Memex will be the intent source, (b) the iteration shape Memex needs (and whether it'll drive the +multi_hop experiments), (c) any V-gate item Memex wants reprioritized.