feat(optimization): adopt official methods across knowledge surfaces#78
Conversation
…l-method # Conflicts: # README.md # src/retrieval-optimization.ts # tests/kb-improvement/candidate.test.ts # tests/rag-improvement-loop.test.ts # tests/retrieval-eval.test.ts
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — b0f33334
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-07-24T16:49:31Z
tangletools
left a comment
There was a problem hiding this comment.
🟢 Value Audit — sound
| Verdict | sound |
| Concerns | 1 (1 weak-concern) |
| Heuristic | 0.0s |
| Duplication | 0.0s |
| Interrogation | 153.9s (2 bridge agents) |
| Total | 153.9s |
💰 Value — sound
Removes a maintained-in-parallel grid-search optimizer and routes all knowledge optimization through official agent-eval methods with immutable identity refs and fail-closed activation gates — coherent and in the codebase's grain.
- What it does: Deletes the local bounded-retrieval grid search (boundedRetrievalConfigMethod, buildBoundedRetrievalConfigs, ~300 lines) so every optimization surface (runRetrievalImprovementLoop, runRagOptimization, optimizeKnowledgeBasePolicy, runAgentMemoryImprovement) requires a complete OptimizationMethod and flows through the existing runSerializedKnowledgeOptimization adapter (src/optimization.ts). Adds as
- Goals it achieves: (1) Single source of truth for candidate search/selection: stop maintaining a parallel optimizer and its resume/cost/concurrency bugs; let agent-eval own it. (2) Make optimization results non-silently-reusable across incompatible implementations (immutable refs + persisted candidate identity) so a resumed run cannot trust stale evidence. (3) Make promoting a winner into production memory/KB safe-b
- Assessment: Good change, built in the grain of the codebase. The deletion is the core value: boundedRetrievalConfigMethod duplicated what the dependency owns (campaign batching, cost-ledger wiring, completeness checks, concurrency) and carried its own edge cases; removing it shrinks the maintenance surface and removes a second optimization path. All four surfaces already shared the runSerializedKnowledgeOptim
- Better / existing approach: Searched src and the installed @tangle-network/agent-interface@0.32.0 dist: the shared package already exports sha256DigestSchema and gitObjectSchema (agent-candidate-schema-common.d.ts:3-4), and src/agent-candidate.ts:41 already consumes sha256DigestSchema. assertImmutableRef re-derives both patterns as local regex instead of composing the shared zod schemas into one assertion guard. This is cosm
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content
🎯 Usefulness — sound
A clean major-version consolidation that deletes a local retrieval sweep in favor of official agent-eval optimizers, and hardens resume/activation safety with immutable identity refs threaded through every optimization surface.
- Integration: Fully wired and reachable. The four public entry points (runRetrievalImprovementLoop, runRagOptimization, optimizeKnowledgeBasePolicy, runSerializedKnowledgeOptimization at src/index.ts:45/39/34; runAgentMemoryImprovement via /memory) all route through runSerializedKnowledgeOptimization (src/optimization.ts:75), which now requires and validates executionRef. runRagKnowledgeImprovementLoop consumes
- Fit with existing patterns: Fits the codebase grain precisely. AGENTS.md layering states agent-knowledge imports agent-eval and agent-interface downward; this PR moves JsonValue to AgentCandidateJsonValue from @tangle-network/agent-interface (src/rag-optimization.ts:6, src/retrieval-eval.ts:7, src/optimization.ts:13) and pins agent-eval to an exact 0.126.2. The deleted boundedRetrievalConfigMethod/buildBoundedRetrievalConfig
- Real-world viability: Hardened for realistic use. assertImmutableRef (src/immutable-ref.ts:4) enforces sha256:<64 hex> or git:<40 hex> on every identity-carrying option: executionRef (src/optimization.ts:83), policyApplicationRef (src/kb-improvement/optimization.ts:108), implementationRef/activation.ref/executeStepRef (src/memory/improvement/validation.ts:18,31,40), and candidate.ref (src/memory/improvement/candidate.t
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
💰 Value Audit
🟡 assertImmutableRef re-derives shared digest schemas [duplication] ``
src/immutable-ref.ts:1-8 hardcodes /^sha256:[a-f0-9]{64}$/ and /^git:[a-f0-9]{40}$/ locally, while @tangle-network/agent-interface (already a dependency, already used at src/agent-candidate.ts:41 via sha256DigestSchema) exports both sha256DigestSchema and gitObjectSchema (confirmed in agent-interface@0.32.0 dist/agent-candidate-schema-common.d.ts:3-4). If gitObjectSchema's accepted format ever widens (e.g. abbreviated/object hashes, case rules), the local copy won't track it. Consider composing
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — b033f182
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-07-24T17:07:53Z
tangletools
left a comment
There was a problem hiding this comment.
🟢 Value Audit — sound
| Verdict | sound |
| Concerns | 0 (none) |
| Heuristic | 0.0s |
| Duplication | 0.0s |
| Interrogation | 199.1s (2 bridge agents) |
| Total | 199.1s |
💰 Value — sound
Consolidates all four optimization surfaces (retrieval, RAG, KB policy, memory) on official agent-eval methods with immutable content-addressed identity and safety gates; removes ~200 lines of duplicated grid-search code — a clean, well-verified major version.
- What it does: This is a breaking major version (4.1.0→5.0.0) that: (1) removes the built-in boundedRetrievalConfigMethod (a hand-rolled exhaustive retrieval grid search with ~200 lines of helpers: buildBoundedRetrievalConfigs, measureBoundedRetrievalSurface, setConfigPath, campaignIsComplete, etc.) and requires callers to pass a complete OptimizationMethod from agent-eval for all optimization surfaces; (2) adds
- Goals it achieves: (1) Eliminate the duplicated local grid-search code by routing all optimization through the single official agent-eval OptimizationMethod funnel — the bounded method was a parallel reimplementation of candidate search that agent-eval already owns. (2) Make resume correctness verifiable: immutable content-addressed identity means if implementation, config, or external dependencies change, the ident
- Assessment: Sound and well-structured. The single biggest win is removing ~200 lines of hand-rolled grid-search logic (boundedRetrievalConfigMethod and its helpers) that duplicated agent-eval's OptimizationMethod abstraction — the old code ran its own runCampaign loops, cost ledgers, concurrency batching, and completion checks in parallel to the official path. Requiring a complete method aligns with AGENTS.md
- Better / existing approach: none — this is the right approach. Searched for existing equivalents: the codebase already has sha256DigestSchema from @tangle-network/agent-interface (used in src/agent-candidate.ts:3 and src/kb-improvement/activation.ts:286) but that only validates sha256 digests, not git commit hashes — assertImmutableRef correctly covers both formats for execution identity, so it is not a duplicate. The reject
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content
🎯 Usefulness — sound
Consolidates all knowledge optimization surfaces onto one complete OptimizationMethod primitive, removes the local bounded-retrieval sweep, and verifies the packed package against pinned official GEPA/SkillOpt — coherent, reachable, and in the grain of the codebase.
- Integration: runSerializedKnowledgeOptimization (src/optimization.ts:75) is the hub, publicly exported at root (src/index.ts:34) and required by verify-package.mjs:26. Four public wrappers consume it: runRetrievalImprovementLoop (src/retrieval-optimization.ts:85), runRagOptimization (src/rag-optimization.ts:75), optimizeKnowledgeBasePolicy (src/kb-improvement/optimization.ts:116), and runAgentMemoryImprovement
- Fit with existing patterns: Aligns with AGENTS.md:78 ('Use a complete OptimizationMethod from @tangle-network/agent-eval'). Removes the competing local sweep (boundedRetrievalConfigMethod/buildBoundedRetrievalConfigs/buildRetrievalParameterCandidates/retrievalParameterSweepProposer — confirmed absent from src/tests/docs/skills, and verify-package.mjs:29 asserts their absence). The immutable executionRef/policyApplicationRef/
- Real-world viability: rejectUnsafePromotionEvidence (src/rag-improvement-loop.ts:391) blocks promotion on missing evidence, incomplete cost accounting, unobserved package identity, negative lift-CI lower bound, or cost-ceiling exceedance before the caller hook runs. Memory activation (src/memory/improvement/activation.ts:65) uses compare-and-set plus a re-read of observed config, with an append-only journal that detect
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
No concerns — sound change, no better or existing approach found. ✅
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
✅ No Blockers —
|
tangletools
left a comment
There was a problem hiding this comment.
✅ Approved — 11 non-blocking findings — b033f182
Full multi-shot audit completed 8/8 planned shots over 50 changed files. Global verifier still owns final merge decision.
Full immutable report for this review: trace
Summary comment for this run: full summary
tangletools · 2026-07-24T17:52:24Z · immutable trace
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — fda8e254
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-07-24T18:27:47Z
tangletools
left a comment
There was a problem hiding this comment.
🟢 Value Audit — sound
| Verdict | sound |
| Concerns | 1 (1 weak-concern) |
| Heuristic | 0.0s |
| Duplication | 0.0s |
| Interrogation | 218.6s (2 bridge agents) |
| Total | 218.6s |
💰 Value — sound
Major-version refactor that deletes the local retrieval/search campaign code and routes all four optimization surfaces (retrieval, RAG, KB policy, memory) through agent-eval's compareOptimizationMethods, adding an immutable execution-identity layer that composes into agent-eval's dispatchRef plus fa
- What it does: Removes ~300 lines of locally-reimplemented campaign machinery (runCampaign/fsCampaignStorage/createRunCostLedger/campaignMeanComposite/costFromLedgerSummary) from src/retrieval-optimization.ts and src/rag-optimization.ts; both now delegate to runSerializedKnowledgeOptimization -> compareOptimizationMethods (agent-eval). Deletes the local boundedRetrievalConfigMethod/buildBoundedRetrievalConfigs s
- Goals it achieves: Statistical integrity of optimization runs: strict train/selection/final partitioning with content-fingerprint overlap detection (src/optimization.ts:305, src/memory/improvement/validation.ts:108), final cases touched exactly once, and no resuming a run whose callbacks/models/indexes changed. Fail-loud safety before activation: no promoting on regressing, unaccounted, over-budget, or unobserved-id
- Assessment: This is a strong change on its merits. The central move - deleting runCampaign/fsCampaignStorage/createRunCostLedger/campaignMeanComposite/costFromLedgerSummary and routing through compareOptimizationMethods - is the correct de-duplication: agent-eval already owns candidate search, resume, and final scoring, and the knowledge package had been parallel-reimplementing it. The identity layer is corre
- Better / existing approach: Searched for existing equivalents before answering. agent-eval already exposes dispatchRef (the resume-identity primitive) and provenance/accountingComplete/ComparisonCost; this change composes into those rather than replacing them, so no existing capability is being reinvented. The only adjacent duplication is agent-eval's internal
const SHA256 = /^sha256:[a-f0-9]{64}$/in src/campaign/surface- - Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content
🎯 Usefulness — sound
Consolidates four knowledge-optimization surfaces onto one serialized runner backed by official agent-eval methods, removing a local bounded-retrieval reinvention and adding durable immutable-identity and final-case isolation boundaries — all proven end-to-end via packed-install verification.
- Integration: Fully wired and reachable. Five public surfaces (runSerializedKnowledgeOptimization at src/optimization.ts:75, runRagOptimization at src/rag-optimization.ts:62, runRetrievalImprovementLoop at src/retrieval-optimization.ts:64, optimizeKnowledgeBasePolicy at src/kb-improvement/optimization.ts:88, runAgentMemoryImprovement at src/memory/improvement/run.ts:47) all funnel through one core runner requir
- Fit with existing patterns: Exactly in the grain. AGENTS.md already documents this as the integration boundary: 'Use a complete OptimizationMethod from @tangle-network/agent-eval with runRetrievalImprovementLoop(), runRagOptimization(), optimizeKnowledgeBasePolicy(), or runAgentMemoryImprovement().' Package layering is respected (knowledge imports eval, never vice versa). The serialized-candidate + codec pattern is consisten
- Real-world viability: Holds up well past the happy path. assertImmutableRef (src/immutable-ref.ts:6) enforces sha256/git hex identity, preventing resume-state corruption across implementation changes — validated at every entry point (validation.ts:18, evaluation.ts:49, optimization.ts:83). The final-evaluation boundary (finalEvaluationStartedAt ledger event + one-way gate in evaluation.ts:140-160) prevents final-case r
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
💰 Value Audit
🟡 exceedsCostCeiling duplicated verbatim across two modules [duplication] ``
function exceedsCostCeiling(totalCostUsd, costCeiling)with the identical epsilon-tolerance body exists in both src/rag-answer-evidence.ts:76 and src/rag-improvement-loop.ts (same epsilonmax8 formula). The non-negative-finite cost-ceiling validation (assertOptionalCostCeiling) also repeats theNumber.isFinite(value) || value < 0pattern in src/memory/improvement/validation.ts:59 and src/memory/improvement/candidate.ts. These are small (3 lines each) and in the same RAG-evidence domain, so
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
What changed
OptimizationMethodChecks
AGENT_KNOWLEDGE_NETWORK_TESTS=1 pnpm test: 452 passed, 12 skippedpnpm lintpnpm typecheckpnpm buildpnpm verify:packageAGENT_EVAL_TEST_PYTHON=... pnpm verify:official-optimizers: 2 passed