Self-improvement engine — rebuild on v0.11 base (replaces #100)

[dfrostar]

Context

Closing PR #100 (May 14 draft) without rebasing. This issue preserves the design intent so the work can be picked up cleanly when the time is right, with v0.11's directional transitions in scope from the start.

Original PR #100 shipped Phases 1–2 of the self-improvement engine:

• D1 — logging substrate (memory.py): wakeup-event logging, session_id on events, read-side aggregation helpers (recent_events, escalation_rate, re_query_rate, wakeup_only_rate).
• D2 — subsystem A, selector auto-tuning (self_improve.py new module): adjusts l2_recall_k from re_query_rate, persisted in the synapse meta table. Off by default behind NEURALMIND_SELECTOR_AUTOTUNE=1. Hysteretic dead band, 50-event warm-up, fail-open.
• Planning docs at evals/self_improvement/PLAN.md and evals/faithfulness/PLAN.md.
• 16 test cases on the tuner; full suite green when last run on May 14.

Branch preserved: claude/plan-next-task-nwJak (head 588b838).

Why deferred

Three reasons:

1. Stale against v0.11. synapses.py, core.py, hooks.py, cli.py, and matching tests were heavily modified in v0.10 (compressor + recovery cache) and v0.11 (directional transitions). Rebasing would be non-trivial; merge conflicts likely throughout the synapse layer.

2. v0.11 adds tunable surfaces the original D2 doesn't know about. The directional-transition signal has its own config knobs (TRANSITION_DECAY_RATE, TRANSITION_PRUNE_THRESHOLD, transition top-K, transition min-prob for the auto-memory export). A redesigned auto-tuner should consider these alongside l2_recall_k, not bolt them on after a rebase.

3. Overlap with Deprecate the learned_patterns reranker + neuralmind learn (superseded by the synapse layer) #143. The learned_patterns reranker deprecation work in Deprecate the learned_patterns reranker + neuralmind learn (superseded by the synapse layer) #143 should land before (or at minimum alongside) any new self-tuning mechanism, so the auto-tuner doesn't ship targeting a signal that's being removed.

Recommended path when picking this up

1. Land Deprecate the learned_patterns reranker + neuralmind learn (superseded by the synapse layer) #143 first (reranker deprecation). Removes one tunable surface that's already inert.
2. Re-survey tunable surfaces post-Deprecate the learned_patterns reranker + neuralmind learn (superseded by the synapse layer) #143: L2 recall k, transition decay rate, transition prune threshold, hub-degree normalization, LTP threshold. Each is a candidate for the auto-tuner.
3. Re-derive the signal model. re_query_rate was the original driver. With v0.11, additional signals exist: top-1 transition probability margin (when high, we can be more aggressive with context; when flat, fall back to broader L2/L3), transition-prediction hit rate (did the agent actually edit the predicted next file?), synapse spread-activation overlap with semantic search hits.
4. Re-implement D2 (or D2-equivalent) from scratch. The original D1 logging substrate may largely transplant; the tuner itself should be redesigned.
5. Continue with Phase 3 (subsystem C, eval-driven). The original evals/faithfulness/PLAN.md is still the right north star — a fitness function that lets the engine self-tune without humans deciding thresholds.

Effort estimate

• ~1 day to land Deprecate the learned_patterns reranker + neuralmind learn (superseded by the synapse layer) #143 cleanly
• ~3-5 days for the rebuild of D1+D2 with v0.11 surfaces in scope
• ~2 weeks for Phase 3 (subsystem C) given the eval scaffolding required

Related

• Original PR: Self-improvement engine: plans + D1 logging substrate + D2 selector auto-tuning #100 (closed, branch preserved)
• Reranker deprecation: Deprecate the learned_patterns reranker + neuralmind learn (superseded by the synapse layer) #143
• Hygiene cleanup tracker: [v0.6.1 Phase 0] Hygiene cleanup — branches, About-box, topics #117