Improvement: add observation→learning synthesis pipeline; strengthen harvest dedup

[raphasouthall]

Summary

Memory type distribution skews heavily to raw capture over synthesis: 227 observations (38%), 134 bugs (22%), 131 decisions (22%), 65 learnings (11%). Ratio of observation:learning is 3.5:1. Nothing in the codebase promotes observations into learnings, and harvest dedup is weak enough that near-duplicate learnings land together (e.g., three paraphrased "Harrods supply chain" learnings from one session with identical empty tags).

Root causes

1. No synthesis pipeline. Nothing transitions observations → learnings. `merge_memories` picks "more specific type" when merging duplicates, but there is no synthesis job that scans related observations and emits a consolidated learning.
2. Harvest observation prefilter over-captures. `src/neurostack/harvest.py:488-492` matches generic terms (credential, api-key, endpoint, host, port, url, stored at, config.toml). Nearly every technical transcript hits this. Anything that doesn't match a more specific pattern defaults to observation. `claude-code` harvest accounts for 494/599 memories (82%).
3. Learning prefilter too narrow. `"discovered that"`, `"TIL:"`, `"turns out"` — misses most synthesis moments. Of 65 learnings, 33 came from direct `vault_remember` and only 31 from harvest.
4. FTS-only dedup. `harvest._is_duplicate` (`harvest.py:542`) uses FTS5 keyword overlap. No cosine similarity check. Paraphrased memories with different keywords all pass dedup.

Proposed fix

1. Synthesis/promotion job. Add `neurostack synthesize`: scan observations older than N days with ≥3 FTS5-related siblings, ask the LLM to emit one consolidated `learning`, tag originals with `superseded_by`. Invoke from `session_end` auto_harvest path and from a scheduled timer. Effect: drains the 227-observation bucket; improves learning quality.
2. Replace FTS dedup with cosine similarity. In `harvest.py:542-558`, call `memories.find_similar_memories(..., threshold=0.88)` instead of FTS overlap. Effect: kills the "three Harrods paraphrases" class of duplicates.
3. Tighten observation prefilter. Split `harvest.py:488-492`: credential-pattern → `context` (with TTL); url/endpoint/host → `observation`; require at least one concrete identifier. Effect: reduces observation noise; lifts context count organically (see separate context-type issue).
4. TTL defaults by type at tool layer. `src/neurostack/tools/memory_tools.py:22` — default `ttl_hours=168` when `entity_type=='context'` and `ttl_hours=720` for `observation` unless overridden. Effect: "0 expired" becomes meaningful; real forgetting curve emerges.
5. Clearer type guidance for users/agents. Expand `src/neurostack/skills/memory-management.md:40-46` — current text is terse and says "observation is noisy, not written to vault", pushing users to default to observation when unsure. Add explicit guidance for context (ephemeral state, credentials) vs learning (insight worth keeping).

Expected effect

Observation:learning ratio drops from 3.5:1 toward a healthier 1:1 to 2:1. Harvest stops emitting near-duplicates. Context bucket populates from sessions, not just manual writes. Memories have realistic TTL, so `expired` becomes a non-zero signal.

Key files

• `src/neurostack/harvest.py:488-493, 542-558, 600-605, 647, 753`
• `src/neurostack/memories.py:455` (`find_similar_memories`)
• `src/neurostack/tools/memory_tools.py:22-75`
• `src/neurostack/skills/memory-management.md:40-46`

Relationship to other issues

Complementary to the "context type unreachable from harvest" bug — that one is a one-line fix, this one is the broader synthesis-pipeline design.

[raphasouthall]

Acceptance criteria

• `neurostack synthesize` CLI command exists; runs over observations older than N days with ≥3 FTS5-related siblings
• LLM emits one consolidated `learning`; originals are tagged (not deleted) with `superseded_by=<new_memory_id>`
• `harvest._is_duplicate` replaced with `memories.find_similar_memories(threshold=0.88)` (cosine); regression test on the "Harrods supply chain" near-duplicate scenario
• `harvest.py:488-492` observation prefilter tightened: credential patterns route to `context`; observations require at least one concrete identifier
• `tools/memory_tools.py:22`: default `ttl_hours=168` for `context`, `ttl_hours=720` for `observation` unless overridden
• `skills/memory-management.md:40-46` expanded with clear type guidance (especially context vs learning)
• On a test transcript, observation:learning ratio improves from current ~3.5:1 toward ≤2:1 after synthesis runs

Dependencies

• Depends on: Bug: harvest can never produce 'context' memories (type missing from valid set) #30 (`context` type must be reachable from harvest before observation prefilter can re-route credential patterns to it).
• Benefits from: Bug: memory embeddings silently dropped on failure; no backfill path #29 (cosine dedup is stronger at 100% embedding coverage).

Sequencing

Phase 4 (rework). Ship after #30 (Phase 1) and #29 (Phase 2).