Feat/memory - #17
Merged
Merged
Feat/memory#17
Conversation
New pure dates.ResolveRelative resolves "today/tomorrow/yesterday, in N days|weeks|months, next week|month, [next] <weekday>" against the capture clock - no extra LLM call, absolute date strings deliberately pass through unresolved. Resolved forms land in two places: frontmatter renders '- tomorrow -> 2026-08-27' via an index-aligned DateResolutions slice on the entities block, and the entities table gains a nullable resolved_date column (idempotent ALTER) so follow-up queries can filter on real dates.
Assembles a per-capture context block and injects it into the four enrichment prompts so tags/summaries/entities are consistent with who the user is and what khayal already knows. Block composition, in order: config user_context (own 2000-char cap), people/orgs config blurbs matched case-insensitively against the capture text, LLM-maintained memory file contents, entity glossary (most frequent persons+orgs), and top-3 semantically similar past note chunks via TopSimilarChunks. Derived content shares a 1200-char cap; empty sources yield no injection and clients without call-context support are unaffected. memory.enabled=false short-circuits before any embed or retrieval. Enrichment now runs four LLM calls per capture (entities added in phase 1 runs sequentially after the parallel three).
…se 2.5c) <inbox>/memory.md (filename configurable via memory.file) is khayal's long-term memory: seeded as a headed skeleton on startup (# Memory / About the author / People / Ongoing threads / Preferences), consolidated by the LLM in a merge-style rewrite, and injected back into enrichment as part of the context block. - new chained 'memory' job type, gated by a throttle: runs only when >24h since the last consolidation or >=5 new distinct persons accumulated (markers in stats_cache) - consolidation prompt preserves unrelated content and manual edits, keeps headings stable; output sanitized + atomic write behind mtime guard; previous file survives until a rewrite succeeds - managed-file exclusions: reindex scan and connection wikilink targets never touch memory.md; it is never indexed or retrieved as a note - queue gains GetStat/SetStat/CountPersonsSince; vault gains WriteManagedFile/ManagedFileExists/ReadManagedFile Verified live: skeleton seeded on start; one capture chained text -> connections -> memory; John Doe/Acme facts merged into People and Ongoing threads sections.
consolidation_interval_hours and new_persons_threshold move from hard constants into the memory config section. Both are *int with nil/negative falling back to the previous defaults (24h / 5); explicit 0 is honored - interval 0 means every capture can trigger a consolidation, threshold 0 means any single new person does. Docs updated: config.example.yaml documents both knobs and clarifies the mechanism (background job queue chaining, never a cron); SPEC.md gains the static-user-memory + memory-file bullet points; VAULT.md notes the throttle fields.
The consolidation model sometimes echoed the user-prompt section labels (RECENT CAPTURED FACTS: / CURRENT MEMORY FILE: / NEW PEOPLE SINCE LAST RUN:) into its output. Because consolidation is merge-don't-remove, that pollution then compounded every cycle. memory.SanitizeConsolidatedOutput now truncates everything from the first echoed label, requires output to start with '# Memory' and contain all five canonical headings, and cleans dangling bullet dashes. Broken output returns an error so the job retries instead of persisting garbage; the previous file survives untouched until a rewrite succeeds. Consolidation prompt hardened with matching explicit rules. Verified live: previously polluted memory.md rewritten clean (0 fragment leaks) with new facts still merged.
chainMemoryConsolidation linked its job id onto the ingest job, overwriting the connections pointer that chainConnections had just set - clients polling an ingest job were silently redirected to the memory job (whose result is empty), hiding live connection results. The implicit call inside chainConnections is also decoupled: processJob now invokes both chain functions explicitly, so each can be reasoned about (and tested) independently. Comprehensive worker regression suite added covering: pointer set + target validation, disabled/empty-path no-ops, the clobber regression, throttle gates (disabled memory, fresh marker suppression, new-persons threshold trigger), and managed-file exclusion from connection links. testdata rows repointed to their real connections jobs; verified live: fresh capture's pointer resolves to a connections job with populated results.
…se 2.5 wrap-up) Live captures showed small-model flakiness: names filed under tags but missing from people, leaving enrichment blind. Three layers of fix: - Glossary rescue pass: when extraction returns ZERO people, known glossary names appearing in the capture text are promoted deterministi- cally (no extra LLM call). Non-empty extractions are never overridden. - Root-cause bug fixed en route: GetEntityGlossary scanned one column from a two-column grouped query and silently swallowed the per-row errors - the glossary has been empty since it was written. - Default system prompts redesigned from observed failure modes: ExtractEntities insists names are extracted even when used as topics, bans years-as-amounts, requires consistent casing per entity; Consoli- dateMemory gains anti-drift rules (collapse repeated meetings into one counted line, no cross-person fact copying, drop rhetorical noise, opinions only in Preferences). Verified live: 'what is bobs issue...' - previously produced zero people/connections - now extracts Bob and surfaces three person connections.
…ned memory rewrites The 3b text model kept violating consolidation merge discipline: duplicated sections, echoed prompt labels as headings, preserved rhetorical questions. Model upgrades beat more prompt nudging. - config: optional llm.consolidation_model; empty or equal to text_model reuses the primary client via NewConsolidationLLM - worker: SetMemoryLLM installs the dedicated client; processMemory calls GenerateWithSystemTemp at 0.2 when available - consolidation is a deterministic merge, not a creative task - sanitizer: collapse canonical headings that reappear later along with everything under them (the repeat is always the weaker copy); table-tested against real observed garbage output - timeouts: generation/vision now use OllamaGenerateTimeout/Vision- Timeout constants bumped to 300s - cold-loading a 7b model blew the hard-coded 120s and cost a retry - prompts: each heading exactly once, never write "None" filler Verified live on qwen2.5:7b: single People section, meetings collapsed with counts, repayment fact recorded accurately (owes 50, not 100), zero structural artifacts across two clean cycles.
…escue Documents the phase 2.5 wrap-up work that shipped in fb175ca/e536128: llm.consolidation_model config, temp-0.2 consolidation, structural sanitizer rules, and the zero-people glossary rescue pass.
Google-Gemini pattern: an explicit CTA synthesizes a grounded answer from the search results the user already has. Never automatic, no config option - the param/button IS the switch. API (GET /v1/search?overview=true): - Search runs unchanged first; overview is one RAG call over the top min(5, len) result excerpts at temperature 0.3 via optional-interface - Fail-open: any generation failure returns overview:null with full results and a warn log; zero results or missing param = zero LLM calls - extractCitations keeps unique in-range [n] refs in first-appearance order; out-of-range refs are dropped, never clamped into wrong notes - Mock-LLM handler tests cover all four contract cases kl: - kl search --answer fetches with overview and renders a themed panel above the results with muted source lines; null -> 'AI answer unavailable'; 180s request timeout for the LLM round-trip PWA: - Gold-accented CTA pill beside the filter chips, rendered only once results exist; clicking is the only trigger - nothing watches state - Skeleton thinking-state reuses the shimmer loader: pulsing badge + four staggered bars where the answer lands - Answer card: gradient-border panel, [n] tokens as gold superscript chips that smooth-scroll to anchored result cards, regenerate/dismiss actions; error state offers retry without touching results Verified live against testdata vault with qwen2.5:7b: 10 results, answer citing [2] and [4], no-param requests carry no overview field. Docs catch-up for consolidation routing + glossary rescue included.
Replaces the header pill + floating box with a first-class row at the top of the results list. Collapsed it looks like part of the list (sparkles + 'AI Answer' + chevron); clicking expands it fluently in place - height animation, shimmer skeleton, then the answer. Expansion state is decoupled from fetch state: collapse keeps the answer cached, re-expanding is instant and free; dismiss resets fully. Header click semantics: idle -> ask+expand, ready -> toggle. Component tests rewritten for the six interaction states; assets rebuilt; live-verified end to end.
SPEC: inline expanding row wording, temp 0.3, top-5 excerpts, citation clamping, zero-LLM-call contract. REPO_STRUCTURE: overview.go, AIAnswer.tsx, useAIAnswer.ts. UI_SPEC: AI answer row interaction spec. ARCHITECTURE: search flow gains the optional overview leg.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.