Feat/proactive connections - #16
Merged
Merged
Conversation
After a successful ingest, the worker chains a connections job for the
note. Polling the original job now reveals connections_job_id; polling
that job returns ranked connections stored in a new jobs.result column.
Three detectors, all gated by config and never surfacing notes younger
than min_age_days (7 default):
- similar: reuses the note's own stored chunk embedding via SearchSemantic
with a date-cutoff filter - no extra LLM call
- person / amount: entity-table lookups with SPEC-format labels
("Alice also appears in N other notes")
Ranking dedupes per note keeping the highest-priority occurrence
(person > amount > similar), sorts, and caps at max_per_capture.
Detector errors skip their type instead of failing the job; chaining
failures are logged and non-fatal.
Deviations from the phase doc, forced by the real codebase: no
re-embedding step (chunks already exist), correct age-filter direction
(doc had it inverted vs SPEC), connections job created post-ingest rather
than at capture time since note_path does not exist yet.
initSchema ran the phase-2 ALTER statements through the strict fail-on-error loop, so any database already carrying the new columns failed to open on second server start. Column adds now run in their own pass where duplicate-column errors mean already-migrated and are treated as success; a regression test opens the same DB twice.
ListJobs and GetPendingJobs scanned note_path/source_url/source_file/ user_context/content/error into plain strings, so any row with a NULL in those columns 500'd the whole queue view. Hand-seeded SQL fixtures triggered it; nothing prevented production rows from doing the same. Both scans now use sql.NullString per column (FetchAndLockPendingJobs already did), covered by a regression test that inserts a row with every optional column NULL.
After a connections job completes, ranked targets that exist on disk are written into the note's frontmatter as a connections: block of quoted wikilinks (- "[[note-name]]"), so Obsidian's graph and properties see what khayal found. Body text is never touched. SetConnections splices only the top-level connections key (replacing any stale one), skips the write entirely when nothing changed, restores mtime after writing so external-change detection elsewhere keeps working, and the worker pre-verifies every target on disk - vault safety rule #5, never write broken wikilinks. Verified live: capture mentioning an older note's entities produces the block while the body survives byte-for-byte.
…, amount corroboration Live results were dominated by coincidental number matches: any capture mentioning $2,000 connected to every other note containing 2000, regardless of context. Meanwhile genuinely related notes never surfaced because the 0.85 threshold sits above most useful nomic-embed-text similarities (measured: unrelated ~0.49, related ~0.79, identical ~0.90). - TopSimilarChunks: raw-cosine connection search (no rescaling, self and age excluded) so Connection.Score is true confidence - default similarity_threshold 0.85 -> 0.72, inside the measured gap - amount corroboration guard: an amount match surfaces only when the pair also shares a person entity or scores within 0.10 of the similarity threshold; bare number equality is suppressed - person detector unchanged; ranking/dedup unchanged Verified live: John Doe/$2,000/Acme capture now yields empty connections (was: spurious Alice-budget match); Alice captures still connect with correct labels.
… explicit zero min-age
Three stacked issues surfaced when same-day Bob captures produced no
connections:
1. Person casing varied per LLM run ("Bob" vs "bob") and entity SQL
compared exact strings, so cross-note person joins silently missed.
Entity lookups now match case-insensitively via LOWER().
2. No guard against extractor junk: "I" was being stored as a person.
Normalization now rejects single letters and a pronoun stoplist, and
the extraction prompt forbids author/pronoun/single-letter people.
3. ApplyDefaults coerced an explicit connections.min_age_days: 0 back to
the 7-day default (indistinguishable from unset), making same-day
connection setups impossible to configure. MinAgeDays is now *int:
nil/negative -> 7, explicit 0 honored.
testdata config sets min_age_days: 0 for same-day experimentation.
- REPO_STRUCTURE: real connections/ tree entry (drop six phantom files), remove nonexistent internal/search package, jobs table gains result + connections_job_id, entities table schema documented with the two independent writers, JobStore + Writer interface snippets refreshed - ARCHITECTURE: capture flow shows the connections chaining step - VAULT.md: frontmatter connections wikilink block contract - phases/v1.1: phase 2 marked shipped - openapi.yaml: Job schema gains result and connections_job_id
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.