feat(db): schema v3-v4 and data access layer for knowledge pipeline

[gordonkjlee]

Infrastructure Change

Summary

Schema migrations v3-v4 and synchronous data access layer for the DIKW knowledge pipeline. Creates all database tables (session_facts, facts with FTS5, entities, graph edges, domains, consolidation lock) and their CRUD functions. Second of 5 PRs.

Stakeholder

N/A

Links

Ticket: N/A
Requirements: N/A

---

Description

Schema v3 creates the Information layer and supporting tables:

• session_facts — captured/extracted facts awaiting consolidation, with content_hash (SHA-256, UNIQUE per session for dedup), source_origin ('explicit'/'inferred' for hybrid capture), consolidation_id (claimed by which run). Partial index on consolidation_id IS NULL for fast unclaimed-fact queries.
• session_fact_sources — provenance junction table linking facts to multiple source events with relevance and extraction_type.
• domains — domain registry (data, not code). Seeded from config or created at runtime.
• consolidation_lock — single-row advisory lock (CHECK(id = 1)) preventing concurrent consolidation. 2-minute stale threshold for crash recovery.

Schema v4 creates the Knowledge layer:

• facts + facts_fts (FTS5 virtual table) + INSERT and DELETE sync triggers. Graduated, entity-linked, deduplicated facts. is_latest flag for fast current-state queries. FTS5 UPDATE trigger intentionally omitted: fact content is immutable (never modified, only superseded).
• entities — graph nodes with canonical_name NOT NULL and UNIQUE(canonical_name, type) constraint for dedup. access_count/last_accessed_at for activation tracking (inspired by ACT-R frequency + recency signals).
• fact_entities — junction table linking facts to entities with relationship type.
• entity_edges — entity-to-entity relationships with saturating potentiation for strength (spreading activation, Collins & Loftus, 1975). Strength follows new = old + (1 - old) × α (α=0.3), inspired by LTP saturation — early co-occurrences cause large jumps, later ones diminish. Monotonically increasing by construction, no floating-point precision loss. EDGE_POTENTIATION_ALPHA is a named constant for future parametric feedback.
• sources — provenance records.
• consolidations — run records with stats (facts_in, graduated, rejected, entities_created, supersessions).

Data access modules (all synchronous, better-sqlite3):

• session-facts.ts — insertSessionFact (INSERT OR IGNORE for hash dedup), getSessionFacts, getUnconsolidatedFacts, claimForConsolidation (atomic claim; caller must hold consolidation lock), linkFactSource, getFactSources
• facts.ts — insertFact (validates INSERT succeeded), getFact, getFactsByDomain, getFactsByEntity, supersedeFact (transaction: mark old + insert new; throws if old fact not found), keywordSearch (FTS5 BM25; throws on malformed FTS5 syntax — callers should sanitise or catch), incrementFactAccess
• entities.ts — findEntity (canonical name match; non-deterministic without type filter), findEntityByCanonical (exact match, no normalisation), findOrCreateEntity (transaction-wrapped, safe with UNIQUE constraint), createEntity, linkFactEntity, upsertEntityEdge (saturating potentiation with EDGE_POTENTIATION_ALPHA), getEntityEdges, updateEntityAccess
• domains.ts — getDomains, createDomain, ensureDomain (idempotent)
• consolidation-lock.ts — acquireLock (with 2-minute stale detection and takeover), releaseLock (returns boolean), getLockState

---

Testing

48 new tests across 3 test files + 2 updated assertions in sessions.test.ts:

• session-facts (10 tests): insert with auto-hash, intra-session dedup rejection, cross-session allowed, ordered retrieval, unclaimed filtering, atomic claim, provenance linking
• facts (14 tests): insert, is_latest boolean cast, domain filtering, subdomain filtering, entity-linked retrieval, supersession chains (A→B→C with only C is_latest), FTS5 keyword search with rank type assertion, access count increment, throws on superseding nonexistent fact
• entities (25 tests): canonical name normalisation, case-insensitive find, findEntityByCanonical exact-match contract, type filter, find-or-create, metadata round-trip, fact-entity linking + idempotency, saturating edge potentiation (monotonic increase verified across 50 iterations), edge retrieval, access tracking, domain CRUD + idempotency, lock acquire/release/stale takeover (2-min threshold)

All tests use :memory: databases — no file system side effects.

---

Impact Assessment

Breaking changes: None. New tables only — existing sessions and session_events tables unchanged.

Components affected: Schema version moves from 2 → 4. Migrations are additive (CREATE TABLE IF NOT EXISTS). Existing databases auto-migrate on server start.

Rollback plan: Drop tables via DROP TABLE IF EXISTS session_facts, session_fact_sources, domains, consolidation_lock, facts, facts_fts, entities, fact_entities, entity_edges, sources, consolidations and reset PRAGMA user_version = 2.

---

Complexity

• Simple (config tweak, dependency update)
• Moderate (new workflow, migration change)
• Complex (core infrastructure change, many dependencies)

---

Checklist

• npm run build succeeds
• npm test passes
• Testing complete (migration tested, workflow triggered, or config validated)
• Code follows project standards
• Code is production-ready and can be reviewed