Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ All notable changes to engram are documented here. The format follows
- A stdio start no longer bridges its per-process scoping away (#87, P1). `engram mcp` / `dist/interfaces/mcp/server.js` with `ENGRAM_SCOPE` or `ENGRAM_READ_SCOPES` in the environment — every Hermes profile child, or any host following `docs/integrate-your-agent.md` — used to bridge to a healthy daemon, which ran each forwarded call under *its* env: every profile wrote `global` and read every scope, silently. Such a start now runs inline without probing, with the reason on stderr (`running inline (ENGRAM_SCOPE=hermes:career is per-process; the daemon would ignore it)`). The daemon's `/health` gained `dbPath`; with `ENGRAM_DB_PATH` set the bridge is used only when it matches the daemon's, otherwise inline (`ENGRAM_DB_PATH=… but the daemon serves …`). A daemon that does not report `dbPath` (older build) keeps bridging as before.
- `forget` no longer honours `scope: "global"` as a scope override when the server's own env pins reads (`ENGRAM_SCOPE` / `ENGRAM_READ_SCOPES`): an env-pinned stdio child (the Hermes memory-provider transport) could pass it to delete, or search for and delete, another tenant's memories (#109). The override still works from an unrestricted server. The tool description says so; the `tool-schemas` snapshot is re-recorded for that sentence only.
- Per-call `scope` / `read_scopes` could escape an env-pinned tenant (#108). When `ENGRAM_SCOPE` / `ENGRAM_READ_SCOPES` are set (a stdio child scoped by the Hermes plugin), `resolveCallScoping` now intersects `read_scopes` with the env read scopes (empty intersection is an error) and rejects a `scope` that is not one of them — a tenant may only write where it may read — instead of letting the params replace the env. The shared HTTP daemon, where the env is unset and the params are the tenant identity, is unchanged.
- `memory_suppressions` was keyed on `content_hash` alone, so one tenant's `forget` suppressed re-extraction of the same sentence for every tenant and one tenant's `remember` lifted every tenant's suppression (#106). The table is rebuilt with `PRIMARY KEY (content_hash, scope)` (checkpoint `suppressions_scope_v1`, existing rows kept, schema version 6); dream extract matches suppressions in the conversation's scope or `global`, and `remember` / `restore` clear only their own scope's row. `clearSuppression` and `filterSuppressedFacts` take a `scope` argument.

### Changed
- Test scaffolding shared instead of copied (#126, part of the #114 over-engineering burn-down). No behaviour change and no test lost: `tests/mocks/embeddings.ts` replaces eleven copies of the hash-based embeddings `vi.mock` factory, `tests/mocks/llm-fetch.ts` the four `stubFetch` / `makeAnthropicMock` copies in the `*-factory` suites, `tests/mocks/fake-worker.ts` the two `FakeWorker` classes; `tests/helpers.ts` gains the raw row inserts (`insertEntity`, `insertRelationship`, `insertCluster`, `insertBridgeScore`, `insertConversation`, `insertExchange`, `insertMemory`), `createTestMemory`, `makeExchanges` and the built-CLI guard (`builtCli`, `itBuilt`), and loses its `cosineSimilarity` copy (tests import `src/_core/search/vector.ts`) and the one-caller `createTestFixture` / `createSyntheticToolCall`. `vitest.config.ts` sets `test.unstubEnvs`, so suites isolate env with `vi.stubEnv(key, undefined)` instead of hand-rolled save/restore loops.
Expand Down
2 changes: 1 addition & 1 deletion src/_core/db/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ SQLite database connection, schema management, and thin data access helpers. All
## Contains

- `connection.ts` — Database connection with WAL mode, PRAGMA tuning
- `schema.ts` — Full schema definition and initialization. Checkpointed migrations in `schema_migrations`: `commitments_v1`, `conversations_scope_v1`, `graph_scope_v1`, `exchanges_author_v1`, `forget_v1` (#55: `memories.deleted_at` / `deleted_by`, `memory_changes`, `memory_suppressions`, `entities.stale_since`, `relationships.stale_since`). Schema version (#65): `SCHEMA_MIGRATIONS` is that list in apply order, `SCHEMA_VERSION` its length, `schemaVersion(db)` the checkpoints a database has recorded, and `BREAKING_MIGRATIONS` the checkpoints an older build cannot read past (empty: every migration is additive). `engram update` persists the version, and `--rollback` refuses a code-only rollback across a breaking one — add a checkpoint to the set in the same change that makes it breaking
- `schema.ts` — Full schema definition and initialization. Checkpointed migrations in `schema_migrations`: `commitments_v1`, `conversations_scope_v1`, `graph_scope_v1`, `exchanges_author_v1`, `forget_v1` (#55: `memories.deleted_at` / `deleted_by`, `memory_changes`, `memory_suppressions`, `entities.stale_since`, `relationships.stale_since`), `suppressions_scope_v1` (#106: `memory_suppressions` rebuilt with `PRIMARY KEY (content_hash, scope)`). Schema version (#65): `SCHEMA_MIGRATIONS` is that list in apply order, `SCHEMA_VERSION` its length, `schemaVersion(db)` the checkpoints a database has recorded, and `BREAKING_MIGRATIONS` the checkpoints an older build cannot read past (empty: every migration is additive). `engram update` persists the version, and `--rollback` refuses a code-only rollback across a breaking one — add a checkpoint to the set in the same change that makes it breaking
- `helpers.ts` — Thin data access helpers (upsert, batch insert)
- `fts.ts` — FTS5 index rebuild and sync utilities
- `vector.ts` — sqlite-vec vector table operations
Expand Down
2 changes: 1 addition & 1 deletion src/_core/db/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Database connection management, schema ownership, and thin data access layer. Pr
- **REQ-3**: The module shall provide typed helpers for common patterns: getById, search, upsert, transaction wrapping. *(traces to ADR-004)*
- **REQ-4**: The module shall expose the raw database connection for complex domain queries. *(traces to ADR-004)*
- **REQ-5**: The module shall let several processes write one WAL database without surfacing `SQLITE_BUSY` on the recall hot path: `busy_timeout = 5000` on every connection, read-then-write transactions opened as `BEGIN IMMEDIATE`, and `withBusyRetry` (`busy.ts`) for best-effort writes. *(#26)*
- **REQ-6**: The schema shall carry the memory lifecycle surface (#55, checkpoint `forget_v1`): nullable `memories.deleted_at` (ISO-8601) and `deleted_by`; `memory_changes(id, memory_id, op ∈ forget|edit|purge|restore, before, after, actor, at)`; `memory_suppressions(content_hash PRIMARY KEY, memory_id, scope, created_at)`; and nullable `entities.stale_since` / `relationships.stale_since`. The migration is idempotent and runs after the entities/relationships rebuild.
- **REQ-6**: The schema shall carry the memory lifecycle surface (#55, checkpoint `forget_v1`): nullable `memories.deleted_at` (ISO-8601) and `deleted_by`; `memory_changes(id, memory_id, op ∈ forget|edit|purge|restore, before, after, actor, at)`; `memory_suppressions(content_hash, memory_id, scope, created_at)` keyed by `(content_hash, scope)` (#106, checkpoint `suppressions_scope_v1`, rebuilt from the `content_hash`-only table keeping every row); and nullable `entities.stale_since` / `relationships.stale_since`. The migrations are idempotent and run after the entities/relationships rebuild.
- **REQ-7**: The module shall expose a schema version (#65): `SCHEMA_MIGRATIONS` (every checkpoint `initDatabase` records, in apply order), `SCHEMA_VERSION` (its length), `schemaVersion(db)` (the checkpoints a database has recorded, read-only, never throwing on a pre-checkpoint schema) and `BREAKING_MIGRATIONS` (the checkpoints an older build cannot read past). `engram update` persists `{version, applied}` before an upgrade and `engram update --rollback` refuses a code-only rollback when a checkpoint added since is in `BREAKING_MIGRATIONS` (decision #66). The set is empty while INV-1 holds by additivity; a change that breaks it must add its checkpoint to the set.

## Concurrency model
Expand Down
2 changes: 2 additions & 0 deletions src/_core/db/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,10 @@ export {
backfillEventTs,
migrateCommitments,
migrateForget,
migrateSuppressionsScope,
COMMITMENTS_MIGRATION,
FORGET_MIGRATION,
SUPPRESSIONS_SCOPE_MIGRATION,
MEMORY_CHANGE_OPS,
EVENT_TS_SUBQUERY,
SCHEMA_MIGRATIONS,
Expand Down
52 changes: 48 additions & 4 deletions src/_core/db/schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -399,6 +399,10 @@ function createSchema(db: Database.Database, config: EngramConfig): void {
// migrateExpandedTypes (entities/relationships rebuild).
migrateForget(db);

// #106: memory_suppressions keyed by (content_hash, scope) so one tenant's
// forget neither silences nor is lifted by another tenant. After migrateForget.
migrateSuppressionsScope(db);

// FTS5 virtual tables (created separately — can't use IF NOT EXISTS)
createFtsIfNeeded(db, "exchanges_fts", `
CREATE VIRTUAL TABLE exchanges_fts USING fts5(
Expand Down Expand Up @@ -756,17 +760,56 @@ export function migrateForget(db: Database.Database): boolean {
CREATE INDEX IF NOT EXISTS idx_memory_changes_at ON memory_changes(at);

CREATE TABLE IF NOT EXISTS memory_suppressions (
content_hash TEXT PRIMARY KEY,
content_hash TEXT NOT NULL,
memory_id TEXT,
scope TEXT DEFAULT 'global',
created_at TEXT NOT NULL
scope TEXT NOT NULL DEFAULT 'global',
created_at TEXT NOT NULL,
PRIMARY KEY (content_hash, scope)
);
`);
if (!hadChanges || !hadSuppressions) added = true;
db.prepare("INSERT OR IGNORE INTO schema_migrations (name) VALUES (?)").run(FORGET_MIGRATION);
return added;
}

/** Checkpoint name recorded in schema_migrations when memory_suppressions is keyed by (content_hash, scope). */
export const SUPPRESSIONS_SCOPE_MIGRATION = "suppressions_scope_v1";

/**
* #106: `memory_suppressions` was keyed on `content_hash` alone, so one
* tenant's forget overwrote (and one tenant's remember lifted) every other
* tenant's suppression of the same sentence. Rebuild the table with
* `PRIMARY KEY (content_hash, scope)`, keeping every existing row (NULL scope
* becomes 'global'). Additive for an older build: its `INSERT OR REPLACE` /
* `WHERE content_hash = ?` statements still run. Idempotent (detects the
* composite key via `PRAGMA table_info`); returns `true` only on the open
* that rebuilt the table.
*/
export function migrateSuppressionsScope(db: Database.Database): boolean {
ensureMigrationsTable(db);
const pkColumns = (db.prepare("PRAGMA table_info(memory_suppressions)").all() as Array<{ name: string; pk: number }>)
.filter((c) => c.pk > 0)
.map((c) => c.name);
const rebuilt = !pkColumns.includes("scope");
if (rebuilt) {
db.transaction(() => db.exec(`
CREATE TABLE memory_suppressions_new (
content_hash TEXT NOT NULL,
memory_id TEXT,
scope TEXT NOT NULL DEFAULT 'global',
created_at TEXT NOT NULL,
PRIMARY KEY (content_hash, scope)
);
INSERT OR IGNORE INTO memory_suppressions_new (content_hash, memory_id, scope, created_at)
SELECT content_hash, memory_id, COALESCE(scope, 'global'), created_at FROM memory_suppressions;
DROP TABLE memory_suppressions;
ALTER TABLE memory_suppressions_new RENAME TO memory_suppressions;
`)).immediate();
}
db.prepare("INSERT OR IGNORE INTO schema_migrations (name) VALUES (?)").run(SUPPRESSIONS_SCOPE_MIGRATION);
return rebuilt;
}

// ─── schema version (#65) ────────────────────────────────────────────

/**
Expand All @@ -781,9 +824,10 @@ export const SCHEMA_MIGRATIONS = [
CONVERSATIONS_SCOPE_MIGRATION,
EXCHANGES_AUTHOR_MIGRATION,
FORGET_MIGRATION,
SUPPRESSIONS_SCOPE_MIGRATION,
] as const;

/** Number of checkpointed migrations this build applies (5 as of `forget_v1`). */
/** Number of checkpointed migrations this build applies (6 as of `suppressions_scope_v1`). */
export const SCHEMA_VERSION: number = SCHEMA_MIGRATIONS.length;

/**
Expand Down
5 changes: 3 additions & 2 deletions src/dream/daemon.ts
Original file line number Diff line number Diff line change
Expand Up @@ -973,7 +973,8 @@ function getConversationScope(db: Database.Database, conversationId: string): st
/**
* #55: drop extracted facts whose content hash is in memory_suppressions — a
* statement the user forgot must not come back from the same exchanges on
* the next run. Counted in the report as `suppressedFacts`.
* the next run. #106: matched in the conversation's own scope (plus
* global). Counted in the report as `suppressedFacts`.
*/
function dropSuppressed(
db: Database.Database,
Expand All @@ -982,7 +983,7 @@ function dropSuppressed(
logPath: string,
report: DreamReport,
): ExtractedFact[] {
const { kept, suppressed } = filterSuppressedFacts(db, facts);
const { kept, suppressed } = filterSuppressedFacts(db, facts, getConversationScope(db, conversationId));
if (suppressed.length > 0) {
report.suppressedFacts = (report.suppressedFacts ?? 0) + suppressed.length;
logEntry(logPath, "extract", `Suppressed ${suppressed.length} forgotten fact(s) re-extracted from ${conversationId}`, {
Expand Down
4 changes: 2 additions & 2 deletions src/interfaces/shared/remember.ts
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ export async function rememberFact(
// statement the user once forgot lifts its extraction suppression (#55).
const newId = crypto.randomUUID();
const now = Math.floor(Date.now() / 1000);
clearSuppression(db, params.content);
clearSuppression(db, params.content, scope);

insertMemory(
db,
Expand Down Expand Up @@ -295,7 +295,7 @@ export async function storeMemoryBatch(
if (!isDuplicate) {
const newId = crypto.randomUUID();
const now = Math.floor(Date.now() / 1000);
clearSuppression(db, input.content);
clearSuppression(db, input.content, scope);

insertMemory(
db,
Expand Down
2 changes: 1 addition & 1 deletion src/semantic/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ The semantic domain extracts, consolidates, and retrieves structured knowledge f
- **REQ-7**: The domain shall mark low-confidence memories as prune-eligible. *(traces to L0 REQ-9)*
- **REQ-8**: The domain shall let a user forget a memory (#55, decision #56): in one transaction set `is_active = 0`, `deleted_at`, `deleted_by`; delete its `vec_memories` and `memories_fts` rows; append a `memory_changes` row (`before` = content, actor); record its content hash in `memory_suppressions`; and detach it from graph evidence (decision #57: decrement `mention_count` once per evidenced entity, stamp `stale_since` at zero, never delete a graph row). A forgotten memory shall not be returned by any recall path (vector, FTS, hybrid search, session drill, active-memory readers) nor reinforced. Hard deletion removes the row; the dream prune phase hard-deletes rows older than `ENGRAM_FORGET_RETENTION_DAYS`.
- **REQ-9**: The domain shall provide edit (re-embed + re-index, logged with before/after), restore (inside the retention window; lifts the suppression), purge-by-conversation, a change log, and provenance inspection (source exchanges and conversations, source/extraction basis, scope, FSRS health, graph evidence). Forget/edit/restore shall refuse a memory outside the caller's read scopes unless scope `global` is passed explicitly (#25).
- **REQ-10**: Extraction shall skip a fact whose content hash is in `memory_suppressions` (dream extract counts them as `suppressedFacts`); an explicit remember of the same content lifts the suppression.
- **REQ-10**: Extraction shall skip a fact whose content hash is in `memory_suppressions` for the conversation's scope or `global` (dream extract counts them as `suppressedFacts`); an explicit remember of the same content in a scope lifts only that scope's suppression (#106: one tenant's forget neither silences nor is lifted by another tenant).

## Interface Contract

Expand Down
27 changes: 20 additions & 7 deletions src/semantic/forget.ts
Original file line number Diff line number Diff line change
Expand Up @@ -167,27 +167,38 @@ export function contentHash(content: string): string {
return createHash("sha256").update(normalized).digest("hex");
}

/** Remove the suppression for this content (an explicit remember wins). Returns true when one existed. */
export function clearSuppression(db: Database.Database, content: string): boolean {
const r = db.prepare("DELETE FROM memory_suppressions WHERE content_hash = ?").run(contentHash(content));
/**
* Remove the suppression of this content in one scope (an explicit remember
* wins). #106: keyed by tenant, so a remember in `hermes:personal` never
* lifts a `hermes:career` (or global) forget. Returns true when one existed.
*/
export function clearSuppression(db: Database.Database, content: string, scope: string): boolean {
const r = db
.prepare("DELETE FROM memory_suppressions WHERE content_hash = ? AND scope = ?")
.run(contentHash(content), scope);
return r.changes > 0;
}

/**
* Split extracted facts into the ones to keep and the ones whose content
* hash is suppressed (a forgotten statement re-extracted from the same
* exchanges). Used by dream extract; the count feeds the phase summary.
* exchanges). #106: a suppression applies to its own scope; only a global one
* applies to every tenant. Used by dream extract; the count feeds the phase
* summary.
*/
export function filterSuppressedFacts<T extends Pick<ExtractedFact, "content">>(
db: Database.Database,
facts: readonly T[],
scope: string,
): { kept: T[]; suppressed: T[] } {
if (facts.length === 0) return { kept: [], suppressed: [] };
const lookup = db.prepare("SELECT 1 FROM memory_suppressions WHERE content_hash = ?");
const lookup = db.prepare(
"SELECT 1 FROM memory_suppressions WHERE content_hash = ? AND scope IN (?, 'global')",
);
const kept: T[] = [];
const suppressed: T[] = [];
for (const fact of facts) {
if (lookup.get(contentHash(fact.content))) suppressed.push(fact);
if (lookup.get(contentHash(fact.content), scope)) suppressed.push(fact);
else kept.push(fact);
}
return { kept, suppressed };
Expand Down Expand Up @@ -445,7 +456,9 @@ export function restoreMemory(db: Database.Database, options: RestoreOptions): M
"UPDATE memories SET is_active = 1, deleted_at = NULL, deleted_by = NULL, updated_at = unixepoch() WHERE id = ?",
).run(row.id);
reindex(db, row, options.embedding);
db.prepare("DELETE FROM memory_suppressions WHERE content_hash = ?").run(contentHash(row.content));
db
.prepare("DELETE FROM memory_suppressions WHERE content_hash = ? AND scope = ?")
.run(contentHash(row.content), row.scope ?? GLOBAL_SCOPE);
return logChange(db, { memoryId: row.id, op: "restore", before: null, after: row.content, actor: options.actor, at: now });
});
return run.immediate();
Expand Down
Loading
Loading