[semantic] index-integrity enumerates memories_fts via fts5vocab 'instance', which omits zero-token documents — a forgotten memory whose text tokenizes to nothing is never flagged or repaired

[devinmlowe]

Evidence

src/semantic/index-integrity.ts:39-47:

export function ftsIndexedRowids(db) {
  const name = `memories_fts_vocab_...`;
  db.exec(`CREATE VIRTUAL TABLE temp.${name} USING fts5vocab('main', 'memories_fts', 'instance')`);
  try {
    return db.prepare(`SELECT DISTINCT doc FROM temp.${name}`).all().map((r) => r.doc);
  } ...
}

An fts5vocab(... 'instance') table has one row per (term, doc, col, offset). A document whose content/context tokenize to nothing under porter unicode61 (e.g. "...", "---", emoji-only, whitespace) contributes no terms and therefore no rows. It is still a member of the index: FTS5 records it in the memories_fts_docsize shadow table.

Repro (better-sqlite3, in-memory):

insert into m(id,content) values('a','...'),('b','hello world');
insert into memories_fts(rowid,content,context) select rowid,content,context from m;
select distinct doc from <fts5vocab instance>   -> [{doc: 2}]
select id from memories_fts_docsize             -> [{id: 1}, {id: 2}]

Mechanism

auditMemoryIndex (:57-69) iterates only the rowids ftsIndexedRowids returns, so a forgotten memory whose text tokenizes to nothing is never reported in orphanFtsForgotten / orphanFtsMissing, and engram validate --fix never triggers the rebuild for it. Impact is small — a zero-token document can never match a MATCH query, so nothing leaks — but the audit's contract ("must have no FTS entry") is not checked for these rows and validate reports clean on a dirty index.

Suggested fix

Enumerate membership from the shadow table instead of the vocab table: SELECT id FROM memories_fts_docsize (present because the table is created without columnsize=0). One statement, no temp table, and it covers empty documents.

Related: #55.