Repair older memories re-scans the same 500 already-filed documents on every click and never reaches the pre-migration skill-gmail records

[YellowSnnowmann]

Summary

"Repair older memories" (the #6012 backfill, shipped in #6015 / #6017 / tinyhumansai/tinymemory#136) re-examines the same 500 already-filed documents on every click and never reaches the documents it was built for. After the first click it reports Filed 0 into the memory tree (500 already there, 0 skipped). More remain. Click Repair older memories again. on every run, forever.

Problem

Steps to reproduce (desktop, macOS, memory module pinned at tinymemory-module 1.14.1; profile with one Gmail and one Notion connection plus pre-migration skill-gmail documents):

1. Brain → Sources → Repair older memories → confirm. First pass files a small number and says "More remain".
2. Click Repair older memories again.
3. Toast: Filed 0 into the memory tree (500 already there, 0 skipped). More remain. Click Repair older memories again.
4. Repeat. Same toast every time. The confirmation modal also keeps saying "Up to 500 synced documents were stored before memory-tree filing was fixed".

Expected: each pass files up to 500 documents that are not yet in the tree, and after enough passes the button reports nothing to repair.

Actual — ~/.openhuman/logs/openhuman.2026-09-07.log (tinymemory_core::backfill pass-complete lines, dry_run=false only):

time	scanned	ingested	already_present	more_pending
13:14:19	500	20	480	true
13:18:32	500	0	500	true
13:18:52	500	0	500	true
13:29:54	500	0	500	true

Store on that profile (memory/memory.db memory_docs vs memory_tree/chunks.db mem_tree_ingested_sources), in the order the backfill walks its targets:

target namespace	docs	already in tree	outcome
source:notion:ca_-hh_1bNgCAsI	20	20	filed by the first click
source:gmail:ca_SdxSfFuXHLGn	600	600	synced after the #6007 fix, so the sync path treed them
skill-gmail (legacy)	958	0	never reached
skill-notion (legacy)	2	0	never reached

Notion's 20 plus the first 480 Gmail documents fill the 500-document budget, all of them already in the tree, and the pass stops there. The 960 pre-migration documents — the exact records #6012 exists to recover — sit behind that wall and are still invisible to tree-backed recall, the Memory Tree graph and the source row's ingest status. Only the first 500 list positions can ever be filed by this button.

Solution (optional)

Root cause is in the walk, crates/tinymemory-core/src/backfill.rs in tinyhumansai/tinymemory (unchanged from v1.13.8 through v1.15.0). Three things combine:

• No cursor, by design. The module doc calls the pass "idempotent by construction, not by bookkeeping" and BackfillTreesRequest.limit says "resuming is just calling again". Every pass restarts at the head of list_documents(namespace) for each target, in registry order with the legacy skill-{toolkit} namespaces last.
• Already-filed documents consume the budget. report.scanned is incremented before the document is read or ingested (backfill.rs#L140), and scanned is what the limit check compares against (#L126-L128). The ingest gate's already_ingested answer arrives after the slot is spent. break 'targets on the limit then skips every later namespace.
• The list is newest-first. NamespaceStore::list_documents is ORDER BY updated_at DESC (documents.rs#L526), and the newest documents are precisely the ones the post-Gmail/Composio sync writes docs and vectors but skips memory-tree ingest after migration #6007 sync path already filed. So the head of every list is "already there", on every profile that has synced since the fix.

The UI never sends a limit (app/src/components/intelligence/MemorySourcesRegistry.tsx calls the RPC with dryRun only), so the driver default of 500 (DEFAULT_BACKFILL_LIMIT) always applies. The FE, the host RPC (backfill_connector_trees_rpc) and the wire types are all behaving as written; the fix is in tinymemory-core, followed by a patch release and a module re-pin (#6049 is the pending re-pin to v1.15.0 and could move to the patched version).

Secondary: the dry run counts without probing the tree gate, so it reports already_present=0 and the confirmation modal says "Up to 500" even when all 500 are already filed.

Proposed fix (tinymemory-core only, no wire change):

1. Add one connector_item_source_id(toolkit, connection_id, item_id) beside ingest_connector_item_into_tree in engine/sync.rs, so the walk and the funnel share a single owner of the {toolkit}:{connection_id}:{item_id} identity (the Gmail/Composio sync writes docs and vectors but skips memory-tree ingest after migration #6007 lesson; the gate rows on the profile above are exactly that bare id under source_kind = document, so a pre-probe matches).
2. In the walk, probe is_source_ingested(config, SourceKind::Document, &id) first (one primary-key lookup per document). Count already_present there without charging limit; charge limit only for documents that need a read and an ingest. The transactional gate inside the ingest stays authoritative, so a race just becomes an already_ingested result counted as before.
3. Dry run does the same probe, so scanned / the modal's "Up to {scanned}" means documents that still need filing.
4. Test: a second bounded pass advances past already-filed documents and eventually reports more_pending = false. The existing a_bounded_pass_reports_that_more_is_pending only asserts the flag.

Workaround today: invoke the RPC with an explicit bound, e.g. memory_tree.backfill_connector_trees with { "dry_run": false, "limit": 2000 }. The handler accepts limit, and one pass then covers all targets. Not something to ask a user to do.

Acceptance criteria

• Repro gone — On a profile whose first 500 list positions are already filed, a real pass files documents from later positions and later targets (including the legacy skill-{toolkit} namespaces); repeated clicks converge on "Nothing to repair".
• Budget semantics — Already-filed documents no longer consume limit; already_present is reported for them in both dry and real passes.
• Preview honest — The dry-run scanned (and the modal's "Up to {scanned}") counts only documents still waiting to be filed.
• Regression safety — tinymemory-core test covering a second bounded pass advancing past already-filed documents, plus the existing backfill tests staying green.
• Diff coverage ≥ 80% — the fix PR meets the changed-lines coverage gate (Vitest + cargo-llvm-cov, enforced by .github/workflows/ci-lite.yml).
• Shipped to users — tinymemory patch release and openhuman module re-pin (registry, gitlink, capability pin, CI workflow digests) so the fix reaches the prebuilt module.

Related

• Backfill existing connector memories into the memory tree after the #6007 routing fix #6012 (the backfill this button exposes), Gmail/Composio sync writes docs and vectors but skips memory-tree ingest after migration #6007 (the forward-only routing fix it complements), Memory embedding failures silently store chunks without vectors — misleading 'managed budget' banner shown instead of auth error #5324 (embedding-budget cost of a pass)
• Shipping PRs: feat(memory): backfill connector memories into the memory tree #6015 (RPC and wire), fix(memory-sources): honour the sync depth, say why a sync stopped, repair older memories, release leases on quit #6017 (Repair button), feat(maintenance): backfill stored connector documents into the memory tree tinymemory#136 (the walk)
• chore(modules): re-pin tinymemory to v1.15.0 #6049 (pending re-pin to tinymemory v1.15.0)

[YellowSnnowmann]

Fix PR: tinyhumansai/tinymemory#144 (tinymemory-core only; no wire change). Verified on a copy of the reporting profile: bounded passes converge (500 → 458 → nothing waiting). Keeping this open until the patch release and the module re-pin ship the fix. One more finding on that profile, to be filed separately: the registry carries three stale duplicate rows for a second gmail connection, so resolve_targets counts 4 gmail connections and skips the legacy skill-gmail namespace as ambiguous — the reason lands in notes, which the UI does not show.