You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up to #6007, deliberately out of scope there (that fix is forward-only).
Why this is needed
#6007 fixed the routing: MemorySourceSink::accept_source_items now feeds each connector item into the memory-tree ingest funnel, so new Composio syncs produce mem_tree_chunks rows. Fix in tinyhumansai/tinymemory#134.
That does nothing for records already on disk. An affected profile has connector content sitting in the namespace document store with no tree rows at all:
Those memories stay invisible to tree-backed recall, the Memory Tree graph, and the source row's ingest status until something writes the tree rows.
Why a re-sync may not be enough
The obvious remedy — "just hit Sync again" — is not reliable. The per-item sync gate treats already-ingested items as done, so a re-sync fetches nothing new and creates no tree rows. Recovery today likely means removing and re-adding the source, which re-fetches the whole account (real Composio spend) and is not a thing to ask a user to work out for themselves.
Rebuilding or wiping the memory tree does not help either: the tree is built from tree-ingested content, and this content never entered that path.
What a backfill has to do
Read the existing namespace documents and create the corresponding tree rows, without re-fetching from the provider:
Enumerate memory_docs in the connector namespaces — source:{toolkit}:{connection_id}, plus the legacy skill-{toolkit} namespaces from before the connector migration.
For each document, call the shared funnel tinymemory_core::engine::ingest_connector_item_into_tree (added in fix(sources): route connector items into the memory tree tinymemory#134 — it already owns the scope rules, so the backfill must not re-derive them). Identity: path_scope = {toolkit}:{connection_id}, per-item source_id = {toolkit}:{connection_id}:{item_id}.
Idempotent: re-running must not duplicate chunks. Worth deciding whether the per-item ingest gate already gives this or whether the backfill needs its own watermark.
Batched and interruptible. ~3000 documents on the profile above, each producing chunk embeddings — see the cost note below.
A trigger surface: on module upgrade, on demand from the memory settings UI, or a maintenance member on the contract. Also needs progress reporting, since it is long-running.
Cost
Backfilling embeds every recovered chunk. The profile above implies roughly 8000+ chunk embeddings for one Gmail account, which lands squarely on the embedding budget that #5324 added exhaustion warnings for. Whatever triggers this should be visible and interruptible rather than silently spending a user's budget on upgrade.
Notes
Mostly a tinymemory change (the funnel and the document store both live there), with an openhuman trigger surface. Should not be started before tinyhumansai/tinymemory#134 lands, since it depends on the shared funnel that PR introduces.
Follow-up to #6007, deliberately out of scope there (that fix is forward-only).
Why this is needed
#6007 fixed the routing:
MemorySourceSink::accept_source_itemsnow feeds each connector item into the memory-tree ingest funnel, so new Composio syncs producemem_tree_chunksrows. Fix in tinyhumansai/tinymemory#134.That does nothing for records already on disk. An affected profile has connector content sitting in the namespace document store with no tree rows at all:
memory/memory.db—source_gmail*: 2120 docs, 7986 vector chunks, 7986 embeddings; legacyskill-gmail: 958 docs, 33800 vector chunksmemory_tree/chunks.db— Gmail tree chunks: 0Those memories stay invisible to tree-backed recall, the Memory Tree graph, and the source row's ingest status until something writes the tree rows.
Why a re-sync may not be enough
The obvious remedy — "just hit Sync again" — is not reliable. The per-item sync gate treats already-ingested items as done, so a re-sync fetches nothing new and creates no tree rows. Recovery today likely means removing and re-adding the source, which re-fetches the whole account (real Composio spend) and is not a thing to ask a user to work out for themselves.
Rebuilding or wiping the memory tree does not help either: the tree is built from tree-ingested content, and this content never entered that path.
What a backfill has to do
Read the existing namespace documents and create the corresponding tree rows, without re-fetching from the provider:
memory_docsin the connector namespaces —source:{toolkit}:{connection_id}, plus the legacyskill-{toolkit}namespaces from before the connector migration.tinymemory_core::engine::ingest_connector_item_into_tree(added in fix(sources): route connector items into the memory tree tinymemory#134 — it already owns the scope rules, so the backfill must not re-derive them). Identity:path_scope={toolkit}:{connection_id}, per-itemsource_id={toolkit}:{connection_id}:{item_id}.Cost
Backfilling embeds every recovered chunk. The profile above implies roughly 8000+ chunk embeddings for one Gmail account, which lands squarely on the embedding budget that #5324 added exhaustion warnings for. Whatever triggers this should be visible and interruptible rather than silently spending a user's budget on upgrade.
Notes
Mostly a tinymemory change (the funnel and the document store both live there), with an openhuman trigger surface. Should not be started before tinyhumansai/tinymemory#134 lands, since it depends on the shared funnel that PR introduces.
Refs #6007