Skip to content

Backfill existing connector memories into the memory tree after the #6007 routing fix #6012

Description

@YellowSnnowmann

Follow-up to #6007, deliberately out of scope there (that fix is forward-only).

Why this is needed

#6007 fixed the routing: MemorySourceSink::accept_source_items now feeds each connector item into the memory-tree ingest funnel, so new Composio syncs produce mem_tree_chunks rows. Fix in tinyhumansai/tinymemory#134.

That does nothing for records already on disk. An affected profile has connector content sitting in the namespace document store with no tree rows at all:

  • memory/memory.dbsource_gmail*: 2120 docs, 7986 vector chunks, 7986 embeddings; legacy skill-gmail: 958 docs, 33800 vector chunks
  • memory_tree/chunks.db — Gmail tree chunks: 0

Those memories stay invisible to tree-backed recall, the Memory Tree graph, and the source row's ingest status until something writes the tree rows.

Why a re-sync may not be enough

The obvious remedy — "just hit Sync again" — is not reliable. The per-item sync gate treats already-ingested items as done, so a re-sync fetches nothing new and creates no tree rows. Recovery today likely means removing and re-adding the source, which re-fetches the whole account (real Composio spend) and is not a thing to ask a user to work out for themselves.

Rebuilding or wiping the memory tree does not help either: the tree is built from tree-ingested content, and this content never entered that path.

What a backfill has to do

Read the existing namespace documents and create the corresponding tree rows, without re-fetching from the provider:

  1. Enumerate memory_docs in the connector namespaces — source:{toolkit}:{connection_id}, plus the legacy skill-{toolkit} namespaces from before the connector migration.
  2. For each document, call the shared funnel tinymemory_core::engine::ingest_connector_item_into_tree (added in fix(sources): route connector items into the memory tree tinymemory#134 — it already owns the scope rules, so the backfill must not re-derive them). Identity: path_scope = {toolkit}:{connection_id}, per-item source_id = {toolkit}:{connection_id}:{item_id}.
  3. Idempotent: re-running must not duplicate chunks. Worth deciding whether the per-item ingest gate already gives this or whether the backfill needs its own watermark.
  4. Batched and interruptible. ~3000 documents on the profile above, each producing chunk embeddings — see the cost note below.
  5. A trigger surface: on module upgrade, on demand from the memory settings UI, or a maintenance member on the contract. Also needs progress reporting, since it is long-running.

Cost

Backfilling embeds every recovered chunk. The profile above implies roughly 8000+ chunk embeddings for one Gmail account, which lands squarely on the embedding budget that #5324 added exhaustion warnings for. Whatever triggers this should be visible and interruptible rather than silently spending a user's budget on upgrade.

Notes

Mostly a tinymemory change (the funnel and the document store both live there), with an openhuman trigger surface. Should not be started before tinyhumansai/tinymemory#134 lands, since it depends on the shared funnel that PR introduces.

Refs #6007

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

priority: p2Soon. Real but survivable — a rough edge, a gap, a thing that will bite later.

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions