Memory sources: Sync History never records a manual sync, and "Queued" points at a queue no screen shows

[YellowSnnowmann]

Summary

Pressing Sync or Apply all on Brain › Sources never adds a row to Brain › Sync › Sync History, for any source kind. The runs succeed and their items land in memory, but the history only ever shows runs of the 20-minute background scheduler for folder-type sources. Around it, the only "queue" the screens mention is one no screen lists, and a manual Folder, GitHub repo, RSS or web page sync never reports that it finished.

Problem

Reported (paraphrased): the sync in Brain says it was added to a queue, but there is no way to see what the queue is or what is happening, and after the sync Brain › Sync › Sync history does not update.

Steps to reproduce

1. Brain › Sources: on a connected Gmail, Slack, Notion or GitHub source, press Sync and wait for the row's result chip ("N items synced").
2. Open Brain › Sync. Sync History has no row for the run, and leaving the tab and coming back does not add one.
3. Add a Folder source containing at least one file and press Sync. The row shows "Queued · queued chunk extraction for mem_src:…", never shows a result, and Sync History again gets no row.

Expected: every run (Sync button, Apply all, background, first sync after connecting) appears in Sync History with its item count, duration and outcome, without leaving the tab. The source row names what is happening in plain words and ends with a result.

Observed on a local staging profile, 2026-09-10: five manual syncs (Slack ×3, Gmail ×2, one Gmail run writing 100 items) added nothing to <workspace>/memory_tree/sync_audit.jsonl. The Sync tab fetched the log eight times across four visits in under two minutes and got an empty history each time; the file's only row was written later, by a background folder sync with 0 items. On three other local profiles the newest Composio row is from 2026-08-13, 2026-08-14 and 2026-08-27.

Impact

• Sync History cannot be used to verify a sync, which is what it is for.
• A manual Folder / GitHub repo / RSS / web page sync leaves its row on "Queued" for up to 30 minutes and never triggers the post-sync embedding pass.
• The monthly sync-cost summary reads the same log and under-counts.

Version: code references are main at 1cc1f59 with the tinymemory module at v1.16.0 and tinyconnectors at v0.9.0; the field evidence is from a macOS desktop build against staging.

Solution (optional)

Root cause

Sync History is <workspace>/memory_tree/sync_audit.jsonl, read through openhuman.memory_sources_sync_audit_log (cost_reporting.rs#L33). The read is correct (newest first, capped at 1,000 rows in engine/mod.rs#L3306-L3324). Three defects stack on top of it.

1. No manual path writes a row. The one remaining writer is sources::sync::sync_source in tinymemory (sync.rs#L168-L196, #L214-L242). Its only caller is the background scheduler (periodic.rs#L235), which handles GithubRepo, Folder, RssFeed and WebPage only (#L68-L73). The Sync button (source_sync.rs#L109-L181) and Apply all (apply_all.rs#L114-L147) dispatch elsewhere:

• Composio rows go to composio_sync_budgeted (providers_ops.rs#L272-L454), which publishes running / completed / failed and records nothing.
• Every other kind goes to MemorySourceSync::run_source_sync, which the engine serves with run_source_pipeline (engine/mod.rs#L3242-L3266). That records nothing and emits no requested / completed / failed stage either, because emit_sync_stage is only called inside sync_source. The tinymemory bus test pins the missing row on purpose (module_e2e.rs#L1900-L1916).

The remaining Composio runs share the gap: the background loop, the first sync after connecting and the Slack ingest RPC all go through run_sync_within_budget (pass_budget.rs#L87-L97).

Two changes introduced it:

• Route the recall, reset and flush paths through the contract #5725 (14a23b994, merged 2026-08-26) moved the Sync RPC and Apply all from sync_source, which wrote the row and the start and finish stages, onto run_source_sync, which does neither.
• Remove the Composio connector tree tinymemory#118 (cb9221b, merged 2026-08-30, first released in v1.13.4) removed the engine's Composio sync loop and the audit rows it wrote. The host-side replacement never gained a writer.

2. The panel loads once. SyncAuditPanel fetches in a mount-only effect with no polling and no event subscription (SyncAuditPanel.tsx#L58-L73), so even a row that is written stays invisible until the tab remounts.

3. The "queue" is not visible anywhere. The queue wording on these screens is either the Sources row's live stage or the Sync tab's retry toast:

• The sync-stage bridge emits stored and then queued ("queued chunk extraction for …") for every canonicalized document (sync_events_bridge.rs#L124-L141), and the row prints the stage name as-is (MemorySourceRow.tsx#L176).
• Retry failed jobs on the Sync tab toasts "Failed jobs requeued" / "Jobs queued to run again: N" (MemoryTreeStatusPanel.tsx#L130-L139).

Both refer to the memory-tree job queue, but the Sync tab renders only its failed count (MemoryTreeStatusPanel.tsx#L192), never the ready (queued) count it already polls (memoryTree.ts#L892-L893). Ingestion queue depth appears only in the chat screen's background-activity rows. Because the manual driver path never emits a terminal stage, the row stays on "Queued" until memory::sync_activity gives up on it after 30 minutes (sync_activity.rs#L49), and the embedding trigger that waits for completed (sync_events_bridge.rs#L83-L90) never runs.

Proposed fix

All in openhuman; no module release or re-pin.

PR 1: core records and finishes every run

• A host-owned run log at <workspace>/state/sync_runs.jsonl, appended with one write_all per line. sync_audit_log_rpc and the monthly summary return the driver's log merged with it, newest first, capped.
• One helper shared by the Sync RPC and Apply all for driver kinds: publish running, await run_source_sync, publish completed with "ingested N item(s)" (the detail the app already parses) or failed, then record the row with the same verdict fields (tree_ingest_failures, tree_error).
• Composio runs recorded where composio_sync_budgeted settles and inside run_sync_within_budget: source_kind = composio, the source id when known, scope = {toolkit}:{connection_id}, items written, duration and outcome. Recording is best-effort; a failed append logs a warning and never fails the sync.
• A terminal stage wins in memory::sync_activity: a per-document queued that arrives after completed (there are two publishers) must not reopen the run.

Not chosen: a new tinymemory contract member. It would need a new wire slot, a minor release and a re-pin, and it would bring Composio back into the engine that deliberately dropped it.

PR 2: the app shows the queue and refreshes

• SyncAuditPanel refetches when a run ends (subscribeTerminalSyncEvents), polls while a sync is live, and gets a Refresh button.
• A "Now syncing" card on the Sync tab: in-flight sources from useMemorySyncActivity() plus queued, running and failed job counts.
• Plain-language stage labels in every locale instead of raw stage names and mem_src: ids, with the same terminal-wins rule in applyStageEvent.
• History rows labelled from the source registry. scopeLabel expects gmail:<email> and would print gmail:ca_… for the new rows.
• Remove the unused memorySources.sync.successTitle and memorySources.sync.successMessage keys.

Acceptance criteria

• Repro gone — a Sync-button run, an Apply-all run, a background Composio run and a first sync after connecting each add exactly one Sync History row with item count, duration and outcome; a failed run adds a failed row.
• Live history — a run that ends while Brain › Sync is open appears without leaving the tab.
• Manual driver runs finish — a manual Folder, GitHub repo, RSS or web page sync emits completed or failed, the row shows its result, and the post-sync embedding pass is triggered.
• No reopen after finish — a late per-document queued after completed does not relight the row, in either memory::sync_activity or the app store.
• Queue visible — the Sync tab shows in-flight sources and queued, running and failed job counts; no row shows a raw stage name or mem_src: id.
• Regression safety — Rust tests for rows and stages on both dispatch paths (success and failure), the merged read's ordering and cap, and the terminal-wins guard; a JSON-RPC E2E in which memory_sources_sync grows memory_sources_sync_audit_log by one row; Vitest for the panel refresh and the Sync-tab card.
• Diff coverage ≥ 80% — the fix PRs meet the changed-lines coverage gate (Vitest + cargo-llvm-cov, enforced by .github/workflows/ci-lite.yml).

Related

• Route the recall, reset and flush paths through the contract #5725 moved the Sync button and Apply all onto run_source_sync; Manual memory-source Sync button refuses with 'unsupported capability: source_sync' while scheduled sync works: #5725 routed it onto a contract member the host bridge never implemented #5801 was that move's first breakage.
• Memory sources: leaving Brain › Sources mid-sync drops the live progress and the run looks stopped #6019 added memory::sync_activity and the app-side sync store this fix builds on.
• Memory-tree corruption reported as "non-fatal" for 34min while the UI showed successful syncs; recovery then wiped the source registry #5820 defined the audit verdict fields a recorded run must keep.
• Gmail/Composio sync writes docs and vectors but skips memory-tree ingest after migration #6007 introduced SyncDispatch, the seam PR 1 hooks into.
• Remove the Composio connector tree tinymemory#118 removed the engine's Composio sync and its audit writer.
• Drive the connector-sync families over the bus, not just assert they route tinymemory#149 added the bus test that pins RunSourceSync as an unaudited path.