Skip to content

Flows: first-class exactly-once / dedup primitive (never re-process a keyed item) #5224

Description

@graycyrus

Problem

A Workflow can act but cannot reliably tell whether it already did the same work. The workflow_builder improvises broken dedup — e.g. a code node reading a posted_items.json that nothing ever writes, so a "newsletter that never reposts" reposts everything. Hand-wiring recall → condition → remember is fragile and the LLM gets it wrong. Semantic memory (flow_memory_recall) is also wrong here: dedup needs exact keyed membership, not similarity.

The right fix: an engine primitive

A first-class dedup node kind (13th kind) in tinyflows with commit-on-success semantics:

  • Config: { "key": "=item.id" } — an =-expression yielding a stable per-item key.
  • Filter (mid-run, in the node): unseen keys pass through and are written tentative to the durable per-flow flow_state KV; seen keys are dropped.
  • Commit/release (host-side, on FlowRunFinished): run succeeds → tentative keys commit to the permanent seen-set; run fails/cancels → tentative keys released so those items retry next run.

A failed action therefore never marks an item done. Exact keyed membership over durable state; survives restarts and scheduled runs.

Commit granularity

Run-level (the tinyflows engine has no per-item success isolation — one item's failure fails the whole node). This is true exactly-once for single-action flows (the newsletter: compose one digest → send one email). For flows with N separate per-item side-effects it degrades to at-least-once (safe direction: retry, never silently drop). The node config reserves a future additive commit: "per_item" mode (paired with continue-on-error actions) if measured need arises.

Delivery (2 PRs)

  1. tinyflows (vendor): NodeKind::Dedup + DedupNode executor (nodes/control_flow/dedup.rs) + catalog contract + validation + tests (incl. failed-action-does-not-commit).
  2. submodule bump + host + frontend: DedupCommitSubscriber on FlowRunFinished (flows/bus.rs), kv_delete in flows/store.rs, node_contracts.rs overlay (12→13), builder-prompt teaching, types.ts/nodeKindMeta.ts/DedupConfig.tsx, i18n ×14. CURRENT_SCHEMA_VERSION stays 1 (additive).

Drift points

NODE_KINDS (catalog.rs), node_contracts.rs count, types.ts + nodeKindMeta.ts (all 12→13).

Part of the memory-in-workflows track (#5150 #5175 #5176 #5205). This is the deterministic-dedup piece the read-only memory agent should not be doing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions