Skip to content

perf(workflow): batch step persistence after execution - #722

Open
xiami762 wants to merge 5 commits into
devfrom
fix/workflow-step-storage-wait
Open

perf(workflow): batch step persistence after execution#722
xiami762 wants to merge 5 commits into
devfrom
fix/workflow-step-storage-wait

Conversation

@xiami762

@xiami762 xiami762 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • make workflow step callbacks pure in-memory collectors with no SQLite calls or event-loop waits
  • persist all steps, the final execution summary, workflow stats, and retention changes in one atomic completion transaction
  • use a dedicated completion connection plus a completion lock to prevent transaction interleaving
  • skip the initial queued-row write for Kafka, Syslog, Poller, and custom high-frequency triggers while preserving complete step history at the end
  • keep the manual run queued record, ToolContext lifecycle, cancellation, backpressure, restart, and no-overlap behavior

Performance validation

Real SQLite benchmark with 4 concurrent executions and 7 steps each:

  • callback SQL calls: 0
  • callback time per worker: 0.015-0.037 ms
  • completion wall time for all 4 executions: 1.363 ms
  • completion commits: 4
  • persisted steps: 28
  • persisted executions: 4
  • workflow stats call count: 4

Test coverage

  • targeted workflow persistence suite: 85 passed
  • broader workflow and ingest regression suite: 374 passed, 2 skipped
  • 2 unrelated NDR workflow fixture failures remain on the broader unfiltered suite
  • Ruff and git diff checks pass

Persistence guarantees

  • step callbacks never call record_step, record_steps, or run_coroutine_threadsafe
  • each completed execution commits once
  • failures rollback steps, execution summary, stats, and retention together
  • executionLog remains empty in the summary row; step details remain queryable from the step table

@xiami762 xiami762 changed the title perf(workflow): remove synchronous step storage waits perf(workflow): batch step persistence after execution Aug 21, 2026
xiami762 and others added 3 commits August 21, 2026 17:01
Keep workflow callbacks storage-free while batching complete step history and serializing interactive progress writes. Isolate terminal persistence from stats and retention failures.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove obsolete persistence options and duplicated step/row handling while preserving atomic completion and nonblocking progress behavior.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant