fix(core): make workflow runs durable across resumes - #23
Conversation
Persist runWorkflow state to the JSONL database, expose builder resume options, and retry stale running steps after interruption.
📝 WalkthroughWalkthroughWorkflow execution now persists runs to JSONL storage, accepts an explicit resume ID, and resets interrupted running steps to pending so they can execute again. Tests cover database wiring, resume routing, and stale-step recovery. ChangesWorkflow resume and persistence
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant WorkflowBuilder
participant JsonFileWorkflowDb
participant WorkflowRunner
participant WorkflowStep
Caller->>WorkflowBuilder: run with resume ID
WorkflowBuilder->>JsonFileWorkflowDb: load persisted run
WorkflowBuilder->>WorkflowRunner: resume run with resetRunningSteps
WorkflowRunner->>JsonFileWorkflowDb: reset running steps to pending
WorkflowRunner->>WorkflowStep: re-execute stale steps
WorkflowStep-->>WorkflowRunner: complete steps
WorkflowRunner-->>WorkflowBuilder: return resumed run
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The initial commit dropped the third argument from builder.ts's two runner.resume() calls. That argument feeds reconstructRunFromCache(runId, config), the cached-step-output fallback used when workflow-runs.jsonl is absent -- exactly what resume-fallback.test.ts covers. Restore it. run-persistence.test.ts also could not fail: its fixture used 'steps: []', which validateWorkflow rejects, so parseYamlFile threw before any assertion ran. Give it a real deterministic step, and strengthen the builder-resume assertion to require the config argument so this regression cannot recur.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/core/src/runner.ts (1)
3688-3700: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winClear persisted retry counts when resetting a step.
retryCountis retained across the reset, so a step that succeeds on its first resumed attempt can still report old retries. Reset it in memory and in the DB patch.Proposed fix
state.row.status = 'pending'; state.row.error = undefined; state.row.completionReason = undefined; + state.row.retryCount = 0; await this.db.updateStep(state.row.id, { status: 'pending', error: undefined, completionReason: undefined, + retryCount: 0, updatedAt: new Date().toISOString(), });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/core/src/runner.ts` around lines 3688 - 3700, Update the failed/running step reset loop to clear retryCount alongside status, error, and completionReason. Reset state.row.retryCount in memory and include retryCount reset in the updateStep database patch, preserving the existing pending-reset behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/core/src/run.ts`:
- Around line 56-61: Update the resume flow in run.ts to pass the parsed config
to WorkflowRunner.resume(), ensuring cache-only reconstruction via
reconstructRunFromCache() receives the required configuration. Preserve the
existing runner setup and resume behavior otherwise.
---
Outside diff comments:
In `@packages/core/src/runner.ts`:
- Around line 3688-3700: Update the failed/running step reset loop to clear
retryCount alongside status, error, and completionReason. Reset
state.row.retryCount in memory and include retryCount reset in the updateStep
database patch, preserving the existing pending-reset behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1b3541e5-0f59-4bbc-9b53-1f2412a1a45f
📒 Files selected for processing (5)
packages/core/src/__tests__/resume-fallback.test.tspackages/core/src/__tests__/run-persistence.test.tspackages/core/src/builder.tspackages/core/src/run.tspackages/core/src/runner.ts
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e2a2e73b57
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // a running step after a run is resumed. | ||
| for (const [, state] of stepStates) { | ||
| if (state.row.status === 'failed') { | ||
| if (state.row.status === 'failed' || state.row.status === 'running') { |
There was a problem hiding this comment.
When another process calls --resume while the original runner is still alive, resume() accepts the run because its status is running, and this branch unconditionally changes every active step to pending. The resumed scheduler then executes those steps again while the original process may still be executing them, duplicating non-idempotent commands or agent side effects and racing JSONL updates. Only requeue a running step after confirming that its owning runner is no longer active, or reject resumes of actively leased runs.
Useful? React with 👍 / 👎.
- run.ts: pass the parsed config to resume(). Without it, cache-only resume fails when workflow-runs.jsonl is unavailable, because resume() needs the config for reconstructRunFromCache(). This bug predates the PR; builder.ts was already passing it. - runner.ts: clear retryCount when resetting a step, so a step that succeeds on its first resumed attempt no longer reports stale retries. - runner.ts: gate the running-step reset behind ResumeOptions.resetRunningSteps (default false). Runs carry no lease or heartbeat, so a live owner cannot be detected; unconditionally requeueing running steps let a second resume re-execute them alongside the original process and duplicate non-idempotent side effects. The user-facing resume paths (run.ts, builder.ts) opt in, because --resume explicitly means the previous process is gone. A real ownership lease is the proper fix and is out of scope here.
|
Review feedback addressed in 1.
|
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/core/src/__tests__/resume-fallback.test.ts (1)
284-287: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAssert the retry-count and metadata reset explicitly.
The fixture starts with
retryCount: 0,error: undefined, andcompletionReason: undefined, so this test does not prove that stale values are cleared. Seed the running step with non-zero/stale metadata and includeretryCount: 0in the expected update patch.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/core/src/__tests__/resume-fallback.test.ts` around lines 284 - 287, Update the resume fallback test around the db.updateStep expectation to seed the running step with a non-zero retryCount and stale error/completionReason values, then explicitly expect retryCount: 0 alongside error: undefined and completionReason: undefined in the update patch. Keep the existing step identifier and other assertions unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@packages/core/src/__tests__/resume-fallback.test.ts`:
- Around line 284-287: Update the resume fallback test around the db.updateStep
expectation to seed the running step with a non-zero retryCount and stale
error/completionReason values, then explicitly expect retryCount: 0 alongside
error: undefined and completionReason: undefined in the update patch. Keep the
existing step identifier and other assertions unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e755d53c-e06d-4081-a78d-e2e70ee0a051
📒 Files selected for processing (6)
packages/core/src/__tests__/resume-fallback.test.tspackages/core/src/__tests__/run-persistence.test.tspackages/core/src/builder.tspackages/core/src/run.tspackages/core/src/runner.tspackages/core/src/types.ts
🚧 Files skipped from review as they are similar to previous changes (3)
- packages/core/src/builder.ts
- packages/core/src/run.ts
- packages/core/src/tests/run-persistence.test.ts
1 participant
Summary
Fixes three workflow-resume durability defects. Each was found by running the code, not reading it.
packages/core/src/run.ts—runWorkflow()constructedWorkflowRunnerwithout adb, soit fell back to
InMemoryWorkflowDb(runner.ts:703). Run state died with the process, and--resume— accepted atrun.ts:27, dispatched atrun.ts:86— could never succeed. It nowbuilds the cwd-backed
.agent-relay/workflow-runs.jsonlDB, matchingbuilder.ts:523.packages/core/src/builder.ts—WorkflowRunOptionshadstartFrom/previousRunIdbut noresume. The entrypoint that did persist state had no way to resume it, while the entrypointthat accepted
resumehad no persistence. Addsresumeand dispatches torunner.resume(...).packages/core/src/runner.ts—resume()reset onlyfailedsteps topending. A steppersisted as
runningwhen the process was killed stayedrunningforever and the run failed.A step cannot legitimately be
runningwhen no process owns it, so stalerunningis now resettoo.
Evidence
Reproduction, using a three-step deterministic workflow (no agents, so this isolates durability
from agent behaviour),
kill -9during step two:--resume <id>Run "…" not found (no database entry or cached step outputs)3 passed, 0 failedThe "after" column was verified against a locally patched build reproducing these three changes
before this branch existed; the branch itself is verified by the test results below.
Validation
Run on macOS 26.3.1 / Node 25.8.1.
npmhangs at 0% CPU on this machine, so the underlyingbinaries were invoked directly via
bunx. Worth noting for CI: the root scripts chain everythingthrough
npm run --workspace=…, so on any host where npm misbehaves the entire verification pathyields no signal at all.
bunx tsc --noEmit(packages/core) — clean, exit 0bunx tscfor github/slack/browser primitives — all exit 0bunx vitest run run-persistence.test.ts resume-fallback.test.ts— 10 passed / 10Pre-existing failures (not caused by this PR)
9 of those 10 are in
run-script.test.ts(TypeScript strip-types preflight, plus a 30s timeout).Baselined on
mainatb086a87: the same 9 fail there (9 failed | 17 passed). This PRneither causes nor fixes them.
The 10th was a defect in this PR's own first commit, fixed in
e2a2e73— see below.Note on the second commit
The first commit dropped the third argument from both
runner.resume()calls inbuilder.ts. Thatargument feeds
reconstructRunFromCache(runId, config), the cached-step-output fallback used whenworkflow-runs.jsonlis missing — precisely whatresume-fallback.test.tsexists to cover. Thetwo call sites were "harmonised" toward the broken one.
run-persistence.test.tsalso could not fail: its fixture usedsteps: [], whichvalidateWorkflow(runner.ts:3192) rejects, soparseYamlFilethrew before any assertion ran.e2a2e73restores the argument, gives the fixture a real step, and strengthens the assertion torequire the config argument so the regression cannot recur silently.
🤖 Generated with Claude Code