-
Notifications
You must be signed in to change notification settings - Fork 0
Engine Internals
What actually happens when a workflow runs. Read this when something surprises you, or before changing the engine.
Workflow and node statuses each have a declared transition table. Every mutation passes a guard; invalid transitions and terminal violations are typed errors (HTTP 409, not 500). Terminal states are irreversible, with one sanctioned exception: extend may reopen a workflow that completed naturally at a reporting checkpoint, because a reporting leaf means there is a parent awake and listening.
Every change is published as a durable dag.* event. A projector writes the SQLite read model inside the publish transaction — the read model cannot drift behind the log. History is event replay; there is no log table. A drift test compares the projector's guards with the declared tables and fails the build when they disagree.
The events are also the bus: the execution loop, the summary publisher, and the TUI all consume the same stream. Ephemeral summary events (progress counts) are deliberately excluded from the durable manifest — they are recomputed, not replayed.
Nodes are durably queued at admission. A child session is only created once a concurrency permit is held, so a large fan-out never materializes all its sessions at once. Waves are computed from the dependency graph; within a wave, nodes run in parallel. Queue wait counts toward a node's deadline.
The parent never polls. Wakeups are synthetic messages delivered when a report_to_parent node reaches a terminal state, or when the workflow itself terminalizes.
Every failed node carries one of: timeout (deadline exceeded — environmental, the task is not wrong), exec_failed (runtime/session failure — model, auth, connection, or crash-recovery loss), verdict_fail (the node ran but broke its output contract). The class decides the repair: replace with a longer deadline, fix config then replace, or restate the contract. Workflow-level failures attribute to specific node ids in the wake digest, so the parent repairs nodes rather than restarting graphs.
Recovery is lazy, per-workflow, and evidence-based. On restart, nodes left running are reconciled against their child session's durable state: sessions that finished back-fill their captured output; sessions that vanished pause the workflow and hand disposition to the parent (replan, resume, or cancel). Recovery never adopts or restarts provider work on its own — that decision belongs to a woke agent with context.
A replan that supersedes nodes marks them; the workflow row carries a graph_rev counter that bumps once per rewrite. Views filter superseded = false. The filter sits at the rebuild input — the single point where replaced nodes would otherwise re-enter the runtime graph as unsatisfied requirements and fail a healthy workflow (this was a real bug; the fix is pinned by probes). Terminal aggregation, status output, and the TUI all read through the same filtered seam. Completed outputs are never filtered: old-revision results remain resolvable, which is how a rewrite keeps the work it already paid for.
The location authority reads the directory stamp from the workflow row on every check, and writes stamps in exactly two places: creation (from the creating session's durable directory) and SessionMoved (re-stamped from the event payload, in the publish transaction). A NULL stamp matches no instance — fail-closed by design. Adoption and spawn re-admission additionally hold a conditional claim (a single UPDATE that succeeds only while the row exists and is non-terminal), which fences adoption against a concurrent deletion.