Workflow graph integrity: a workflow executes exactly the graph it declares

[yellowman]

Reference validation, tranche 1. Scope is graph-internal integrity and the runtime that executes it — not external reference resolution, which is tranche 2.

The invariant

A workflow executes exactly the graph it declares. Every node id is unique, every explicitly named entrypoint resolves, and every graph edge resolves to a declared node. Invalid persisted graphs fail closed at execution rather than being repaired or truncated silently.

Every defect below is a violation reported as success. The engine did not crash on any of them; it ran a different workflow from the published one and said it worked.

What was wrong, measured

Each row was reproduced by running the engine before anything changed.

declared	what actually executed
entrypoint: "nowhere"	next(iter(node_map)) — whatever node came first
next: "nowhere"	if not node: continue; the continuation vanished
two nodes sharing an id	the node_map comprehension kept the last
on_error: recover, breaker open	tool -> normal, the success edge, status: "ok"
end with next: "side"	nothing; end stops the run
switch with next	only branches[].next
tool_call with after	only next / on_error
parallel with on_error	only next / after
type: "swich"	admitted with 201, then invoked the model as a tool call
parallel -> switch -> side	fan -> choose -> join; side never ran
parallel -> end -> after	the node named end ended nothing
101-node chain	100 nodes ran, success, content "No response generated."
next: ["leaf"] * 150, 3 nodes	150 concurrent tool invocations against a budget of 16, success

Streaming was a second copy of all of it: its own entrypoint fallback, its own if not node: continue, and — because it streams three tools without calling the blocking executor — no circuit-breaker preflight and no on_error handoff at all.

What changed

liminallm/service/workflow_graph.py (new) — one pure validator, returning every problem rather than the first, so each caller raises in its own vocabulary.

The edge fields were measured from the executor, not read off the kind schema. The executor consumes five; after and on_error are not in that schema at all, so a validator written from it would have covered three of five and looked complete. Cardinality was measured the same way: next is the only field read as either a string or a list.

_NODE_EDGES is a per-node-type table, because which fields a node reads is decided by its type, and the field set checked against it is derived from the table — adding a field to one type asks every other type whether it reads it.

_parallel_child_problems covers a third dimension: not what a node reads, but how it was reached. _execute_parallel_nodes discards a child's successor list and reads only "error" out of its status, so a parallel child is a leaf tool_call — SPEC §9.1's "fan-out to multiple nodes, then join", with termination on the after continuation where the ordinary loop can see it.

Two validation altitudes. Admission stops new invalid graphs; the engine checks again before building node_map, because a row can predate the check or arrive by import. The node type is an enum in the kind schema and a semantic check, for that reason. SPEC §9.1 already wrote it as an enum.

One tool-node control plane. _circuit_open_result (the breaker preflight) and _successors (the next-versus-on_error chooser) are shared by the blocking executor, its circuit-open branch, and streaming. Token production stays streaming-specific — that is why the path exists. A streamed failure takes on_error only if it failed before producing a token: once a token has reached the client, recovery would append a second answer to the same bubble, which is the boundary _stream_agent_files_node already keeps.

ExecutionBudget — one object per run, in workflow_limits with the other shared limits, held by the driving loop and the fan-out it dispatches. The reservation sits beside the gather it bounds and is taken before the tasks are built, so an over-budget batch never begins any of it. Each entry in parallel.next costs one, a repeated id included. Both exhaustion mechanisms now fail closed and name which budget ran out.

Evidence

41 mutations, 41 killed, no survivors. Three were retired rather than left surviving, each because it changed no behaviour — a mutation with no effect says the code it adds is dead, not that the tests are weak.

Several rules earned a complementary pair, because losing a rule and over-applying it are both wrong and only one fails loudly:

• extending the parallel-child rule to the after target kills the control that says the rule is about the context, not the node;
• marking every streamed node as having emitted a token kills the zero-token recovery witnesses, which "never recover" would otherwise satisfy;
• reserving one execution for a whole fan-out kills the same four witnesses as making children free — it would satisfy a naive "the budget is consulted" check while leaving fan-out unbounded.

Sixteen anchors went stale across seven passes and the driver reported each as unmeasured rather than as a survivor.

Mutations found three of my own bad witnesses, which is the honest part of this record:

• a control that ended with pending already empty, so it never exercised the case it was named for;
• a cycle fixture with two nodes that was witnessing the step budget, not the visit guard — a two-node cycle reaches its eighth visit at step 15 and the step budget stops it at 14;
• a positive control whose parallel fanned into an end, which under the new rule would have passed on the child rule rather than the field it was named for.

The permanent VALID fixture was the same shape: its parallel children each declared next: "join" while the parallel declared after: "join", so it looked like it exercised a child's next when after was doing all the work.

One witness pins a premise rather than a rule: it runs the refused parallel-child graph with validation disabled and asserts the successor never executes. If parallel ever becomes a recursive subgraph executor, that test fails and says the rule needs revisiting.

Full lane: 3092 passed, 27 skipped. CI's lint selection clean.

Fixture corrections

The node-type enum found 15 existing fixtures declaring llm_call or respond — node types this engine has never executed, in tests that never run the graph. Corrected to a real tool_call rather than loosening the rule.

Not in this PR

Three findings are recorded in docs/ISSUES.md and deliberately left for their own tranches: the PoolClosed teardown race in the xdist lane, streamed failures never incrementing the circuit breaker (SPEC §18 accounting, not §9 graph fidelity), and a refused fan-out leaving its parent out of workflow_trace.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DQtPsg9YSUXaStGXyUjozA

---

Generated by Claude Code

---

Note

Overview
Workflows are no longer silently repaired or truncated at runtime. A new graph_problems validator (used at artifact admission and again before run / run_streaming) rejects dangling entrypoints/edges, duplicate or empty ids, wrong edge cardinality, node types and edges a type never reads, and invalid parallel children—matching what the executor actually consumes, including after and on_error fields absent from JSON Schema.

The workflow kind schema is tightened (four node type values, minLength on ids, scalar branches[].next). Blocking and streaming now share _circuit_open_result and _successors, so open breakers and tool failures follow on_error instead of the success path; streaming only recovers on on_error when no token was already emitted.

ExecutionBudget charges parallel fan-out before gather, and step/revisit exhaustion returns status: error (or a streaming error event) instead of a placeholder success. Tests and fixtures switch obsolete node types to real tool_call graphs; docs/ISSUES.md records the campaign and follow-ups left out of this tranche.

^{Reviewed by Cursor Bugbot for commit 186616b. Bugbot is set up for automated code reviews on this repo. Configure here.}