Investigate deterministic orchestration for intra-stage multi-agent pipelines

[maruiz93]

What happens

ADR 0018 decided that scripted pipelines — not LLM coordinators — should handle multi-agent orchestration. The pipeline definition format was deferred: "Whether the pipeline is defined using existing infrastructure capabilities or a custom pipeline layer is a separate decision."

The review agent (#1550) currently orchestrates six sub-agents using the Claude Code Agent tool, with the LLM deciding dispatch. The pr-review SKILL.md acknowledges this as a "temporary exception to ADR-0018, pending ADR-0018 amendment." As stages gain more agents (ADR 0020) and new sub-agents are proposed (#1797), the gap between the accepted architecture and the implementation grows.

What should happen

Investigate options for a deterministic pipeline format for intra-stage orchestration, starting with the review agent as the concrete case. Options to evaluate:

• Claude Code Workflows — provides agent(), parallel(), pipeline(), phase(), and structured output schemas. Does it fit the review agent's orchestration needs? What are the rough edges? OpenCode does not have an equivalent deterministic workflow tool, so this creates a runtime-specific dependency — with feat(runtime): abstract Claude Code behind pluggable agent runtime #1780 abstracting the agent runtime, the portability story needs consideration. See official docs and Anthropic blog post.
• Custom pipeline layer — a fullsend-native format that could work across runtimes. What would it look like? Is the added maintenance justified?
• Keep current approach — continue with LLM-driven orchestration via the Agent tool. Is the ADR 0018 exception acceptable long-term, or does reliability degrade as sub-agent count grows?

Context

• ADR 0018 — scripted pipeline for multi-agent orchestration (accepted, format deferred)
• ADR 0020 — composable single-responsibility agents with individual sandboxes
• ADR 0026 — stage-based dispatch for agent workflow decoupling
• feat(review): parallel specialized sub-agents for PR review #1550 — parallel specialized sub-agents for PR review (current LLM orchestration)
• feat(runtime): abstract Claude Code behind pluggable agent runtime #1780 — abstract Claude Code behind pluggable agent runtime
• feat(review agent): challenger pass should use a dedicated sub-agent, not the orchestrator #1797 — extract challenger pass into dedicated sub-agent

[fullsend-ai-triage]

Triage Summary

Category: Feature (research/investigation) · Severity: Medium

Problem

ADR 0018 mandates scripted pipelines for multi-agent orchestration, but the review agent currently uses LLM-driven dispatch via the Agent tool — an acknowledged temporary exception. As sub-agent count grows (ADR 0020, new proposals like #1797), this architectural drift increases.

Assessment

This issue is well-scoped with clear options to evaluate and strong architectural context. The three options (Claude Code Workflows, custom pipeline layer, status quo) are concrete and evaluable.

Key considerations for the investigation:

• Claude Code Workflows offer agent(), parallel(), pipeline() primitives that may fit, but create a runtime-specific dependency — relevant given feat(runtime): abstract Claude Code behind pluggable agent runtime #1780's runtime abstraction work
• A custom pipeline layer offers portability but adds maintenance burden
• The status quo may be acceptable if sub-agent count stays bounded, but reliability concerns grow with scale
• Evaluate TEP-0170: Agent-Native Workflows for Tekton #661 (TEP-0170 evaluation) and Research: Evaluate OpenAI Symphony against fullsend's architecture #482 (Symphony evaluation) are related landscape research that may inform this decision

Recommended Approach

Prototype the top candidate against the review agent's current orchestration (6 parallel sub-agents with structured output aggregation). Evaluate against: determinism, cross-runtime portability, maintainability, error handling, and observability. Produce an ADR amendment or new problem doc.

Proposed Acceptance Criteria

1. Chosen format can express the review agent's full orchestration graph
   (6 parallel sub-agents with structured output aggregation)
2. Adding a new sub-agent requires only configuration changes,
   not LLM prompt modifications
3. Portability implications are documented relative to #1780's
   runtime abstraction goals
4. ADR 0018 is amended or a new ADR is produced recording the decision

---

Labels: Research issue investigating orchestration options for the review agent, touching agent harness/runner infrastructure.

[fullsend-ai-triage]

Meeting update — 2026-06-03

Relevant to this issue: The team discussed non-deterministic behavior and race conditions in existing workflows, which directly relates to the need for deterministic orchestration in multi-agent pipelines.

• Decision: further architectural optimization discussions are tabled until the synchronous workflow dispatch implementation (ADR 0041) is complete
• The rationale is that the sync dispatch model will change the concurrency semantics fundamentally, making it premature to optimize the current async model
• Race conditions in the bug reporting system were specifically called out as a symptom

Related PRs: PR #1611 (synchronous workflow_call dispatch, ADR 41 implementation) is the prerequisite work that must land first

Next steps: Revisit deterministic orchestration once ADR 0041 sync dispatch is merged and stable

Meeting notes

[fullsend-ai-triage]

Meeting update — 2026-06-08

Relevant to this issue: The team explored cloud workflows and internal abstraction layers as alternatives to dynamic agent orchestration, directly relevant to determining the pipeline definition format for deterministic orchestration.

• The decision on whether to adopt cloud workflows or build a custom orchestration abstraction layer was deferred pending local experimentation
• The group emphasized moving toward open code to improve determinism and security, aligning with ADR 0018's scripted-pipeline direction
• Trade-off under evaluation: leveraging existing cloud workflow infrastructure vs. building a purpose-built abstraction layer with tighter control

Related PRs: #1611 (synchronous workflow_call event dispatch)

Next steps: Team members will experiment with cloud workflow features during day-to-day operations to assess utility and inform the final decision.

Meeting notes

[fullsend-ai-prioritize]

RICE Priority Score: 0.6

Score breakdown

Dimension	Score	Reasoning
Reach	1.5	This affects the review agent (used across all PRs) and sets the architectural pattern for all future multi-agent stages (ADR 0020). As sub-agent count grows (#1797 and beyond), every user of the platform is affected by orchestration reliability. Scored 1.5 for multiple strategic customers impacted through the review agent workflow.
Impact	1	The current LLM-driven orchestration works but introduces non-determinism and race conditions (noted in meeting updates). Moving to deterministic pipelines improves reliability and makes adding new sub-agents a config change rather than prompt modification. However, the current approach is functional — this is a noticeable improvement, not a blocker. Scored 1 for medium impact.
Confidence	0.6	The issue is well-described with clear options and strong architectural context (ADRs 0018, 0020, 0026). However, key dependencies are unresolved — ADR 0041 sync dispatch must land first, and the team has explicitly deferred decisions pending experimentation. The research nature adds uncertainty about final outcomes. Scored 0.6 for medium confidence.
Effort	1.5	This is a research issue requiring evaluation of three options, prototyping against the review agent's 6-sub-agent orchestration graph, assessing cross-runtime portability (#1780), and producing an ADR amendment. Multiple components and design decisions are involved, but the scope is bounded to investigation rather than full implementation. Scored 1.5 for moderate effort.

Formula: (1.5 x 1 x 0.6) / 1.5 = 0.6

[fullsend-ai-triage]

Meeting update — 2026-06-22

Relevant to this issue: The team debated separating agent permissions versus utilizing unified architectures to manage security in multi-agent workflows — directly relevant to this issue's investigation of deterministic orchestration patterns.

• No consensus reached on whether to use separate agents, GitHub workflows, or a single-agent approach for managing security boundaries
• Research proposed into using issue attachments (GitHub/Jira) for inter-agent data management to avoid complex agent chaining
• Trade-offs discussed between permission isolation (separate agents per stage) and operational simplicity (unified agent with sub-agents)
• This remains an open architectural question requiring further discussion

Unresolved: The group requires further discussion to align on the architectural strategy for managing multi-agent workflows and security permissions.

Next steps: Draft and propose a solution architecture for handling custom agent workflows and concurrency. Assess dynamic workflow features for managing sequences and evaluate potential impacts of tool coupling. Test whether storing agent data as issue attachments eliminates the need for agent chaining.

Meeting notes