Skip to content

research(pipeline): extract ChildRunner from MetaPipelineExecutor + composition primitives #1563

@nextlevelshit

Description

@nextlevelshit

Background

Issue #1506 closed with two deferred items. ChildRunner was one:

ChildRunner extraction (current state already serves; deferred)

Both MetaPipelineExecutor and the composition primitives (iterate / aggregate / branch / loop / sub_pipeline) launch child pipelines today. The launch logic is duplicated:

  • internal/pipeline/meta.goMetaPipelineExecutor constructs and runs generated child pipelines, owns depth/step/token budgets, parent ID tracking.
  • internal/pipeline/composition.go + executor_composition.go — composition primitives launch named or inline sub-pipelines via runNamedSubPipeline / compositionLaunchInfo.

Proposal: ChildRunner

Extract a single internal/pipeline/childrunner.go (or sibling package) that owns:

  • Child pipeline construction from spec or YAML
  • Parent-child run linkage (run_id, parent_run_id, run_kind)
  • Cross-pipeline artifact propagation (already wired in feat(pipeline): propagate parent artifacts to sub-pipeline children #1557)
  • Budget enforcement (depth, total steps, total tokens) — currently meta-only
  • Event emitter wiring with parent context
  • Workspace allocation for child runs

Research scope (do NOT implement until decided)

  1. Inventory: find every site that launches a child pipeline. Likely:
    • MetaPipelineExecutor.executeStep family
    • runNamedSubPipeline
    • executeIterateInDAG / executeBranchInDAG / executeLoopInDAG / executeAggregateInDAG
  2. For each site, document: what context it threads, what budgets it enforces (or doesn't), how it tracks parent linkage, how it surfaces results.
  3. Identify the common kernel — the smallest set of fields + behaviour every launch site needs.
  4. Identify divergent fields — what genuinely differs (meta has token budgets; composition does not — should it?)
  5. Decide: is the kernel large enough to justify extraction, or is the duplication actually parallel-but-divergent and extraction would couple unrelated concerns?

Out of scope

  • Step-level Meta block (`step.Meta:`) — separate feature, needs own design (non-determinism, resume, validation concerns).
  • Forcing MetaPipelineExecutor through StrategyExecutor — already evaluated as wrong abstraction (free-form vs step-level signature mismatch).

Acceptance for research

  • A short ADR-style document under docs/adr/ listing every child-launch site, the common kernel, and a recommendation: extract / partially extract / leave alone.
  • Concrete cost estimate for extraction (LOC, files touched, test impact).
  • Concrete benefit estimate (duplication removed, future-feature unblocking).
  • Decision: GO / NO-GO. Tag follow-up implementation issue if GO.

Why research-first

Memory rule: "no premature abstractions; three similar lines is better than premature abstraction." Two launch surfaces is borderline — research determines whether the third use case (or visible drift between meta and composition behaviour) makes extraction load-bearing.

Related

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions