feat(daemon): pipeline v2 — multi-runner, process/passthrough, compound steps#165
Merged
claude-puntlabs merged 13 commits intomainfrom Apr 18, 2026
Merged
feat(daemon): pipeline v2 — multi-runner, process/passthrough, compound steps#165claude-puntlabs merged 13 commits intomainfrom
claude-puntlabs merged 13 commits intomainfrom
Conversation
… compound steps DES-030: multi-runner commands (claude + cli) DES-031: pipeline v2 data flow with process/passthrough modes, compound CLI steps, JSON pipe payload, output schema validation Three rounds of architect review, all findings addressed. Full design doc at docs/pipeline-v2-design.md.
Rewrite the intro to describe beadle as a programmable agent daemon with a Unix-like command model: commands are programs, the daemon is the shell, pipelines are pipes, GPG signatures are sudo. Includes a concrete 3-stage pipeline example showing claude + cli runners.
Update press release, solution, getting started, customer quote, competitor analysis, timeline, feature appendix, and risk assessment to reflect current state: Go binary (not Python), YAML commands (not Markdown), two runners (claude + cli), process/passthrough pipe modes, shipped pipeline v1, and pipeline v2 design in progress. Replace PyPI references with GitHub Releases. Bump to v3.0.
Replace ad-hoc mission dispatch with pipeline-first workflow. Pipeline selection table with 8 options evaluated by work nature. Execution loop: fresh agent per stage, mission carries context. Direct dispatch retained for rare single-mission work only.
Codifies the standard pipeline workflow: pipeline selection, 5-stage execution (design → implement → test → review → document), stage 4 multi-round review cycle, setup checklist, shipping protocol.
Beads mirror pipeline stages. Session task list via TaskCreate maps to bead IDs for user visibility. Epic bead closes when all stages close and PR merges.
Stage 1 output from standard pipeline. Exact struct changes, function signatures, migration steps, test cases for T1-T7. Commit order specified. Ready for mechanical implementation in stage 2.
…nd steps
T1: Command struct v2 with Runner/Mode/OutputSchema, runner-conditional
validator, YAML + test fixture migration (atomic)
T2: Runner interface, ClaudeRunner extracted from executor
T3: CLIRunner with BinaryWhitelist (EvalSymlinks), arg assembly, 1MB cap
T4: Compound CLI steps with goroutine-per-step io.Pipe chaining
T5: Process/passthrough pipe in executor, initial EmailMeta JSON
T6: Output schema validation via santhosh-tekuri/jsonschema/v6
T7: Auto-reply reads inputs.pipeline_output, fireElse uses fixed-text pipe
66 tests passing. go vet clean. Implements docs/pipeline-v2.md spec.
15 new tests: hybrid pipeline (claude+cli), passthrough data survival, schema validation rejection, text mode bypass, compound step mid-chain failure, whitelist deletion, pipe initial value, empty plan else handler, fireElse fixed-text pipe, unknown runner, invalid JSON, process mode pipe update, compound resolve failure, pipeline store integration, multiple consecutive passthrough stages.
H1: CLIRunner sets explicit minimal env (PATH+HOME+USER+declared),
no longer inherits daemon credentials to CLI subprocesses
H2: stderr buffers capped at 1MB via cappedWriter, prevents memory
exhaustion from malicious binaries
M2: escapeYAMLPipe for pipeline_output has no 500-char cap, prevents
silent truncation of large pipe values in mission contracts
L1: fireElse pipe parameter removed (was dead — function constructs
its own fixed-text error)
Stage 5: CHANGELOG entries for pipeline v2 (6 Added, 4 Security). DES-030 and DES-031 status PROPOSED → SETTLED. DES-030 YAML examples updated to v2 schema (mode, output_schema, no input/output fields).
There was a problem hiding this comment.
Pull request overview
Pipeline v2 upgrades the daemon’s pipeline orchestrator to support multiple execution backends (Claude + direct CLI), a persistent “pipe” data flow with process/passthrough semantics, compound CLI step chaining, and optional JSON Schema validation for process outputs—plus updated docs/ADRs and changelog.
Changes:
- Introduces
Runnerabstraction withClaudeRunnerandCLIRunner(binary whitelist, env isolation, stderr cap, compound steps). - Adds pipeline “pipe” semantics (process replaces pipe; passthrough preserves) and JSON Schema output validation (
output_schema). - Updates command YAML schema (runner/mode/output_schema/cli fields), tests, and project documentation/ADRs/FAQ/README.
Reviewed changes
Copilot reviewed 22 out of 23 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| prfaq.tex | Updates PR/FAQ (v3.0) and product narrative/press release text |
| internal/daemon/schema.go | Adds output schema compilation + output validation helpers |
| internal/daemon/schema_test.go | Unit tests for schema compilation/validation behavior |
| internal/daemon/runner.go | Adds Runner interface, ClaudeRunner, CLIRunner, whitelist + compound step execution |
| internal/daemon/runner_test.go | Extensive CLIRunner tests (whitelist, args, timeouts, caps, compound steps, env isolation) |
| internal/daemon/pipeline.go | Refactors executor to use runners, introduces pipe semantics + schema validation + reply wiring |
| internal/daemon/pipeline_test.go | Updates pipeline tests for runner dispatch + pipe behavior + reply args changes |
| internal/daemon/pipeline_edge_test.go | Adds edge/integration tests for hybrid pipelines, passthrough survival, schema rejection, etc. |
| internal/daemon/mission.go | Adds escapeYAMLPipe for untruncated pipeline output in mission contracts |
| internal/daemon/handler.go | Wires Executor construction to Runners map (currently only ClaudeRunner) |
| internal/daemon/command.go | Updates command schema: runner/mode/output_schema, CLI binary/steps, arg positions, validation |
| internal/daemon/command_test.go | Updates loader tests and fixtures for new command schema + CLI commands |
| go.mod | Adds jsonschema/v6 dependency (and adjusts uuid direct dep / x/text indirect) |
| go.sum | Updates sums for new/updated dependencies |
| docs/process.md | Adds/defines “Beadle Development Process” doc (pipeline-based workflow) |
| docs/pipeline-v2.md | Adds Pipeline v2 implementation design (task-level spec) |
| docs/pipeline-v2-design.md | Adds Pipeline v2 architecture/design document |
| README.md | Updates top-level product positioning + adds pipeline v2 design link |
| DESIGN.md | Adds/settles DES-030 and DES-031 ADRs for multi-runner + pipe semantics |
| CLAUDE.md | Updates delegation guidance to emphasize pipelines + selection rules |
| CHANGELOG.md | Documents Pipeline v2 features + security notes |
| .ethos/missions.jsonl | Records mission pipeline metadata for the epic execution history |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 60d51a1. Configure here.
F1: register CLI runner in handler.go with BinaryWhitelist
F2: auto-reply runner lookup with ok-check, skip on missing
F3: compound step Start failure cleanup — cancel, close pipes, Wait
F4: drain stdout to io.Discard after 1MB LimitReader (single binary)
F5: same stdout drain for compound steps
F6: wire buildStageContract through to ethos mission create — pipe
data and command-specific instructions now reach Claude workers
F7: cappedWriter always returns len(p) to prevent ErrShortWrite
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pipeline v2 adds CLI runners, process/passthrough data flow, compound CLI steps, and JSON schema validation to the beadle daemon pipeline orchestrator.
exec.Command. Binary whitelist withfilepath.EvalSymlinksat load + exec time. 1MB stdout cap viaio.LimitReader.process, default) or pass it through as a side-effect (passthrough). Side-effect commands don't destroy pipeline data.io.Pipe. First nonzero exit cancels all steps. No shell invoked.santhosh-tekuri/jsonschema/v6.output_schema: textbypasses validation.Also includes: pipeline v2 architecture doc (3 architect reviews), task-level implementation design, approved development process doc, updated PR/FAQ (v3.0), updated README, DES-030/031 ADRs settled, CHANGELOG.
Test plan
go vet ./...cleango test -race -count=1 ./internal/daemon/— 81 tests pass (66 existing + 15 edge-case)Epic: beadle-mvd | Pipeline: standard-2026-04-18-7c0858
Note
High Risk
High risk because it changes the daemon’s core pipeline execution model (new
runnerdispatch, pipe semantics, and JSON schema validation) and introduces direct subprocess execution/whitelisting logic plus new dependencies, which could affect correctness and security if misconfigured.Overview
Pipeline execution is refactored to Pipeline v2. Commands now declare
runner(claudeorcli),mode(processvspassthrough), andoutput_schema(eithertextor inline JSON Schema), with runner-conditional validation and CLI-specific fields likebinary,fixed_args,steps, and positional args.Executor behavior changes. The daemon now maintains a JSON “pipe” (initialized from email metadata), dispatches each stage through a
Runnerinterface, preserves the pipe forpassthroughstages, and validatesprocessoutputs against precompiled schemas (newjsonschema/v6dependency); reply handling is updated to sendinputs.pipeline_outputand else-path replies are fixed-text only.Docs and project metadata are updated. Adds/updates pipeline v2 design docs (
docs/pipeline-v2-design.md,docs/pipeline-v2.md), formalizes delegation/pipeline workflow inCLAUDE.mdanddocs/process.md, expands README/CHANGELOG, and records new DES entries (DES-030/031).Reviewed by Cursor Bugbot for commit 42a9a21. Bugbot is set up for automated code reviews on this repo. Configure here.