fix: resolve mypy error and add DataNode source_path test coverage - #15
Draft
colehurwitz wants to merge 68 commits into
Draft
colehurwitz wants to merge 68 commits into
colehurwitz wants to merge 68 commits into
Conversation
Reconcile with main's removal of dead workflow modes (#1346/#1376). Main deleted the mode definitions, tests, and routing entries; this change keeps the complementary piece: graceful migration of legacy mode names to design via the DEAD_MODES map, auto-routing of all project states to design, and the deprecation/migration test coverage. Squashed from 25 commits on factory/run-bf11e015.
) * feat: package ecosystem prototype — composable workflow subgraphs Introduces the Package primitive and four composition operators (Sequential, Parallel, Conditional, Loop) for building workflows from reusable, typed subgraph units. Key abstractions: - Package: wraps a Workflow subgraph with typed Ports (data plane), StateContract (control plane), OptKnobs (optimization surface), and MemoryDeclarations (pluggable persistence) - Sequential: chains packages, wiring exit to entry - Parallel: forks into N packages via ForkNode/JoinNode - Conditional: routes to branches via GateNode verdicts - Loop: re-enters body on RELOOP verdict - compile(): lowers compositions to flat mutable Workflow IR Includes design doc (docs/design/package-ecosystem.md) covering vision, system architecture (8 components), composition model, three-representation model (author/optimize/distribute), and migration path. 21 tests demonstrate wrapping existing subgraph patterns (study, deep-QA) as Packages and composing them into larger workflows. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add interactive demo for package ecosystem prototype 7-section walkthrough: standard library, composition, design mode before/after, optimization surface, parallel/conditional operators, nested composition with loops, and the compile/serialize round-trip. Run: uv run python examples/package_ecosystem_demo.py Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: end-to-end package ecosystem demo Demonstrates the full composability story: 1. Two teams build packages independently (security audit, perf optimizer) 2. A third team composes them three ways (shift-left, parallel, loop) 3. All three compile to valid executable graphs 4. Optimizer searches over compositions and knob configurations 5. Winner is serialized and round-tripped as a distributable package Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: real optimization demo using factory's outer loop Runs factory's actual mutation operators (mutate_params, mutate_prompt) and structural analysis (compute_features, structural_hash) over composed Package workflows. 8 generations, 32 mutations, tournament selection, deduplication. No faked scores. Also fixes the composition demo to include strategy package and produce valid graphs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: real optimization demo using factory's outer loop 5 candidates (3 topologies + 2 mutations) ran through factory's real WorkflowExecutor with live Claude agents against a task queue project. Results show topology and knob choices producing measurably different outcomes (0.875-0.955 score range). Frozen nodes protect package internals while mutations tune agent parameters. Baseline: 5 tests, 123 lines, score 0.417 Winner: 18 tests, 285 lines, score 0.955 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: real optimization demo using factory's outer loop Iterative optimization over 4 generations with real Claude agents. Score trajectory: 0.991 → 0.992 → 0.998 → 0.998 → 1.000 Fix: validate_and_repair now follows ForkNode.targets and JoinNode.sources when computing reachability, preventing fork/join branch nodes from being pruned as "unreachable." 9 evaluations, 50 minutes. Each generation mutates the best from the previous generation (model, prompt, timeout), runs through the real WorkflowExecutor, scores on real pytest results, and selects. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove superseded demo scripts Keep only package_e2e_optimize.py (the real iterative optimization). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: lint errors in e2e demo Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: compute_features includes OptKnob values for MAP-Elites diversity (#1388) Package.compile() now propagates knob defaults to Workflow.knob_values. compute_features() hashes knob values into additional feature dimensions, giving the MAP-Elites archive diversity pressure across the optimization surface — not just graph topology. Without this, all knob-only mutations (prompt styles, thresholds, boolean flags) map to the same archive cell, collapsing the grid to a single-best selector and eliminating quality-diversity search. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: chess evolution demo — Opus-driven pipeline optimization with live UI End-to-end demo of the Package ecosystem optimizing an LLM chess pipeline. Haiku plays chess through a composable pipeline (Sequential, Parallel, Conditional, Loop), while Opus drives experiment selection and invents new prompt variants at runtime. Key capabilities demonstrated: - All 4 Package composition primitives in one pipeline - MAP-Elites archive with knob-aware features for diversity - Opus reflects on full history, then proposes experiments - Auto-expanding OptKnobs: Opus invents new prompt variants that get registered and available to future generations - Composite scoring (avg_eval - 20*blunders + 8*moves) - Live web UI with game replay, eval charts, best-game tracking - Recording/replay system for demos Results: Opus-invented prompt "capture_refutation_gate" became the top-scoring tactical style across 700+ experiments, outperforming all hand-authored variants. ASCII board representation improved median score by +28 points over FEN-only. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add expandable OptKnob for runtime surface expansion OptKnob gains two fields: - expandable: bool — when True, the outer loop may propose values beyond the initial bounds (e.g. new prompt variants, extended ranges) - expansion_hint: str — tells the optimizer how to generate new values Validated in the chess evolution demo: Opus invented 8 new prompt variants at runtime, with "capture_refutation_gate" becoming the top-scoring tactical style across 700+ experiments — outperforming all hand-authored variants. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: KNOB_MUTATE operator for OptKnob-aware evolution Adds a new mutation operator that mutates knob_values on compiled Workflows. When a Package declares OptKnobs with bounds, KNOB_MUTATE picks a random knob and changes it to a different value from its bounds. Changes: - MutationType.KNOB_MUTATE added to the enum - mutate_knob() picks from knob_bounds carried through compile() - Workflow.knob_bounds propagated by Package.compile() alongside knob_values - Default weight 0.25 (highest single operator) — knob tuning is the most common optimization when the graph topology is already good - WeightedRandomStrategy handles KNOB in guided operator selection This closes the gap where the outer loop could only mutate graph topology (nodes, edges) but not the optimization surface declared by OptKnobs. Partially addresses #1389. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: expandable knobs in KNOB_MUTATE via expander callback When a knob is marked expandable and all its bounds are exhausted, KNOB_MUTATE calls an expander callback to generate a new value: expander(knob_name, expansion_hint, current_value, bounds) -> new_value The expander can be backed by any LLM — Opus authoring a new prompt variant, or a simpler model extrapolating a threshold range. New values are automatically added to knob_bounds for future mutations. Workflow now carries knob_expandable (name -> expansion_hint) through Package.compile(), and apply_random_mutation accepts a knob_expander kwarg to thread the callback through. This completes the OptKnob lifecycle: declare -> compile -> mutate -> expand. The chess demo's Opus prompt invention is now expressible entirely through factory's mutation operators. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: default_knob_expander — factory invents new knob values via CLI KNOB_MUTATE now ships with a default expander that calls `claude` CLI (Opus) to invent new values when expandable knobs exhaust their bounds. No custom code needed — the caller gets prompt invention for free: apply_random_mutation(wf, strategy, gen) # KNOB_MUTATE auto-expands when needed The expander is a simple subprocess call — synchronous, no async dependency, works in any context. Returns a short name for prompt knobs or a number for threshold knobs. This eliminates the last piece of custom optimization code from the chess demo — the main loop is now pure factory calls (OuterLoopReflector, apply_random_mutation, MAPElitesArchive). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: _deep_copy_workflow preserves knob_values through graph mutations Graph mutations (NODE_INSERT, NODE_REMOVE, EDGE_REDIRECT, etc.) were dropping knob_values, knob_bounds, and knob_expandable because _deep_copy_workflow didn't copy them. This meant KNOB_MUTATE always returned None after any graph mutation, silently disabling knob optimization. Adds tests for: - KNOB_MUTATE within bounds - KNOB_MUTATE with expander callback for expandable knobs - Knob preservation through insert_node and remove_node Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: lint errors in chess demo and mutation tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: mypy error in KNOB_MUTATE expander passthrough Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address 6 bugs from Factory QA review B3 (Critical): to_dict()/from_dict() now serialize knob_values, knob_bounds, knob_expandable — no more silent data loss on checkpoint/restore. B4 (High): Parallel() now detects node ID collisions via _merge_graphs(), matching Sequential's behavior. B5 (High): Loop() now adds a PROCEED exit edge from gate to a new FnNode, giving loops a termination path. B6 (High): compute_features() uses hashlib.sha256 instead of hash() for deterministic knob hashing across processes. B7 (Medium): _try_mutation mutable_nodes guard now exempts KNOB_MUTATE alongside NODE_INSERT. B8 (Medium): Conditional() raises ValueError for unknown branch labels instead of creating unconditional edges. Adds 11 new tests covering all fixes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extends the outer loop to optimize OptKnob values alongside graph topology, with reflection-guided exploitation. KNOB_MUTATE operator: - Mutates knob_values within declared bounds (25% default weight) - When bounds exhausted + knob is expandable, default_knob_expander (Opus via CLI) invents new values at runtime - Guided by reflector: 70% picks the winning knob+value from contrastive analysis, 30% explores randomly Knob-contrastive reflection (OuterLoopReflector): - New knob_values_by_id param on reflect() - Per-knob gradient: avg score per value across all individuals - Top-K vs bottom-K knob contrast: which values consistently win - Produces "KNOB_MUTATE: name=value outperforms ..." suggestions Infrastructure: - _deep_copy_workflow preserves knob fields through graph mutations - _try_mutation exempts KNOB_MUTATE from mutable_nodes guard - reflection_report threaded through apply_random_mutation to mutate_knob for guided selection Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sequential, Parallel, and Loop all collected knobs and memory from their child packages. Conditional was missing these, silently dropping OptKnobs and MemoryDeclarations from any package inside a Conditional branch. Adds 2 tests: knob/memory propagation, compile round-trip. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
) * feat: add design-v2 workflow — inference-time scaling for design mode Adds the design-v2 workflow as a contributed workflow with dynamic research/strategy/QA directors that scale agent count at inference time based on project complexity, replacing the static fork/join patterns. - Research Director: decides N research directions dynamically (3-7) - Strategy Director: spawns M strategy perspectives (2-5) - QA Director: creates K tailored adversarial test approaches (2-5) - User Intent Ledger: tracks idea + feedback throughout the session - Design Doc: rewrites strategy into human-readable design document - Synthesize QA: merges adversarial reports with confidence scoring 29 nodes, 32 edges. 63 tests covering graph structure, node properties, edge wiring, post checks, and removed-node assertions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: extract design-v2 prompts to separate module, fix unused import - Extract all 7 prompt template constants from workflow.py into prompts.py - Remove unused `glob` import from synthesize_qa inline Python (Path.glob used instead) - Add comment on ADVERSARIAL_PROMPT noting it's used by QA Director when spawning testers Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: recognize design-v2 (and contributed workflows) in CLI mode routing The CLI mode validation only checked CEO_MODES and project-local workflows, ignoring builtin registry entries like design-v2. Two fixes: 1. Add "design-v2" to CEO_MODES in _helpers.py 2. Change fallback validation from project-only to all workflow registry entries, so any registered workflow (builtin, contributed, project) is automatically valid 3. Add "design-v2" alongside "design" in all mode routing checks so it supports --focus, --auto-approve, --from-plan, --just-plan Closes #1392 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove design-v2 from CEO_MODES — discovered via WorkflowRegistry design-v2 is a builtin workflow discovered by WorkflowRegistry.discover() and passes the fallback check in _ceo_helpers.py without needing an explicit CEO_MODES entry. The hardcoded entry was redundant. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: wire auto_approve through to CEO task string for design/design-v2 modes When --auto-approve is passed, the CEO now receives instructions to act as the user at approval gates — reviewing plans against user-intent.md and making PROCEED/feedback decisions instead of waiting for human input. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address 5 code review issues on design-v2 workflow (#1392) - Add design-v2 to mode checks in run.py (auto_approve, focus, skip_improve) - Fix init_user_intent FOCUS env var: read from FACTORY_IDEA env, backlog.md fallback, skip if user-intent.md already exists - Fix single-quote injection in init_user_intent and synthesize_qa by passing project_path via sys.argv instead of string interpolation - Update error messages in _ceo_helpers.py to mention design-v2 - Remove dead ADVERSARIAL_PROMPT constant, fold output format into QA_DIRECTOR_PROMPT Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: rewrite init_user_intent FnNode to avoid SyntaxError in python3 -c The command used compound if/else blocks after semicolons in a python3 -c one-liner, which Python does not allow. Rewritten using ternary expressions and or-chains: early exit via (sys.exit(0) if cond else None), idea fallback via env or backlog or default. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
) * Revert "feat: design-v2 workflow — inference-time scaling for design mode (#1392)" This reverts commit 2275f57. * feat: design-v2 workflow — inference-time scaling for design mode Replaces #1392 (reverted + re-landed with all fixes). Design-v2 applies inference-time scaling (parallel generation → synthesis → gate) to every stage of the design pipeline: 1. Research Director — CEO designs N research directions dynamically (3-7 based on complexity), spawns N researchers with tailored prompts 2. Strategy Director — CEO designs M strategy perspectives dynamically (2-5), spawns M strategists with tailored prompts 3. Design Doc — rewrites strategy as human-readable design document 4. QA Director — CEO designs K adversarial testing approaches (2-5), spawns K testers + 1 code reviewer in parallel 5. User Intent Ledger — append-only file tracking all user input, read by QA as ground truth for acceptance criteria 6. Structured QA Gate — checks user-intent.md as primary criteria Consistent Director pattern across all 3 stages. 29 nodes, 32 edges. 63 tests. CLI routing supports --auto-approve (CEO acts as user at approval gates) and --focus. Fixes from external code review: - run.py mode checks updated for design-v2 - init_user_intent uses sys.argv (no single-quote injection) - init_user_intent reads from backlog fallback (no missing env vars) - Error messages mention design-v2 - Dead ADVERSARIAL_PROMPT removed - init_user_intent SyntaxError fixed (no compound if in python3 -c) - Code review runs in parallel as K+1th agent Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- README: add design-v2 section under Design Mode with usage examples and comparison table (design vs design-v2) - README: add design-v2 to Built-in Workflows table - CLAUDE.md: add design-v2 CLI examples to Running the factory section Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…#1400) Design-v2 applies inference-time scaling (parallel generation → synthesis → gate) to every stage of the design pipeline. Three Director agents dynamically design N/M/K approaches with project-specific prompts. 29 nodes, 32 edges. 63 tests. All review fixes included. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
#1401) The prompts.py landed on main was missing the last two fixes: - Mandatory code review as part of QA Director's Phase 2 - All K+1 agents (K adversarial + 1 code reviewer) run in parallel Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three enforcement points prevent the strategy from drifting away from what the user asked for: 1. Strategy Director — extracts intent_items from user-intent.md, verifies every ask is covered by at least one perspective before spawning strategists. Each strategist prompt includes the relevant user asks. 2. Synthesize Strategy — 3-step process: extract intent list, synthesize plan, then audit coverage. Writes an Intent Coverage table mapping every user ask to where it appears in the plan. Fails if any ask is missing. 3. Gate Strategy — mandatory intent fidelity check before approval. Verifies the Intent Coverage table, rejects plans that drop or reinterpret user asks. RELOOP goes back to strategy_director. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add intent-fidelity checks to design-v2 strategy pipeline Three enforcement points prevent the strategy from drifting away from what the user asked for: 1. Strategy Director — extracts intent_items from user-intent.md, verifies every ask is covered by at least one perspective before spawning strategists. Each strategist prompt includes the relevant user asks. 2. Synthesize Strategy — 3-step process: extract intent list, synthesize plan, then audit coverage. Writes an Intent Coverage table mapping every user ask to where it appears in the plan. Fails if any ask is missing. 3. Gate Strategy — mandatory intent fidelity check before approval. Verifies the Intent Coverage table, rejects plans that drop or reinterpret user asks. RELOOP goes back to strategy_director. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add Overwatch agent to design-v2 workflow Add a final verification step after QA passes that checks completeness and honesty of the entire pipeline output before the PR reaches the user. - overwatch (AgentNode, CEO, timeout=1800): reads user intent, strategy, builder output, and all QA reports; performs intent checklist, evidence audit, and mandatory spot checks; writes overwatch-latest.md - gate_overwatch (GateNode, agent, CEO): proceeds to doc freshness if Overwatch passes, reloops to builder with specific findings if not - Edge chain: gate_qa [PROCEED] → overwatch → gate_overwatch → gate_doc_freshness (replaces direct gate_qa → gate_doc_freshness) - Graph: 29 → 31 nodes, 32 → 35 edges, validates clean Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…config (#1391) Closes #1391 - factory/state.py: Auto-bootstrap .factory/config.json from tracked factory.md when config is missing (no discovery, reparse only) - factory/worktree.py: Add _finalize_greenfield() to commit untracked factory.md on greenfield branches before worktree cleanup; retain worktree with recovery instructions on finalization failure - Tests for all new paths: auto-bootstrap, malformed factory.md, greenfield detection, finalization commit, worktree retention
* fix: 7 code review findings in design-v2 workflow
- Fix IndexError crash in init_user_intent when backlog.md is whitespace-only
- Add design-v2 specific task instructions instead of inheriting P0-P3 from design mode
- Include design-v2 in banner_mode check so it shows correctly in CLI output
- Increase dedup key truncation in synthesize_qa from 80 to 200 chars
- Fix two E501 violations in _ceo_helpers.py (108 chars → multi-line)
- Add comment noting bootstrap nodes mirror definitions.design_workflow()
- Extract 45-line inline synthesize_qa script to qa_synthesis.py module
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: 8 code review findings from design-v2 round 2
- Thread auto_approve through run.py's _run_single_cycle to _build_ceo_task
- Fix new-idea design-v2 getting wrong P0-P3 instructions (check mode, not shown_mode)
- Add comment noting naive string dedup limitation in qa_synthesis
- Fix error messages referencing "--mode design" to say "--mode design/design-v2"
- Extract DESIGN_MODES constant to replace 14 scattered tuple checks
- Add TODO for bootstrap copy-paste between design and design-v2
- Show available modes in unknown mode error instead of filesystem path
- Guard qa_synthesis.main() against missing argv with usage message
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: mandatory run-the-code testers in QA Director
Two hardcoded adversarial testers that ALWAYS run, non-negotiable:
1. smoke-run — actually execute the built application (CLI, server, lib)
with example inputs. Show commands + output. No reading code, no
checking tests — run it.
2. user-scenario — read user-intent.md, then use the app exactly as the
user described. The user's words become the test script.
These run alongside the K designed testers + code reviewer, all in
parallel. Total agents = K + 2 (hardcoded) + 1 (code review).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: 3 code quality findings in design-v2 qa_synthesis and workflow
- qa_synthesis: filter out positive/neutral bullet lines (PASS, verified,
etc.) — only lines with negative signals (fail, error, bug, missing,
etc.) are now treated as findings
- qa_synthesis: add comment documenting dedup limitation (exact text match
vs semantic — accepted for v1, false separation safer than false merge)
- workflow: extract init_user_intent inline script to intent_init.py module,
matching the qa_synthesis.py extraction pattern
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: synthesize_strategy reads field + qa_synthesis empty report warning
- Add strategy-plan.json to synthesize_strategy reads (was missing,
affects SKILL.md generation and executor ordering)
- qa_synthesis warns explicitly when no adversarial reports exist
instead of producing a false-clean report
- Switch from negative-signal allowlist to positive-signal denylist
so security findings ("authentication bypassed") aren't filtered out
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PROMPT_MUTATE now supports a PromptRewriter callback that REPLACES agent prompts instead of appending snippets. default_prompt_rewriter uses Opus via CLI — receives current prompt + reflection hint, writes a complete replacement that can remove bad advice and fix contradictions. Falls back to append when rewriter is unavailable. KNOB_MUTATE now accepts a reflection_report and uses guided mutation: 70% exploitation (pick the knob+value the reflector identified as best) and 30% exploration (random). _parse_knob_suggestion extracts structured suggestions from ReflectionReport.mutation_suggestions. Also changes the ReflectionReport import from TYPE_CHECKING to direct, since it's now used at runtime in mutate_knob. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: include prompt content in MAP-Elites feature extraction (#1388) compute_features() now hashes each AgentNode's prompt_template into a feature dimension. Without this, PROMPT_MUTATE variants with the same knob config all land in the same MAP-Elites cell and only the highest-scoring one survives — effectively discarding prompt diversity. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: cover all mutation types in MAP-Elites feature extraction (#1388) compute_features() now includes feature dimensions for every mutation type so that different mutations land in different MAP-Elites cells: - NODE_INSERT/REMOVE → agent_count, depth (already covered) - PARALLELIZE/SERIALIZE → fork_degree (already covered) - EDGE_REDIRECT → edge_hash (sorted edge signature, 8 buckets) - PARAM_MUTATE → param_hash (model + timeout per agent, 8 buckets) - KNOB_MUTATE → knob value hashes (10 buckets each) - PROMPT_MUTATE → prompt content hashes (8 buckets per agent) Without this, PROMPT_MUTATE and PARAM_MUTATE variants with the same topology and knob config all collapse into one cell — only the highest-scoring one survives, discarding all diversity. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: fixed-length feature tuples with aggregate hashes Addresses CEO review feedback: 1. Feature tuples are now always 8 elements regardless of agent count: (depth, fork_degree, agent_count, gate_count, edge_hash, param_hash, prompt_hash, knob_hash). This fixes the variable-length tuple issue that caused silent truncation in zip-based Pareto comparisons. 2. Prompt and param hashing now use sorted node IDs for deterministic ordering across different workflow construction paths. 3. Fixed test_population.py assertion (len == 8, not 4). 4. Added tests: prompt mutations produce different features, no-agent workflows still produce 8-element tuples. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Bump CLI timeout from 60s to 120s (Opus needs time for quality rewrites) - Add structured logging for all outcomes: success, empty, timeout, error - Log prompt_mutate_validation_failed when rewritten prompt fails model_copy - Log knob_expander outcomes (empty, timeout, error, success) Every failure path in the mutation pipeline is now observable via structlog, making it possible to diagnose why mutations fail silently. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…#1394) sample_parent() gains rank_weighted option: individuals are drawn with probability proportional to their rank (best=N, worst=1) instead of uniformly. This biases toward stronger parents while preserving diversity. Changes: - MAPElitesArchive.sample_parent(rank_weighted=True) uses random.choices with rank-proportional weights - SwarmConfig.rank_weighted_selection: bool = False (backward compat) - SwarmEngine threads the config through to sample_parent - 2 new tests: rank bias verification, uniform baseline Fixes #1393 Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…1412) sample_parent() now automatically enables rank-weighted tournament selection when the archive reaches auto_rank_cell_threshold (default 8) occupied cells, indicating enough diversity that biasing toward stronger parents is beneficial. This mirrors the on_plateau() pattern of adapting strategy when the search state calls for it. On by default; opt out with auto_rank_weighted=False. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…1408) ## Summary - New `--mode create-v2` factory workflow mode that replaces generic QA pipeline with workflow-specific testing and Overwatch verification gate - Inherits from `create_workflow()` via inherit-and-mutate pattern (29 nodes, 33 edges) - Adds Research Director, Strategy Director, QA Director (with mandatory workflow-validate and cli-integration testers), and Overwatch (4-step verification) - User intent ledger threaded through 6 stages for requirement tracking - QA synthesis glob fix: `adversarial-*` → `adversarial*` to match actual filenames - SKILL.md line limit raised 600→1200 to accommodate v2 director patterns ## Test plan - 150 dedicated tests (5 smoke + 145 integration) across 12 test classes - All 16 SKILL.md validations pass - Full suite green (no regressions) - Ruff clean, mypy clean Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two fixes to mutate_knob(): 1. Guided mutation no-op detection: when the reflector suggests a value that's already the current value (e.g. theory -> theory), skip it and fall through to random selection instead of wasting an eval slot. 2. Exclude synthetic _prompt_* knobs from random selection. These are created by PROMPT_MUTATE to persist prompts through compile() round-trips and should not be randomly selected by KNOB_MUTATE. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…1411) * fix: PROMPT_MUTATE persists in knob_values for compile() round-trips mutate_prompt() now stores the rewritten prompt in knob_values under a synthetic `_prompt_<node_id>` key. Package.compile() reads these back and applies them to node prompt_templates. Without this, any consumer that rebuilds a Package from config and calls compile() loses prompt mutations — the rewritten prompt lives only on the Workflow IR that apply_random_mutation returned, and the fresh compile() overwrites it with the original. Fixes #1410. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: preserve _prompt_* entries when compile() has declared OptKnobs compile() was replacing knob_values/knob_expandable entirely from declared OptKnobs, destroying _prompt_* entries before they could be read back. Now saves and restores _prompt_* entries across the overwrite. Found by CEO adversarial QA (Test 3): any workflow with both PROMPT_MUTATE and declared OptKnobs silently lost all prompt mutations. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…1418) (#1423) _extract_knob_patterns now recognizes _prompt_* and prompt_* keys: - Truncates prompt values to 100 chars for comparison (full text would make every value unique, defeating the contrastive analysis) - Reports PROMPT_MUTATE suggestions instead of KNOB_MUTATE for prompt-type keys - Displays truncated values (60 chars) in suggestions Fixes #1418 Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Non-zero exit codes from invoke_agent now log a warning and return the stdout (which may contain partial output) instead of raising RuntimeError that halts the entire workflow. This allows downstream nodes to proceed even when one agent fails transiently. Fixes #1415 Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
) (#1426) Adds async_prompt_rewriter alongside the existing sync default_prompt_rewriter. Uses asyncio.create_subprocess_exec instead of subprocess.run, so async consumers don't block the event loop for 30-120s per prompt rewrite. The sync version remains as the default for backward compatibility. Async consumers can pass async_prompt_rewriter as the rewriter parameter to mutate_prompt. Fixes #1417 Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…cts (#1425) * fix: use exec mode for python3 -c gate commands to avoid quote conflicts (#1416) Gate evaluator_commands with python3 -c had nested double quotes (e.g. print("PROCEED")) that broke shell parsing. Now detects python3 -c commands and runs them via create_subprocess_exec with the code as a separate argument, bypassing shell quoting entirely. Also removed shlex.quote on {project_path} in gate commands since exec mode doesn't need it and shell mode handles it naturally. Fixes #1416 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address CEO review — restore shlex.quote, use _run_shell for gates 1. Gate evaluator calls _run_shell (not _run_shell_or_exec directly) to preserve test mockability 2. Restored shlex.quote on project_path for gate commands — needed for shell-path commands with spaces Both smoke tests now pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
create-v2 was missing from the focus allowlist in _validate_ceo_flags(), causing `factory ceo --mode create-v2 --focus "..."` to error out despite the mode being properly registered. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Custom/plugin-defined modes (e.g. find-upstream-patch, buildroot) would complete their workflow in headless mode but then chain into discover/improve cycles via _chain_modes(), running for 23+ minutes until timeout. The chaining logic failed for unregistered modes because WorkflowRegistry.get_workflow() returned None, bypassing the terminal check. Remove _chain_modes() entirely — every mode now completes and exits. If further work is needed, the user or CI pipeline invokes the next mode explicitly. Fixes: https://github.com/akashgit/refactory-midstream/issues/2415 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add TaskRef model and --task-module CLI flag for outer loop task discovery Enable projects with custom Task subclasses to pass their task class to `factory outer-loop calibrate/evaluate` via `--task-module module:ClassName`. - Add TaskRef(BaseModel) in factory/task.py mirroring EvaluatorRef pattern - Add task_module field to SwarmConfig with 3-tier get_task() precedence - Add --task-module flag to calibrate and evaluate CLI subparsers - Add 7 tests covering resolution, errors, precedence, serialization - Document Task Discovery in docs/outer-loop.md Closes #1462 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: skip _inner_loop_factory when task_module is set in evaluator When a Task is available via config.get_task(), the compose() path uses the workflow object directly and never needs CLI mode resolution. Move the task check before the _inner_loop_factory call so ephemeral mode names (e.g. 'evolve-gen0-xxx') don't trigger noisy 'unknown mode' errors. The task path now uses 'task-eval' as a simple label instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: restore subset_selector wiring and deleted tests in evaluator Restores the subset_selector wiring in _evaluate_via_inner_loop that was removed when the task_module skip logic was added. Also restores the two deleted tests (test_task_path_wires_subset_selector and test_task_path_no_subset_selector_when_empty_instances), updated to reflect the new mode_name='task-eval' behavior and assert that _inner_loop_factory is NOT called when a task is set. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add --dangerously-skip-permissions to Task.run() CEO subprocess Task.run() spawns a 'factory ceo' subprocess but was missing --dangerously-skip-permissions, causing 'Not logged in' failures in eval worktrees where the directory isn't trusted by Claude Code. This matches invoke_agent() which already defaults this flag to True. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…I subprocess (#1464) Task.run() shells out to `factory ceo`, not to `claude`. The --dangerously-skip-permissions flag belongs to the claude CLI, not the factory CLI, causing `factory: error: unrecognized arguments`. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: close outer loop reflection→mutation wiring gaps Wire up the 7 confirmed gaps that made the CLI-driven outer loop's guided mutation pipeline structurally dead: 1. _cmd_evolve now loads persisted ReflectionReport from disk and passes it to apply_random_mutation, activating guided mutations on CLI path 2. _cmd_reflect builds knob_values_by_id from registry workflows and passes it to reflector.reflect(), activating knob-contrastive analysis 3. engine.py builds knob_values_by_id from population individuals and passes it to reflector.reflect() on the in-process path 4. _extract_knob_patterns populates prompt_improvements for prompt knobs, so _extract_prompt_hint returns targeted guidance instead of generic 5. _PROMPT_VARIANTS uses domain-neutral reasoning strategies instead of coding-specific strings 6. Benchmark configs clear dead seed_workflow name='improve' to name='' 7. Delete orphaned factory/outer_loop/prompts/reflect.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: update seed_workflow assertion to match featurebench.toml change The Bug 5 fix changed benchmarks/configs/featurebench.toml seed_workflow from 'improve' to '' (empty string), but the backward-compat test still asserted 'improve'. Align the assertion with the config. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove unused import and variable flagged by ruff - tests/test_outer_loop/test_cli.py: remove unused MagicMock import (F401) - tests/test_outer_loop/test_mutations.py: drop unused `result` assignment (F841) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add coverage for knob_values_by_id wiring and malformed reflection fallback Cover three previously-untested code paths in the outer loop: - _cmd_reflect building knob_values_by_id from registry workflows (lines 433-437) - _cmd_evolve gracefully handling malformed reflection JSON (lines 484-485) - engine.evolve_generation building knob_values_by_id from population (lines 271-277) These tests raise patch coverage from ~74% to above the 79% threshold. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…/Workflow separation (#1472) * feat: LLM-based reflector diagnosis, typed mutation suggestions, and OptKnob propagation Part A: Add _llm_reflect() to OuterLoopReflector — single claude -p call per generation contrasts top-K vs bottom-K individuals, populates the previously dead report.prompt_improvements field. Enabled via llm_reflect=True kwarg (defaults False so existing tests don't hit real subprocess). Part B: Add MutationSuggestion dataclass and typed_suggestions field to ReflectionReport. All suggestion-building sites (_generate_mutation_suggestions, _generate_structural_recommendations, _extract_knob_patterns) now produce typed objects alongside backward-compatible string fields. select_guided_operator and mutate_knob in mutations.py prefer typed suggestions with direct enum matching, falling back to string substring path when typed_suggestions is empty. Part C: Wire seed_workflow and Task.workflow() knob propagation in _cmd_calibrate so knob_values/knob_bounds/knob_expandable from source workflows survive into seed Individuals. Includes 31 tests covering all three parts plus E2E integration tests verifying reflector→mutation typed suggestion consumption and serialization round-trips. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: guard _llm_reflect against non-dict JSON and reject empty knob values Bug 1: _llm_reflect crashes with AttributeError when json.loads returns a non-dict type (list, int, str, null, bool). Added isinstance(data, dict) guard with early return. Bug 2: _parse_knob_suggestion accepts empty-string values that silently corrupt knob settings. Strengthened guard to also reject whitespace-only strings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: boost patch coverage for outer loop reflector, mutations, and CLI Add tests covering new code paths introduced in the LLM reflector, typed mutation suggestions, and OptKnob propagation changes: - reflector.py: avg_steps NODE_INSERT/REMOVE suggestions, ForkNode parallel recommendations, _collect_individual_details edge cases, _llm_reflect payload truncation and error handling, _save_report typed_suggestion serialization - mutations.py: select_guided_operator string fallback for NODE_INSERT and NODE_REMOVE, mutate_knob guided no-op when value matches current, float coercion in guided knob path - cli/outer_loop.py: fix broken seed_workflow/task_module calibrate tests (patching load_config instead of args namespace), add reflect not-enough-candidates error path, evolve no-modes error, evolve non-dict/non-list typed_suggestions handling, evolve registry.load returning None Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: split domain execution from workflow execution in Task Remove Task.run() entirely — the Task interface is now just four hooks: instances(), setup(), prompt(), verify(). Execution is driven by InnerLoop._step_with_task() which runs setup → WorkflowExecutor → verify per instance, with task.prompt() seeded as initial_context. - Add initial_context parameter to WorkflowExecutor for domain-specific per-instance prompts (seeds node_context on start_node) - Update ChessEvolveTask: remove run() override and _apply_mutation() - Update SWEBenchTask: docstring-only (already used default path) - Rewrite tests to mock WorkflowExecutor instead of task.run() Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add --seed-workflow CLI flag to outer-loop calibrate Adds explicit --seed-workflow flag that accepts a 'module.path:callable' ref returning a Package or Workflow with OptKnobs. This is the topology concern counterpart to --task-module (domain concern), completing the Task/Workflow separation for the outer loop. Precedence: --seed-workflow > config.seed_workflow > task_module fallback. Replaces the implicit Part C knob-loading logic with cleaner elif chain. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: make --seed-workflow required for calibrate, remove fallback paths The outer loop evolves workflows — it needs an explicit seed workflow. The generic benchmark workflow was always a placeholder. Remove the implicit registry and task_module fallback paths in favor of the explicit --seed-workflow flag. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove unused imports flagged by ruff Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove accidentally committed .claude and cache files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use None sentinel for required_capabilities to distinguish unset from empty TaskConstraints.required_capabilities defaulted to [] (falsy), so TaskCapabilities.from_task() always fell through to exit_code inference, adding CAN_MODIFY_CODE/CAN_RUN_TESTS/HAS_BUILDER even for domain tasks like chess-evolve that don't need software-dev capabilities. Changed to three-way semantics: - None (default) = not set, fall through to inference (backward compat) - [] = explicitly no capabilities needed (domain tasks) - [...] = explicit capability list (unchanged) Updated from_toml() to yield None for absent key, compose.py to check `is not None` instead of truthiness, and chess-evolve.toml to declare required_capabilities = []. Closes #1472 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: LLM reflection now produces typed mutation suggestions The _llm_reflect prompt only asked for prompt_improvements and failure_patterns (string lists). All typed_suggestions came from heuristic methods which are generic. Updated the LLM prompt to also request structured mutation_suggestions with operator/target/rationale/value, parse them into MutationSuggestion objects, and filter invalid operators. Added workflow node IDs to the prompt context so the LLM can target specific nodes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove unused tomllib import in test_compose Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: drop --max-turns 1 from LLM reflector, add retry --max-turns 1 in print mode conflicts with Claude Code's internal turn counting — sometimes an internal tool call consumes the single allowed turn, returning "Error: Reached max turns" instead of JSON. Removing the flag lets print mode handle turns naturally. Also adds 3-attempt retry with exponential backoff for transient failures (rate limits, network errors). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LL83gxqJ4qjR5oungfcVuf * feat: add experiment context to reflector with top-down budget _collect_individual_details now appends rec.experiments (hypothesis, verdict, score delta) after existing fields, gated by a char_budget parameter. _llm_reflect divides _LLM_PAYLOAD_BUDGET across individuals (85% usable, 15% reserved for headers) and passes per-individual shares. Experiments fill remaining space greedily — earlier experiments get priority, hypothesis text is truncated to fit, and the list stops when budget is exhausted. Closes #1472 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: guard initial_context assignment to AgentNode start nodes only The initial_context code logged 'initial_context_ignored' when the start node was not an AgentNode, but still wrote the context to node_context unconditionally. This meant GateNode and LLMNode paths silently consumed context that the log claimed was ignored. Now initial_context is only written to node_context when the start node is an AgentNode. For other node types, the warning is logged and the assignment is skipped. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: repair 7 broken mutation operators on small-topology workflows Six targeted fixes to factory/outer_loop/mutations.py: 1. Split mutable_nodes into structurally_mutable (excludes start node) and content_mutable (includes start node) so PROMPT_MUTATE and PARAM_MUTATE work when the only AgentNode is the start node. 2. NODE_INSERT now creates contextual complementary nodes with inherited file wiring (upstream writes → new reads, downstream reads → new writes) and a complementary role via _COMPLEMENTARY_ROLES mapping, instead of blank random-role agents. 3. NODE_REMOVE guards against removing the last AgentNode in a workflow. 4. EDGE_REDIRECT pre-filters targets using nx.ancestors() on the unconditional-edge subgraph to avoid creating ungated cycles, reducing wasted retries through validate_and_repair(). 5. insert_node ID generation uses loop-until-unique pattern to prevent silent dict overwrite on collision. 6. mutate_params and mutate_prompt use constructor-based creation instead of model_copy(update=...) so Pydantic validation actually runs on updated fields. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: make test_remove_last_agent_returns_none deterministic The test used a 3-node workflow (start→agent→end) where both agent and end were structurally mutable. random.choice could pick the FnNode ~50% of the time, bypassing the last-AgentNode guard and causing ~30% flake rate. Fix: use a 2-node workflow (start→agent) so agent is the only structurally mutable node, guaranteeing the guard is always tested. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: propagate eval scores to population and add tournament selection in outer-loop CLI Two bugs made the CLI outer-loop path fitness-blind: 1. `_cmd_evaluate` wrote scores to `results/gen{N}.json` but never updated `population/population.json`, so every individual kept `score=0.0`. 2. `_cmd_evolve` mutated `modes[:population_size]` — every candidate reproduced exactly once regardless of fitness, so there was zero selection pressure. Changes (all confined to `factory/cli/outer_loop.py`): - `_propagate_scores_to_population()` loads the population after evaluation, matches modes to individuals via the `evolve-gen{N}-{id[:8]}` naming convention, and writes back score + cost_usd. No-ops when the population file is absent. - `_load_mode_scores()` reads `results/gen{N}.json` into a `{mode_name: score}` map, tolerating a missing or malformed file. - `_select_parent()` performs k=2 tournament selection over mode names ranked by score; unevaluated generations degrade to uniform random. - `_cmd_evolve` now restricts parents to the current generation's candidate modes (excluding `-eval-` mirrors) and draws one parent per offspring slot, printing the parent mode and score for observability. Tests in `tests/test_outer_loop/test_cli.py` cover score propagation (including missing-population and unmatched-mode paths), score loading, tournament distribution, and a 200-trial statistical check that offspring are biased toward higher-scoring parents. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: add offspring individuals to population.json during CLI evolve _cmd_evolve created mode files via registry.register() but never added corresponding Individual objects to population/population.json. This meant gen 1+ individuals had no population entry, so _propagate_scores_to_population() could never update their scores after evaluation — the population was frozen at gen 0. Changes (all in factory/cli/outer_loop.py): - _make_offspring_individual() builds an Individual from a freshly registered offspring mode, with proper id (via _mode_suffix), features (via compute_features), parent_id, and mutation_record. - _cmd_evolve now loads the Population at the start, adds each offspring Individual after registry.register(), and saves back after the loop. Tests in tests/test_outer_loop/test_cli.py cover offspring persistence, id-matching for score propagation, population growth across generations, and the no-population-file fallback. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove model from param_mutate's mutable parameters Model is infrastructure configuration, not an evolvable parameter. param_mutate was randomly switching models between sonnet/opus/haiku, defeating the purpose of baking a specific model into seed workflows. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
… (#1477) 10 tests that chain real main(["outer-loop", ...]) calls against a shared tmp_path project, asserting both exit codes and on-disk artifacts between each step using real loaders (load_config, load_checkpoint, Population.load). Tests cover: - calibrate→evaluate, evaluate→reflect, reflect→evolve, full pipeline+status - evaluate --project-dir mirroring to separate target directory - promote reads mode files written by calibrate - reflect deserializes cycle_summary.json from evaluate - calibrate propagates knob_values/bounds/expandable into seed population - typed_suggestions round-trip: reflect writes, evolve deserializes - _llm_reflect mocked to prevent claude subprocess in tests - _resolve_seed_workflow mocked for required --seed-workflow flag Closes #1468 Claude-Session: https://claude.ai/code/session_01PcxJrv9nztJyDTkdg3v9Eg Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…rkflows Add DataItem and DataNode as workflow graph primitives, enabling data-driven fan-out where a subgraph is executed per data item with Semaphore-throttled concurrency and per-item fault isolation. Touches 8 source files across 3 architectural layers: - Models: DataItem + DataNode with exactly-one-source validation - Executor: _execute_data with task_ref/inline/source_path resolution - Validation: structural + reachability checks for both functions - Skill export: DataNode rendering + subgraph skip-set + trailing else - Inner loop: cached DataNode detection, executor delegation - Compose: CAN_ITERATE capability inference - MAP-Elites: appended has_data_node feature axis (9-tuple) - Diversity: generalized to len(key) instead of hardcoded range(4) Closes #1482 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add assert for self.workflow before WorkflowExecutor construction in _step_with_data_node() to satisfy mypy's type narrowing. Add tests for directory, jsonl, csv source_format paths and non-existent source_path in the executor, plus a test for the inner_loop delegation path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…r loop pipeline Adds TestDataNodeIntegration class with 5 tests covering: - DataNode workflow routing through _step_with_data_node() - DataNode delegates to executor once (not per-instance) - Non-DataNode workflows still use manual per-instance loop - compute_features detects DataNode at feature index 8 - Full pipeline: DataNode → InnerLoop.step() → CycleRecord + features Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DataNodes are infrastructure that drives per-item subgraph execution — the optimizer should never accidentally mutate or remove them. Add _auto_frozen_nodes() helper and apply it at both mutation call sites in SwarmEngine (seed and evolve_generation). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ugj23EAAf1HoHcknrBwhL
…don't timeout The per-item WorkflowExecutor in _execute_data() was created with empty completed_files, causing _wait_for_reads to poll for 60s then halt if the subgraph's start node had reads dependencies on upstream artifacts. Copying the parent's completed_files into each item_executor lets the subgraph inherit the parent's artifact state. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…Node DataNode sub-executors timed out (60s) waiting for files that task.setup() had already created on disk. _wait_for_reads only checked the completed_files set, which tracks node outputs — not filesystem state. Before spawning per-item sub-executors, scan all subgraph node reads and add any that already exist on disk to completed_files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DataNode with task_ref resolved items and ran subgraphs but never called task.verify(), silently scoring every completed subgraph as 1.0 regardless of the task's actual evaluation. This made task-defined pass/fail invisible. Changes: - executor.py: Pair TaskInstances with DataItems so filters apply in lockstep. Move setup()/prompt() from eager pre-loop into run_item() for per-item fault isolation. Call task.verify() after each subgraph completes and use its score/passed/details instead of binary subgraph success. - inner_loop.py: _step_with_data_node() now aggregates per-item verify scores (mean) from DataNode output instead of using binary exec success. - inline_items and source_path paths remain unchanged (no Task, no verify). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…1480) _step_with_task() hand-built a CycleRecord without calling CycleAnalyzer, so total_cost_usd was always 0.0. Now captures event_offset before the instance loop and uses CycleAnalyzer afterward to aggregate costs from events.jsonl — matching the pattern _step_subprocess() already uses. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
# Conflicts: # factory/inner_loop.py
- Fix 1: parallelism default 3→1, Field(ge=1) constraint, worktree isolation for parallelism>1 - Fix 2: write current_item.json for subgraph item visibility (cleaned up in finally) - Fix 3: reject explicit DataNode→subgraph edges in validation - Fix 4: raise on missing source_path, warn on empty data, isolate JSONL errors - Fix 5: seeded shuffle via random.Random(seed), shuffle_seed field on DataNode - Fix 6: direct DataNode score lookup by node ID + populate instance_results - Fix 7: document spec deviations in PR body (applied via gh pr edit) - Fix 8: compose() relaxes capability validation for DataNode workflows executor.py exceeds 500-line gate (was already 1405, now 1507) — _execute_data method is tightly coupled to executor internals, splitting would hurt readability. Addresses review feedback from @lambdabaa on PR #1483. Fixes compose() DataNode gap from issue #1488 Gap 0.
A gate could route only proceed/reloop/halt, so a graph whose gate selects among alternative forward continuations (draft/improve/debug, mutate/merge) had to spell one branch 'reloop' — which every consumer reads as a rewind. An edge condition is now a free label matched case-insensitively; a gate-named label becomes the edge's lowercase condition. The three verdict labels are unchanged, so existing graphs and executors behave identically. GateNode.max_iterations lets a gate carry its own reloop cap through the graph, so a budget-gated loop is not bounded by the runtime default.
…ion (#1487) reads and writes are sets, whose iteration order is not stable across processes, so a given graph could serialize to different bytes on different runs. Sorting them makes graph.json byte-reproducible — required for a wire format that consumers diff, hash, and compare. OptKnob gains a human-readable description so a downstream package can carry prose about a knob without inventing a side channel.
VerdictType(str, Enum) instances ARE strings, so .value is redundant and mypy flags it as attr-defined on the str union branch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reverts str() back to .value (str() produces 'VerdictType.PROCEED' instead of 'proceed'). Adds type:ignore to silence mypy since VerdictType(str, Enum) makes mypy see the str branch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
colehurwitz
force-pushed
the
factory/run-3110c50a
branch
from
September 10, 2026 20:39
0b19496 to
b63265d
Compare
* feat: first-class plugin workflow registration (closes #1490) Plugins can now register workflows via PluginRegistry.add_workflow_search_path() and have WorkflowRegistry.discover() consume them natively, replacing the manual _search_paths bridging downstream consumers currently do. Discovery now layers sources by priority: project > plugin > user > builtin. Name collisions between sources are resolved by priority with a logged warning instead of last-write-wins (which previously let plugin paths silently override project-local workflows). The skill cache checksum now hashes discovered workflow source files in addition to the model dump, so editing a plugin workflow file invalidates the exported SKILL.md cache even when the change is not semantic (e.g. comments). _compute_checksum takes an optional source_files mapping; existing callers are unaffected. Co-Authored-By: Claude Code <noreply@anthropic.com> * feat: warn on plugin mode/workflow drift at discovery time Review feedback on #1491: a plugin mode with no discovered workflow of the same name usually means a typo or renamed file, and without a warning it only surfaced as a silent fallback to the default improve loop. discover() now logs a warning for declared-but-missing modes and an info event for discovered-but-undeclared plugin workflows (legal: subgraph libraries, composed packages). With no plugin modes declared there is nothing to drift against, so the check is skipped. Co-Authored-By: Claude Code <noreply@anthropic.com> --------- Co-authored-by: Claude Code <noreply@anthropic.com>
FIX 1 (BLOCKING): Format/path-kind mismatch raises ValueError - source_format='directory' on a file → ValueError - source_format='jsonl'/'csv' on a directory → ValueError FIX 2 (BLOCKING): diversity_metric uses named structural axes - STRUCTURAL_AXES = (0, 1, 2, 3, 8) module-level constant - Excludes hash-bucket axes 4-7 (edge_sig, param, prompt, knob) - Correctly includes has_data_node (index 8) instead of edge_sig (index 4) FIX 3: Deterministic shuffle seed via hashlib.sha256 - Replaces hash() which is PYTHONHASHSEED-randomized FIX 4: initial_context=item.prompt or None - Suppresses noisy initial_context_ignored warnings for empty prompts FIX 5: Test coverage for _workflow_has_data_node, _step_with_data_node, and format/path-kind validation Also includes uncommitted Edge.condition_label property from prior review round. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
model defaults to '' and provider to 'auto', so a graph that does not pin them leaves the choice to the runtime. 'auto' resolves through the same non-vertex alias table as 'anthropic', so existing consumers behave identically.
The numerator counted all 9-dimensional grid cells (including hash-bucket distinctions) while the denominator used only the 5 structural axes. Two individuals with identical structure but different edge-hash buckets would produce a metric >1.0. Fix: count unique structural-axis cells in the numerator to match the structural denominator. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes akashgit#1483
Changes
assert self.workflow is not NonebeforeWorkflowExecutorconstruction in_step_with_data_node()(factory/inner_loop.py:391). This is safe because the method is only called after_workflow_has_data_node()which already checksself.workflow is not None.directorysource_format: temp dir with subdirs → each becomes a DataItemjsonlsource_format: temp .jsonl file → each JSON line becomes a DataItemcsvsource_format: temp .csv file → each row becomes a DataItem_step_with_data_node()(factory/inner_loop.py): verifies the delegation path produces the correct CycleRecord🤖 Generated with Claude Code