Autoresearch: Workflow engine node execution optimization loop

[sunapi386]

Context

Adapting Karpathy's Autoresearch pattern to autonomously optimize workflow node execution.

Why

The workflow engine executes DAGs of AI nodes — LLM calls, tool executions, branching logic. As workflows grow more complex (10+ nodes, nested sub-workflows), execution time and memory compound. Optimizing scheduling, parallelization, and node implementation directly impacts user experience and compute costs.

What

Set up the autoresearch loop:

File	Role	Who edits
benchmark.py	Runs test workflow suite, measures execution time + memory + correctness	Nobody (read-only)
engine.py	Scheduler, node implementations, parallelization logic	Agent only
program.md	Optimization targets, constraints, test workflows	Human only

Search space

• Scheduling strategies: Topological sort variants, priority queues, speculative execution
• Parallelization: asyncio concurrency limits, node-level vs branch-level parallelism
• Memory management: Streaming intermediate results vs buffering, context window packing
• Node execution: Batch compatible nodes, reuse connections, cache deterministic outputs

Evaluation metric

Primary: Total workflow execution time (seconds)
Secondary: Peak memory (MB)
Tertiary: Correctness (all node outputs match expected)

Prerequisites

• Define test workflow suite (simple linear, branching, nested, 20+ node DAGs)
• Implement benchmark.py with timing + memory + correctness checks
• Baseline measurement of current engine performance
• Scaffold program.md

---

Generated by Claude

[sunapi386]

Agent Bootstrap Prompt

Copy everything below when spawning a new Claude Code agent for this issue.

---

Context

You are working on aceteam-ai/workflow-engine — a Python package (aceteam-workflow-engine) that executes DAGs of AI workflow nodes. The goal is to set up a Karpathy-style autoresearch loop that autonomously optimizes node execution performance.

Related resources:

• Karpathy's Autoresearch — the pattern we're adapting
• aceteam-ai/trust-engine#1 — sibling autoresearch setup for the Trust Engine

Your task

Phase 1: Set up the autoresearch scaffolding alongside the existing codebase.

1. Understand the repo first. Read README.md, CLAUDE.md (if it exists), pyproject.toml, and explore the source structure. Understand:

   • How workflows are defined (DAG structure, node types)
   • How nodes are executed (scheduler, parallelism)
   • What the existing test suite covers
   • Current performance characteristics

2. Create the autoresearch files in a top-level autoresearch/ directory (to keep it separate from the package source):

```
autoresearch/
├── program.md # Autoresearch instructions for the agent
├── evaluate.py # Performance benchmark harness (read-only)
├── engine.py # Thin wrapper that imports from the real package
├── benchmarks/
│ ├── linear_5.json # Simple 5-node linear workflow
│ ├── branching_10.json # 10-node workflow with branches
│ ├── nested_20.json # 20-node nested sub-workflows
│ └── parallel_fan.json # Fan-out/fan-in pattern
└── results.tsv # Experiment log
```

3. Write `autoresearch/program.md` following the Karpathy pattern:

   • Setup: create dated branch, read all files, verify benchmarks exist
   • Experiment loop: modify engine code → run `evaluate.py` → extract metrics → keep or discard
   • Primary metric: Total execution time (seconds) across the benchmark suite
   • Secondary metric: Peak memory (MB)
   • Tertiary metric: Correctness (all node outputs match expected — non-negotiable, must be 100%)
   • Constraints: correctness must never degrade, only modify files in the package source (not evaluate.py or benchmarks), don't install new packages
   • NEVER STOP — loop indefinitely

4. Write `autoresearch/evaluate.py` — the benchmark harness:

   • Loads each benchmark JSON (workflow definition + expected outputs)
   • Executes the workflow using the engine
   • Measures wall-clock time (`time.perf_counter`) and peak memory (`tracemalloc`)
   • Verifies all outputs match expected values
   • Prints grep-able metrics: `total_time_s: X.XXX`, `peak_memory_mb: X.XX`, `correctness: X/X`
   • Uses mock LLM calls (deterministic responses) so experiments are reproducible without API keys

5. Write benchmark workflows (JSON or whatever format the engine uses):

   • `linear_5.json`: A → B → C → D → E, simple sequential
   • `branching_10.json`: Input → 3 parallel branches → merge → output
   • `nested_20.json`: Workflow with sub-workflow calls
   • `parallel_fan.json`: Fan-out to 8 nodes, fan-in to aggregator
   • Each includes expected outputs for correctness verification

6. Baseline measurement: Run `evaluate.py` on the current engine, record results in `results.tsv`

7. Update or create `CLAUDE.md` to document:

   • How to run autoresearch: `cd autoresearch && read program.md`
   • Metric definitions
   • What files are agent-editable vs read-only

Important: The autoresearch agent modifies the actual engine source code — not a copy. The `autoresearch/engine.py` should just import from the real package. This way improvements discovered by autoresearch are real improvements to the package.

Technical notes

• Use `uv` for dependency management
• Python 3.12+
• Mock LLM calls in benchmarks (no API keys needed for autoresearch)
• The engine should already have scheduling/execution logic — don't rewrite from scratch, just set up the optimization loop around what exists

---

Generated by Claude