Multi-Task Configuration Parser & Batch Executor

Solution for Mint-Claw/fleet-tools#1.

Parses a YAML/JSON task configuration and executes tasks in parallel with timeout enforcement, retry logic (exponential backoff), structured JSON output, graceful SIGINT/SIGTERM shutdown, and DAG dependency ordering.

Features

• Parallel execution — respects max_parallelism config field via ThreadPoolExecutor
• Timeout enforcement — per-task timeout (seconds), graceful kill on expiry
• Retry with exponential backoff — configurable retries and retry_delay (base seconds)
• Structured JSON output — per-task status, stdout, stderr, exit_code, duration, retries_used
• DAG dependency ordering — depends_on list with cycle detection; dependent tasks skipped if dep fails
• Graceful SIGINT/SIGTERM shutdown — in-flight tasks finish; pending tasks drained cleanly
• Comprehensive logging — timestamps, task IDs, retry counts, durations

Requirements

pip install pyyaml

Python 3.9+ (no other dependencies beyond stdlib).

Usage

python task_executor.py config.yaml [--output results.json] [--log-level DEBUG] [--dry-run]

Flags

Flag	Default	Description
config	required	Path to YAML or JSON task config file
--output	results.json	Output path for JSON results
--log-level	INFO	DEBUG, INFO, WARNING, ERROR
--dry-run	off	Parse and show plan without executing

Config Format

version: "1.0"
max_parallelism: 3        # max concurrent tasks
log_level: "INFO"

tasks:
  - id: task1
    command: "echo hello"
    timeout: 10            # seconds (default: 60)
    retries: 2             # retry attempts after failure (default: 0)
    retry_delay: 1.0       # exponential backoff base in seconds (default: 1.0)
    depends_on: []         # list of task IDs this task waits for

  - id: task2
    command: "echo world"
    timeout: 10
    retries: 0
    depends_on:
      - task1              # runs after task1 succeeds

Output Format

{
  "run_id": "uuid-v4",
  "started_utc": "2026-01-01T00:00:00Z",
  "completed_utc": "2026-01-01T00:00:05Z",
  "total_duration_seconds": 5.012,
  "config_max_parallelism": 3,
  "tasks": [
    {
      "id": "task1",
      "command": "echo hello",
      "status": "success",
      "exit_code": 0,
      "stdout": "hello\n",
      "stderr": "",
      "duration_seconds": 0.047,
      "retries_used": 0,
      "started_utc": "...",
      "completed_utc": "...",
      "error": null
    }
  ],
  "summary": {
    "total": 1,
    "success": 1,
    "failed": 0,
    "timedout": 0,
    "skipped": 0
  }
}

Task Statuses

Status	Meaning
success	exit code 0
failed	non-zero exit code, all retries exhausted
timedout	exceeded timeout seconds, all retries exhausted
skipped	dependency failed or shutdown before start
error	unexpected exception

Test Configs

Three test configs included in test_configs/:

Config	Tests
test_config_1.yaml	Parallel execution, DAG aggregate step, stdout capture
test_config_2.yaml	Timeout enforcement, retry with backoff, dependency skip
test_config_3.yaml	Complex multi-level DAG (10 tasks, 4 stages)

Run all three:

python task_executor.py test_configs/test_config_1.yaml -o results_1.json
python task_executor.py test_configs/test_config_2.yaml -o results_2.json
python task_executor.py test_configs/test_config_3.yaml -o results_3.json

Expected results:

• Config 1: 6 tasks, all success
• Config 2: 3 success, 1 timedout, 1 skipped (dependency on timed-out task)
• Config 3: 10 tasks, all success, in correct topological order

Performance (on provided test configs)

All 3 test configs complete in under 5 minutes total (Config 2 dominated by 2x2s timeout).

Wallet

Payment wallet (USDC/BTC): 0x7751BdDFC2d5e1A116d0F8d979a377d0f1c31Ef1