BACKLOG.md

Backlog

Known improvements deferred from code reviews and audits.

Testing

• (no outstanding gaps)

Code Quality

• CLI arg parsing refactor — src/index.ts uses a manual index-tracking loop for flag parsing. Low priority: the flag set is small and unlikely to grow.

Features

• True mid-step interjection (kill + resume) — The current i key queues a correction for the next Claude step. To truly stop a running Claude step and redirect it mid-execution, the approach is: kill the subprocess, then re-invoke with --resume <session_id> (captured from the result event) and the user's correction prepended. This preserves conversation context while immediately stopping the bad action. The session_id is available in Claude CLI's result event. The TUI would show a "restarting with correction…" log line. Blocked on: deciding UX (separate keybinding like I vs. a mode toggle), and verifying --resume behavior with --output-format stream-json.

• OpenCode server-mode integration — The current OpenCode runner uses opencode run --format json (CLI subprocess). A more robust integration would use OpenCode's HTTP server API (sessions, SSE event stream, messages endpoint). This enables better session management, lower startup overhead, and potentially mid-session context carry-over. Blocked on: OpenCode server API stabilizing.

• Per-attempt heal/judge child spans — Telemetry v1 models self-healing and judge activity as span events on the step span. Modelling each heal/judge attempt as its own child span needs start-boundary events from the runner (step:attempt / judge:start — only completion is emitted today) and TRACEPARENT propagation into the judge subprocess so its work nests under the attempt span.

• Complete cost accounting — Judge-evaluation cost is dropped inside runClaudeStructured (src/tasks/claude.ts consumes the event stream for output:structured/output:text only), and the OpenCode runner does not parse cost or token usage from its JSON output at all — an OpenCode-only run's workflow:report always shows zero tokens/cost regardless of actual usage. Surfacing both would make the executant.cost.usd metric, per-step cost attributes, and the run report's totals complete.

• Richer nested-workflow progress UI — workflow: steps (src/resolve-workflow.ts, runNestedWorkflow in src/runner.ts) currently surface their child steps as flat output:text log lines under the parent step, not a structured nested view. A StepNestedEvent (mirroring step:inner) plus a dedicated row component (mirroring IterationRow.tsx) would let the TUI render real nested progress. Deferred for v1: the flat-log approach reuses 100% existing plumbing and needed zero UI changes; revisit if users want to see per-child-step status at a glance rather than in the log pane.

• --from-step/--step/--to-step targeting into a nested workflow step — currently rejected outright with a clear error (runNestedWorkflow in src/runner.ts) rather than resuming into the child's own step list. Supporting it would mean extending the dot-notation FromStepTarget path semantics (currently step → forEach-iteration → child-step) to also cross a workflow: boundary.

• workflow: steps inside forEach/repeat — rejected at load time (src/load-workflow.ts). A nested workflow is resolved once, eagerly, before any iteration runs; threading a per-iteration {{item}} into it would mean re-resolving (and re-fetching, for a remote reference) once per item, which breaks the "fail fast before any step runs" guarantee eager resolution is meant to provide.

• Vars-aware caching in resolveWorkflow — two workflow: steps referencing the same file are currently fetched and parsed independently. A cache would need to key on the resolved path/URL and the step's vars: override (serialized), since two steps referencing the same file with different overrides must not share a resolved Workflow — added complexity that isn't justified without evidence repeated-reference workflows are common.

Implemented (status bar, 2026-08)

• ✅ Status bar wraps to a second line under a long repo name or branch — StatusBar (src/ui/StatusBar.tsx) rendered the repo name and branch at their full length with no regard for terminal width; a sufficiently long value (a long feature-branch name, or a checkout directory not named after the repo) wrapped the row onto a second line, breaking the fixed single-line layout the rest of the footer assumes. fitRepoLabel (src/lib/statusline.ts) now shrinks name and/or branch with an ellipsis to fit the terminal width minus the gauge segment's reserved width, handing a short name's unused budget to the branch. App.tsx passes stdout?.columns ?? 80 down as the new required columns prop. Caught via src/tests/statusline-ui.test.ts, whose "names the repo and branch" test also hardcoded the literal string "executant" as the expected repo name (true only when the checkout directory happens to be named after the repo) — updated to assert the shape of the segment rather than its exact text, so the suite doesn't assume a specific checkout location.

Implemented (subprocess lifecycle, 2026-08)

• ✅ timeout_seconds on a script step actually kills the command on dash-family shells — runCommand (src/tasks/command.ts) spawns sh -c "<command>" and, on timeout, called proc.kill() against that sh process alone. On shells that fork a real child for the command rather than exec-replacing themselves (verified with dash, Debian/Ubuntu's default /bin/sh, for anything beyond a single tail-callable command — e.g. sleep 60), killing only the sh PID left the actual command running as an orphan, still holding stdout/stderr open. The step's reader loop was waiting on EOF from that pipe, so it never saw one and hung indefinitely instead of throwing TimeoutError — silently defeating the feature on any Linux box where /bin/sh is dash (i.e. most). Fixed by spawning detached: true (making proc.pid the leader of its own process group) and killing the whole group (process.kill(-proc.pid!, "SIGTERM")) on timeout — startTimeout (src/tasks/stream.ts) now takes an optional onTimeout override for this, defaulting to the original proc.kill() for claude.ts/opencode.ts, which spawn the CLI binary directly and don't have this problem. Detaching moves the child out of executant's own process group, so command.ts also gained explicit SIGINT/SIGTERM/SIGHUP cleanup handlers (mirroring the pattern already used in claude.ts/opencode.ts for SIGTERM/SIGHUP) so a terminal Ctrl+C during a running script step still reaches the whole process tree instead of just executant itself. Caught via src/tests/command.test.ts's existing timeout test, which used to hang for the full 60s test-runner timeout on this sandbox rather than failing fast — a new regression test spawns a real detached grandchild and asserts it's gone (not orphaned) after the timeout fires.

Implemented (eval comparison observability, 2026-08)

• ✅ Run provenance + judge drift markers for eval comparisons — historical eval-comparison trends were previously opaque: a score change on the same eval/model could come from the model under test, or silently from a judge/prompt/eval regime change, with no way to tell which. Every EvalComparison (src/eval/index.ts) now carries a provenance record (src/eval/provenance.ts): runAt, repo/gitSha (from git), judgeProvider/judgeModel/judgeVersion (claude --version, when readable), and sha256 hashes of the judge prompt template and the resolved eval spec, combined into a comparisonFingerprint — the strict-comparability key. Per-case API cost (TestResult.costUsd, Claude only — OpenCode/local models don't report one) is captured alongside score and duration and rolled up into EvalRun.totalCostUsd. JSON/CSV output (src/eval/export.ts) include all of it; CSV repeats the values per row the same way duration_ms already did, keeping the pivot-table shape. --history <path> (new flag, wired into eval:compare) appends one JSONL record per model/eval per run (src/eval/history.ts); npm run eval:trend (src/eval/trend-index.ts) reads that log and renders per eval+model time series in two modes — all (every run, with a marker line wherever the judge/prompt/eval fingerprint changed since the previous run) and strict (only runs matching the latest fingerprint). The separate git-worktree-based workflow-eval harness (src/eval/workflow.ts) was left untouched — this only covers the criterion-judge comparison system docs/eval-comparison.md documents.

Implemented (TUI, 2026-08)

• ✅ Context gauge reads per-call usage, not the step's cumulative total — the gauge shipped reading output:usage, which is parsed from the Claude CLI's final result message. Those counts are summed across every API call the step made: each turn re-reads the whole cached prefix, so cache_read_input_tokens grows by roughly the context size per turn. Dividing that by a 200k window is a category error — it compares throughput to capacity. On a long agentic step it read 3781.1k/200k (~25 turns' worth), with the bar pinned at 100% because buildGauge caps pct. Verified against a real three-turn claude -p transcript: the calls occupied 37,670 → 37,844 → 38,166 tokens while the result message reported 113,680 — exactly the sum of the three. The fix adds a separate output:context event emitted from the stream's assistant messages (deduplicated by message id, since the CLI emits one per content block sharing that turn's usage) carrying the session's input + cacheCreation + cacheRead as of its latest turn; the gauge replaces on each one, and output:usage goes back to being purely the run report's throughput total. The gauge is also cleared on a prompt step's step:start, so each step's session starts from an empty window rather than inheriting the previous step's fill — one session per step, one gauge per session, never carried across or summed. A side benefit is that the gauge now moves during a step rather than only at its end.

• ✅ Context gauge replaces statusline forwarding — the original statusline read a statusLine.command out of .claude/settings.local.json/.claude/settings.json (project, walking up from cwd, then user-level), synthesized the JSON payload Claude Code sends its own statusline commands, spawned that command on start and every 30s, and rendered its first stdout line above the footer. That is now removed outright: findStatusLineCommand, buildStatusLinePayload, runStatusLine, and the useStatusLine hook are all gone. Two reasons. The payload was largely fabricated — executant is not an interactive session, so the session id and model were invented and the "cost" was an approximation of executant's own spend. And a user's statusline script is almost always reporting on the Claude Code session that launched executant: context and cost a child process cannot observe, and which say nothing about the run on screen. What replaced it is derived rather than fabricated — repo and branch from one git rev-parse, and the context window of the most recent invocation executant itself spawned, read straight off the output:usage event stream with no subprocess, no interval, and no settings file. EXECUTANT_STATUSLINE=0 still hides the row, so the existing opt-out is unchanged. This is a silent behavior removal: anyone with a statusLine.command configured keeps the same env var and the same screen row but gets different content, with no warning and no migration path. If the forwarding is ever wanted back, it belongs as an additional segment on StatusBar rather than as a replacement for the gauge.

Implemented (run report, 2026-08)

• ✅ workflow:report on successful completion — every run now ends with wall-clock duration, total cost, total token usage (parsed from the Claude CLI's usage object as new output:usage events — inputTokens/outputTokens/cacheCreationTokens/cacheReadTokens), how much of that fell into Anthropic's >200k-token extended-context pricing tier, and a per-step narrative (name/duration/cost/judge-healing history, kept even for steps that ultimately passed). The overflow figure is computed per call (src/report.ts's computeOverflow), not as a running session total, matching how the tier is actually billed. All of this is free (pure aggregation over events the run already produced) and always shown. Rendered in the TUI end-of-run block, the log file, and as a workflow:report NDJSON event in CI mode; stamped as root-span attributes when telemetry is on. A nested workflow: sub-run never produces its own report (RunOptions.report: false, mirroring retrospective: false).
• ✅ Efficiency suggestion: narrative-grounded and opt-in (v2, superseding the initial cut above) — the first version generated a suggestion automatically on every run, from a structural read of the task file alone. Two problems surfaced after shipping it: (1) it had no visibility into what actually happened during the run, only the static YAML, so it couldn't say where prompting fell short; (2) it ran unconditionally, spending an API call even on unattended/CI runs that never asked for it. Both are fixed: generateEfficiencySuggestion(workflow, stepNarrative) now reads the run's own judge/self-healing history first — a judge FAIL's feedback or a self-healing fix is direct evidence of where the prompting fell short, and takes priority over any structural YAML observation — falling back to the v1 structural checks (concurrency:, type: script, type: workflow, etc.) only when every step passed clean. It also judges whether the task file reads as a reusable template (parameterized vars:, generic language, used across repos) versus a one-off, and phrases the suggestion accordingly. isEfficiencySuggestionEnabled() now defaults to off — nothing runs automatically unless EXECUTANT_REPORT_SUGGESTION=1 is set. The interactive path is src/ui/ReportPrompt.tsx: once the free part of the report is on screen, pressing a runs the analysis on demand (any other key skips it), calling generateEfficiencySuggestion directly since by then runWorkflow has already finished. The timeout grew from 10s to 600s to match — nothing is blocked on this call anymore, either because the run already finished (TUI) or because the caller explicitly opted in and accepted the wait. The prompt (src/prompts/efficiency-suggestion.txt) went through several rounds of eval-driven refinement (evals/efficiency-suggestion.eval.yaml, now 9 cases covering narrative-friction priority, structural fallback, reusable-template framing, and prompt-injection resistance) after adversarial cases caught it hallucinating that an already-present field (type: script, concurrency: N) was missing, and — after the narrative redesign — being too eager to declare "nothing to improve" once the narrative was clean. Both were fixed with explicit "check before claim" instructions and a strict priority order. A suggestion generated automatically (EXECUTANT_REPORT_SUGGESTION=1) is included in the log file's report block; one generated on demand in the TUI is shown on screen only, since the log's report block is written before the interactive analysis is ever requested.

Implemented (workflow authoring, 2026-08)

• ✅ concurrency: N on forEach/repeat — iterations ran strictly one at a time with no way to opt out. Now concurrency: N runs up to N iterations at once in batches (batch two starts only once every item in batch one settles), for the case where iterations are genuinely independent and the per-item work is slow enough that wall-clock time matters. Omitted or 1 keeps the original sequential behavior byte-for-byte unchanged. Required a new step:iteration-complete event and a reducer rewrite: the previous TUI state model inferred an iteration's completion from the next iteration starting, which only held because at most one was ever running — under concurrency that inference is simply wrong, so completion is explicit now (and the sequential path emits the same event, for one consistent mechanism instead of two). No --from-step support into a specific iteration of a concurrent step — a resume target inside one re-runs every iteration from the start instead, with a warning; the intended recovery path is iteration-level idempotency (detect and skip already-done work), not infra-level resume.
• ✅ output: on prompt steps — previously only script/command steps captured stdout to the named file; on a prompt step output: parsed fine but was silently a no-op. Now a prompt step's output: checks the named file exists after the step completes (a prompt step's real artifact is whatever it wrote via tool calls, not its narration text) and fails the step if it doesn't — no self-healing/judge retry on that failure, deliberately, for the same reason self-healing stays off deterministic script steps whose failure should hard-stop the run. Setting output: on a log/workflow step (which have nothing to produce) is now a load-time error instead of a silent no-op.

Implemented (observability, 2026-07)

• ✅ Structured step:healing / step:judge events — the self-healing loop and LLM-as-judge now emit typed events (phase/attempt/exit code; verdict/attempt/feedback) alongside the existing free-text logs, giving CI NDJSON consumers and telemetry machine-readable quality-control progress.
• ✅ Indexed output:cost — cost events now carry the 0-based step index (-1 sentinel patched by runWorkflow, like output:tool), enabling per-step cost attribution.
• ✅ OpenTelemetry telemetry module (src/telemetry.ts) — opt-in via OTEL_EXPORTER_OTLP_ENDPOINT; one trace per run (root → step → iteration spans with tool/healing/judge span events and cost attributes) plus five metrics, exported via OTLP/HTTP and flushed on every exit path including SIGINT; all @opentelemetry/* imports are dynamic, so the SDK is never loaded when the env var is unset.
• ✅ TRACEPARENT propagation — all three spawn sites (claude, opencode, bash) spread traceparentEnv() from the src/lib/trace-context.ts registry into the child env, so subprocesses join the current step's trace; a no-op when telemetry is off.
• ✅ CI mode skips the update check — checkForUpdate now runs only in TUI mode; headless runs no longer keep the event loop alive up to 5 s for a banner only the TUI renders.

Implemented (code review fixes, 2026-06)

• ✅ workDir in RunOptions — .executant-cancel is now checked next to the workflow YAML (dirname(resolve(filePath))) rather than fixed to process.cwd() at module load time; predictable regardless of invocation directory.
• ✅ lastStepOutput on continueOnError failures — workflow:complete.lastOutput now always reflects the last executed step (including failing continueOnError steps), not the last successful step.
• ✅ CI stdout flush before process.exit(4) — workflow:cancelled in CI mode now exits only after the write callback confirms the data was flushed to the OS, preventing truncation on piped streams.
• ✅ lines[] ring buffer — output lines are capped at LAST_OUTPUT_MAX_LINES (100) during collection via shift-on-overflow; memory cost is constant regardless of step verbosity.
• ✅ Shared startTimeout helper — duplicated timeout pattern extracted to stream.ts; command.ts and claude.ts both use startTimeout(proc, taskName, timeoutSeconds).

Implemented (operator feedback, 2025-06)

• ✅ --var KEY=VALUE CLI flag — override or supply workflow vars at runtime without editing YAML; multiple flags accepted; CLI overrides YAML.
• ✅ Auto-create task directories — tasks/todo and tasks/done under .claude/executant.local/ are created automatically on startup.
• ✅ lastOutput on events — step:error and workflow:complete carry the last 100 lines of step output, giving CI consumers structured context without slicing raw NDJSON.
• ✅ File-based cancellation — writing .executant-cancel in the working directory stops execution cleanly between steps; exits 4.
• ✅ Per-step timeout_seconds — script and prompt steps accept timeout_seconds: N; kills the subprocess and throws TimeoutError (exit code 3).
• ✅ Distinct exit codes — 0 success, 1 runtime failure, 2 validation error, 3 timeout, 4 cancelled.