diff --git a/amplifier_app_cli/data/skills/amplifier-config/SKILL.md b/amplifier_app_cli/data/skills/amplifier-config/SKILL.md index 4ef5b12..fa894d6 100644 --- a/amplifier_app_cli/data/skills/amplifier-config/SKILL.md +++ b/amplifier_app_cli/data/skills/amplifier-config/SKILL.md @@ -1,11 +1,11 @@ --- name: amplifier-config description: >- - Explain, inspect, troubleshoot, and safely modify Amplifier CLI configuration, - including provider selection, settings scopes, bundles, routing, skills, source - overrides, and provider-module settings composed into the CLI. Use when someone - describes desired Amplifier CLI behavior and needs effective-value provenance, a - safe change plan, implementation, or outcome-based verification. + Explain, inspect, troubleshoot, and safely modify Amplifier CLI + configuration — providers, settings scopes, bundles, routing, skills, and + source overrides. Use when someone describes desired Amplifier CLI behavior + and needs effective-value provenance, a safe change plan, implementation, or + outcome-based verification. user-invocable: true version: 0.1.0 license: MIT diff --git a/amplifier_app_cli/data/skills/goal-batch/SKILL.md b/amplifier_app_cli/data/skills/goal-batch/SKILL.md index f352d2d..ae1e757 100644 --- a/amplifier_app_cli/data/skills/goal-batch/SKILL.md +++ b/amplifier_app_cli/data/skills/goal-batch/SKILL.md @@ -1,22 +1,25 @@ --- name: goal-batch description: > - Plan a batch of independent work into isolated lanes, get your approval, then - run each lane as its own autonomous /goal session — one git worktree, one - branch, one tmux session each — and verify and merge the results yourself. - Use when work decomposes into pieces that can run at the same time: "run these - in parallel", "goal-batch", "launch lanes for these", "work these N tasks - simultaneously", "batch these as goals". Nothing launches until you have seen - the lane split and said go. This is NOT fire-and-forget: the orchestrating - session re-runs the full test suite itself after every merge and never accepts - a lane's own claim that it finished. NOT for bounded edits that each end in - their own PR — use mass-change for that. Requires git, tmux, the amplifier CLI - on PATH, and the goalify and monitor skills. + Batch independent work into isolated /goal lanes — one worktree, branch, and + tmux session each — get your approval, then verify and merge every lane + yourself. Use when work decomposes into pieces that can run at the same + time: "run these in parallel", "goal-batch", "launch lanes for these", + "work these N tasks simultaneously", "batch these as goals". Nothing + launches until you've seen the lane split and said go, and this is NOT + fire-and-forget: the orchestrating session re-runs the full suite after + every merge and never accepts a lane's own claim that it finished. NOT for + bounded edits that each end in their own PR — use mass-change for that. For + continuous refill-on-drain parallel work toward an outcome, use + ten-lane-highway instead. Requires git, tmux, the amplifier CLI on PATH, + and the goalify and monitor skills. version: 2.0.0 user-invocable: true argument-hint: "" allowed-tools: [bash, read_file, write_file, edit_file, grep, glob, delegate, load_skill, todo] model_role: general +visibility: + summary: "One-shot batch of parallel /goal lanes that launches once and drains together; for continuous refill-on-drain work use ten-lane-highway." --- # Goal Batch diff --git a/amplifier_app_cli/data/skills/goalify/SKILL.md b/amplifier_app_cli/data/skills/goalify/SKILL.md index 1b602ee..bf40998 100644 --- a/amplifier_app_cli/data/skills/goalify/SKILL.md +++ b/amplifier_app_cli/data/skills/goalify/SKILL.md @@ -1,8 +1,8 @@ --- name: goalify description: > - Compose a /goal stop-condition from the current conversation and lint it - against known termination-failure patterns before showing it to the user. + Compose and lint a /goal stop-condition from the current conversation, + catching known termination-failure patterns before showing it to the user. Use when the user wants to turn the current task into a /goal loop, asks to "goalify this", wants a stop condition for autonomous work, says "write a goal condition", "make this a /goal", "turn this into a goal", or asks for diff --git a/amplifier_app_cli/data/skills/ten-lane-highway/SKILL.md b/amplifier_app_cli/data/skills/ten-lane-highway/SKILL.md index 205520a..ffc082f 100644 --- a/amplifier_app_cli/data/skills/ten-lane-highway/SKILL.md +++ b/amplifier_app_cli/data/skills/ten-lane-highway/SKILL.md @@ -1,22 +1,23 @@ --- name: ten-lane-highway description: > - Become the Highway Manager: a strategist that takes a defined outcome plus - constraints and drives many parallel /goal lanes continuously toward it — - plan, get one approval, saturate to width, then verify/merge/refill on every - wake so lanes never sit idle, weaving new feedback into priorities as it - arrives. Use when the user wants continuous parallel throughput toward an - outcome: "run the highway", "/highway", "10-lane highway", "keep N lanes - full", "keep re-feeding lanes as they drain", "drive this for me in - parallel". NOT for a one-shot batch that launches once and drains together — - use goal-batch. NOT for bounded edits that each end in their own PR — use - mass-change. Requires git, tmux, the amplifier CLI on PATH, and the goalify - and monitor skills. + Become the Highway Manager: drive many parallel /goal lanes continuously + toward a defined outcome, refilling lanes on every wake so none sit idle. + Use when the user wants continuous parallel throughput toward an outcome: + "run the highway", "/highway", "10-lane highway", "keep N lanes full", + "keep re-feeding lanes as they drain", "drive this for me in parallel", "do + them all and monitor", "keep the lanes full", "work the whole list in + parallel until done". NOT for a one-shot batch that launches once and + drains together — use goal-batch. NOT for bounded edits that each end in + their own PR — use mass-change. Requires git, tmux, the amplifier CLI on + PATH, and the goalify and monitor skills. version: 1.0.0 user-invocable: true shortcut: highway argument-hint: "" model_role: general +visibility: + summary: "Drive many parallel /goal lanes continuously toward an outcome — refill on drain, verify by evidence, close per completion intent." # Deliberately NOT `context: fork`: the approval gate, mid-flight steering, and # conflict questions must happen in THIS conversation across many turns. A fork # is call-and-return; a highway manager is a standing conversation partner. @@ -36,6 +37,16 @@ capacity. Never wait for a batch. Continuously decide what the best available use of each unit is." You are doing agent capacity management — closer to OS scheduling than to a task list. +## First run (~15 minutes) + +New to the highway? Do a throwaway run first — a scratch repo you can break, +**width 2**. Invoke `/highway` with a small outcome + that repo + "width 2"; +approve at the Phase 3 gate (nothing launches before you say go); then watch +**gate → 2 lanes → watchdog → merges**, refilling until the queue drains. Stop +everything with `tmux -L hw kill-session -t hw-watchdog__` (plus any +`hw____*` lane sessions), then delete `BATCH_DIR`. Full command sequence +and expected output: `examples/first-run.md`. + ## You are the Highway Manager — a strategist, not an intake clerk You take the user's **outcome/goal**, their **constraints** (time, authority, @@ -78,10 +89,12 @@ repo** (a real run left `.amplifier/bin/` behind as untracked pollution). | `launch_lane.sh BATCH_DIR LANE REPO GOAL [BASE_REF]` | Worktree + branch + tmux + `/goal` session, idempotent; the ONLY writer of `manifest.tsv` | Hand-written manifests diverged on column count and broke a real batch | | `verify_lane.sh BATCH_DIR LANE` | Git-facts probe for one landed lane (DONE.json, ahead-count, three-dot diffstat, uncommitted work) | "Ground truth from git and the filesystem, not from what any session said about itself" | | `highway_watchdog.sh BATCH_DIR WIDTH SESSION_ID [INTERVAL] [MAX_HOURS]` | Detached tmux loop that re-wakes THIS session (`amplifier run --resume`) on lane-end / under-width / stale heartbeat | The highway once froze overnight because the manager stopped monitoring the moment it reported status | +| `infra_ledger.sh BATCH_DIR add TYPE ID DESTROY_CMD...` / `infra_ledger.sh BATCH_DIR sweep` | Records any infrastructure a lane OR the manager stands up (DTU, gitea instance, container, service, background process) into `infra.tsv` at creation, each with its teardown command; `sweep` runs those commands and exits non-zero until nothing is left standing | A run closed with a DTU and a gitea container still live — nothing the highway stands up should outlive it (Rule 14) | State lives in `BATCH_DIR` (create one per highway, e.g. `~/dev/hw-`): -`manifest.tsv` (scripts write), `HIGHWAY.md` (you write), `lanes/` (worktrees), -`.manager-heartbeat`, `wake-needed`, `watchdog.log`. +`manifest.tsv` (scripts write), `HIGHWAY.md` (you write), `goals/` (pre-composed +goal files), `lanes/` (worktrees), `.width` (authoritative width), `infra.tsv` +(the infra ledger), `.manager-heartbeat`, `wake-needed`, `watchdog.log`. ## Phase 1 — Intake @@ -121,8 +134,20 @@ strategic value toward the outcome — not arrival order. Mark investment/speculative candidates (recon, spikes, de-risking) to backfill idle capacity later. +**Pre-compose the whole queue now.** `load_skill("goalify")` **inline — never +delegated** (goalify reads the live transcript; a sub-agent cannot) and write a +goal file per queued item up front into `BATCH_DIR/goals/.md`, each with a +disjunctive exit and per-item residuals. This makes every later refill a bare +`launch_lane.sh` call, never compose-then-launch (idle capacity is the enemy — +Phase 5 invariant); an item arriving at weave-in gets its goal file composed +when it enters the queue, not at refill. Each goal file MUST instruct its lane to +register any infrastructure it stands up — DTU, gitea instance, container, +service, background process — with `infra_ledger.sh add +` at creation (Rule 14). + **Success criteria**: a priority queue in `HIGHWAY.md` with a one-line -rationale per item tied to the outcome/constraints. +rationale per item tied to the outcome/constraints, and a pre-composed goal file +in `BATCH_DIR/goals/` for every item in it. ## Phase 3 — Approval gate `[human]` @@ -136,11 +161,10 @@ approval from enthusiasm or silence. ## Phase 4 — Saturate -For each lane in the first wave: -1. `load_skill("goalify")` **inline — never delegated** (goalify reads the - live transcript; a sub-agent cannot) and compose the lane's stop-condition - with a disjunctive exit and per-item residuals. -2. `launch_lane.sh BATCH_DIR [base]`. +For each lane in the first wave, launch from the goal file Phase 2 already +pre-composed in `BATCH_DIR/goals/` — no goalify here: + +- `launch_lane.sh BATCH_DIR BATCH_DIR/goals/.md [base]`. Then start the watchdog. **``, ``, `` below are documentation placeholders — substitute literal values; they do not exist as @@ -150,17 +174,27 @@ not depend on context: ```bash printf '%s\n' "" > /.session-id +printf '%s\n' "" > /.width # authoritative width — the single + # source of truth the scripts read touch /.manager-heartbeat # BEFORE the watchdog starts, so it never # sees an absent heartbeat and wakes a # concurrent instance mid-saturation BATCH=$(printf '%s' "$(basename )" | tr -c 'A-Za-z0-9_-' '_') # same sanitization the scripts use -tmux new-session -d -s "hw-watchdog__${BATCH}" \ +tmux -L hw new-session -d -s "hw-watchdog__${BATCH}" \ "/scripts/highway_watchdog.sh 300 12 2>&1 | tee -a /watchdog.log" ``` Then touch `/.manager-heartbeat` again at the start of every Phase 5 cycle (step 1) and after each merge, so the watchdog defers while your turn is active and only takes over once you have genuinely gone idle. +**Width, the tmux socket, and escalation.** `/.width` is the single +source of truth for the target lane count (the scripts read it there); changing +width mid-run is an explicit act — **edit `/.width` AND log it in the +weave-in log**, nothing else moves width. Every highway tmux command runs on the +socket `HIGHWAY_TMUX_SOCKET` (default `hw`) — hence the `-L hw` above and on the +Phase 7 kill. A wake whose prompt begins `HIGHWAY ESCALATION` means go straight +to Phase 6 and lead with `NEEDS YOU:`; the watchdog stays alive throughout. + Verify launch: run `highway_status.sh` once — every lane LIVE past its first LLM call (log growing), watchdog LIVE. Then build the **todo lane board** (see below). @@ -185,9 +219,11 @@ invariant exists to prevent.) 2. Get READY (count of ready, unblocked items) from the work queue. 3. **Run `highway_status.sh BATCH_DIR WIDTH READY` and paste its output.** 4. **If `DEFICIT>0`: refill FIRST** — before merging, before reporting, before - anything. Pick the top-priority ready items (strategist's choice), goalify - each inline, `launch_lane.sh` each. Under-width with ready work is allowed - only with an explicit one-line justification in the transcript that cycle. + anything. Pick the top-priority ready items (strategist's choice) and + `launch_lane.sh` each from its pre-composed `BATCH_DIR/goals/` file — refill + is a bare launch, never a compose-then-launch. Under-width with ready work is + allowed only with an explicit one-line justification in the transcript that + cycle. 5. **Width first, then merge — never the reverse.** Step 4 (restore width) always precedes this. For each ENDED lane: `verify_lane.sh`, then YOUR own artifact check, then `merge --no-ff` from the main checkout, resolve the item @@ -201,8 +237,10 @@ invariant exists to prevent.) Repair small defects in place or file the honest negative — never let one straggler block the others. 6. Strategize: process anything new (weave-in log: now / queued / declined, - with reasons), re-prioritize, handle STALLED and ENDED-NO-DONE lanes - (inspect the lane log tail; relaunch or reassign). + with reasons) — and the instant you queue an incoming item, goalify it inline + and write its `BATCH_DIR/goals/` file so a later refill stays a bare launch. + Re-prioritize, handle STALLED and ENDED-NO-DONE lanes (inspect the lane log + tail; relaunch or reassign). 7. Update the **todo lane board** and rewrite `HIGHWAY.md`. Regenerate its Landed section from git ground truth — `scripts/landed_from_git.sh [base]` — so the Operating Picture can never drift from what actually merged @@ -271,8 +309,12 @@ deadline reached, the user's release given, or (for *achieve-and-close*) the outcome verified with nothing pending. When you close: final Phase 5 pass; merge -or honestly disposition every open lane; kill the watchdog by the exact -name `highway_status.sh` reports (`tmux kill-session -t `). **Archive +or honestly disposition every open lane; then **run `infra_ledger.sh BATCH_DIR +sweep` and do not treat the highway as closed until it exits clean** — it tears +down every DTU, gitea instance, container, service, and background process the +run ledgered, whether a lane or the manager stood it up (Rule 14). Kill the +watchdog by the exact name `highway_status.sh` reports +(`tmux -L hw kill-session -t `). **Archive the per-lane evidence BEFORE pruning** — pruning the lane dirs otherwise deletes `lane.log` and the markers with them: ```bash @@ -286,6 +328,7 @@ status, landed list from `landed_from_git.sh`, residuals with named reasons); report with `DONE:` or `GAVE UP:` leading. **Success criteria**: no `hw__` tmux sessions, no stray worktrees/branches, +`infra_ledger.sh BATCH_DIR sweep` exits clean (nothing ledgered still standing), final report matches git facts. ## Rules — each bought with a documented failure @@ -320,8 +363,16 @@ final report matches git facts. live shared services are read-only to lanes. 13. **If `$ARGUMENTS` is empty, ask** — do not invent an outcome. (The fork sibling of this failure killed a real goal-batch invocation silently.) +14. **Infrastructure is ledgered at creation and swept at close — nothing the + highway stands up outlives it.** Any DTU, gitea instance, container, + service, or background process a lane OR the manager stands up is recorded + with `infra_ledger.sh BATCH_DIR add ...` at creation, and Phase 7 does not + close until `infra_ledger.sh BATCH_DIR sweep` exits clean. (A run closed + leaving a DTU and a gitea container running.) + +## Known limits (still not built) -## Known limits / v2 mechanisms (noted, not built) +(Width pin, escalation, and the infra ledger are now implemented above.) - A hook that injects `highway_status.sh` output into every turn (the way the todo reminder does) would make drift structurally impossible to ignore — diff --git a/amplifier_app_cli/data/skills/ten-lane-highway/examples/first-run.md b/amplifier_app_cli/data/skills/ten-lane-highway/examples/first-run.md new file mode 100644 index 0000000..0ccb7eb --- /dev/null +++ b/amplifier_app_cli/data/skills/ten-lane-highway/examples/first-run.md @@ -0,0 +1,75 @@ +# First run — a 15-minute throwaway highway + +The point of a first run is to see the whole loop — **gate → lanes → watchdog → +merges → refill → close** — on work you do not care about, at width 2, before +you trust it with something real. Nothing here is special to width 2; it is just +small enough to watch every moving part. + +## Before you start + +- A **throwaway git repo** you can break (a scratch clone, or `git init` a + sandbox with a couple of trivial files). A remote is optional — lanes commit + locally and merge fine with no network. +- `git`, `tmux`, and the `amplifier` CLI on PATH. +- Two or three tiny, independent chores in that repo (e.g. "add a docstring to + each of files A, B, C") — enough to fill two lanes and leave one to refill. + +## The sequence + +1. **Invoke** in the interactive TUI: + + ``` + /highway reach a green throwaway: do the three chores in , + width 2, achieve-and-close + ``` + + In a headless one-shot, name the skill in prose instead of the bare slash + token: `amplifier run "Use the ten-lane-highway skill and drive: "`. + +2. **Intake + strategize (Phases 1–2).** The manager writes `BATCH_DIR` + (e.g. `~/dev/hw-firstrun`), a `HIGHWAY.md`, pins `BATCH_DIR/.width` to `2`, + and **pre-composes a goal file per queued item** into `BATCH_DIR/goals/`. + +3. **Approve at the gate (Phase 3).** You will see one screen: outcome, width 2, + the first wave (lane → repo → item), the priority rationale, watch cadence. + Reply `go`. Nothing launched before this. + +4. **Saturate (Phase 4).** Two lanes come up — each a worktree + branch + tmux + session running `/goal` — then the **watchdog** starts on the `hw` tmux + socket. Confirm with the status instrument: both lanes LIVE, watchdog LIVE, + `DEFICIT=0`. + +5. **Watch it run (Phase 5).** As a lane finishes, the manager verifies it from + git facts, merges `--no-ff`, tears the lane down, and **refills the instant a + slot opens** from the pre-composed queue — a bare `launch_lane.sh` call, no + re-goalify. The todo lane board is your live dashboard. + +6. **Close (Phase 7).** With `achieve-and-close` and nothing pending, the + manager does a final full-suite sweep, runs the infra-ledger sweep + (`infra_ledger.sh BATCH_DIR sweep` must exit clean), archives per-lane + evidence, prunes worktrees/branches, kills the watchdog, and reports + `DONE:`. + +## What you should observe + +- The **gate blocks** until you say go — enthusiasm and silence are not consent. +- Width holds at 2 while work remains: a drained lane refills immediately, not + after the whole wave lands. +- Every merge is proven by the merged lane's **own tests**, with the full suite + on a cadence and at close — never a lane's self-report. +- `HIGHWAY.md` and the todo board agree at the end of every cycle. + +## Stop everything (panic button) + +```bash +# BATCH is the sanitized batch name (basename of BATCH_DIR, non-alnum → _) +tmux -L hw kill-session -t "hw-watchdog__${BATCH}" # the watchdog +tmux -L hw list-sessions -F '#{session_name}' \ + | grep "^hw__${BATCH}__" | xargs -r -n1 tmux -L hw kill-session -t # lanes +infra_ledger.sh "$BATCH_DIR" sweep # tear down anything lanes stood up +rm -rf "$BATCH_DIR" # worktrees, goals, HIGHWAY.md, logs +``` + +All highway tmux commands use `-L hw` (the `HIGHWAY_TMUX_SOCKET`, default `hw`), +so a stray `tmux kill-server` on the default socket never touches the highway, +and this never touches your other tmux work. diff --git a/amplifier_app_cli/data/skills/ten-lane-highway/scripts/highway_status.sh b/amplifier_app_cli/data/skills/ten-lane-highway/scripts/highway_status.sh index f33fe1c..379d1bc 100755 --- a/amplifier_app_cli/data/skills/ten-lane-highway/scripts/highway_status.sh +++ b/amplifier_app_cli/data/skills/ten-lane-highway/scripts/highway_status.sh @@ -19,6 +19,18 @@ STALL_SECS=${HIGHWAY_STALL_SECS:-900} # HIGHWAY_JSON=1 -> emit ONE compact JSON line and nothing else (for the eval's # M1 sampler daemon: `HIGHWAY_JSON=1 highway_status.sh ... >> deficit.jsonl`). JSON=${HIGHWAY_JSON:-0} +# WIDTH PIN: if BATCH_DIR/.width holds a single integer, it overrides the +# positional WIDTH arg. width_source records which value won, so a reader can +# tell a pinned width from an argument-supplied one. +WIDTH_FILE="$BATCH_DIR/.width" +width_source=arg +if [ -f "$WIDTH_FILE" ]; then + _wpin=$(head -n1 "$WIDTH_FILE" | tr -d '[:space:]') + if printf '%s' "$_wpin" | grep -qE '^[0-9]+$'; then + WIDTH=$_wpin + width_source=file + fi +fi # Same sanitization as launch_lane.sh — the watchdog name must match byte-for-byte. # (stat -c %Y below is GNU/Linux; adjust for macOS if this ever travels.) BATCH=$(printf '%s' "$(basename "$BATCH_DIR")" | tr -c 'A-Za-z0-9_-' '_') @@ -31,7 +43,7 @@ live=0; ended=0; done_n=0; stalled=0; gone=0 while IFS=$'\t' read -r lane wt branch base tmuxn goal log ts; do [ "$lane" = "lane" ] && continue - if tmux has-session -t "$tmuxn" 2>/dev/null; then st=LIVE; else st=ENDED; fi + if tmux -L "${HIGHWAY_TMUX_SOCKET:-hw}" has-session -t "$tmuxn" 2>/dev/null; then st=LIVE; else st=ENDED; fi age="-"; ahead="-"; dj="-"; wt_present=yes if [ -d "$wt" ]; then @@ -44,7 +56,7 @@ while IFS=$'\t' read -r lane wt branch base tmuxn goal log ts; do flag="" if [ "$st" = "LIVE" ]; then - # A live tmux session counts as a live lane even if its worktree vanished — + # A live tmux-session counts as a live lane even if its worktree vanished — # that is an anomaly to flag, not a lane to ignore. live=$((live+1)) if [ "$wt_present" = "no" ]; then flag="NO-WORKTREE"; fi @@ -64,19 +76,19 @@ while IFS=$'\t' read -r lane wt branch base tmuxn goal log ts; do done < "$MANIFEST" WD="hw-watchdog__${BATCH}" -if tmux has-session -t "$WD" 2>/dev/null; then wd_st=LIVE; else wd_st=DEAD; fi +if tmux -L "${HIGHWAY_TMUX_SOCKET:-hw}" has-session -t "$WD" 2>/dev/null; then wd_st=LIVE; else wd_st=DEAD; fi open=$(( WIDTH - live )); if [ "$open" -lt 0 ]; then open=0; fi if [ "$READY" -lt "$open" ]; then deficit=$READY; else deficit=$open; fi if [ "$JSON" = 1 ]; then - printf '{"ts":"%s","batch":"%s","live":%d,"ended":%d,"done_marker":%d,"stalled":%d,"gone":%d,"width":%d,"ready":%d,"deficit":%d,"watchdog":"%s"}\n' \ - "$(date -u +%FT%TZ)" "$BATCH" "$live" "$ended" "$done_n" "$stalled" "$gone" "$WIDTH" "$READY" "$deficit" "$wd_st" + printf '{"ts":"%s","batch":"%s","live":%d,"ended":%d,"done_marker":%d,"stalled":%d,"gone":%d,"width":%d,"width_source":"%s","ready":%d,"deficit":%d,"watchdog":"%s"}\n' \ + "$(date -u +%FT%TZ)" "$BATCH" "$live" "$ended" "$done_n" "$stalled" "$gone" "$WIDTH" "$width_source" "$READY" "$deficit" "$wd_st" exit 0 fi echo -echo "SUMMARY batch=$BATCH live=$live ended=$ended done_marker=$done_n stalled=$stalled gone=$gone width=$WIDTH ready=$READY watchdog=$wd_st" +echo "SUMMARY batch=$BATCH live=$live ended=$ended done_marker=$done_n stalled=$stalled gone=$gone width=$WIDTH width_source=$width_source ready=$READY watchdog=$wd_st" echo "DEFICIT=$deficit" if [ "$deficit" -gt 0 ]; then echo "ACTION: launch $deficit lane(s) NOW - refill before anything else." diff --git a/amplifier_app_cli/data/skills/ten-lane-highway/scripts/highway_watchdog.sh b/amplifier_app_cli/data/skills/ten-lane-highway/scripts/highway_watchdog.sh index 71e4df3..246e076 100755 --- a/amplifier_app_cli/data/skills/ten-lane-highway/scripts/highway_watchdog.sh +++ b/amplifier_app_cli/data/skills/ten-lane-highway/scripts/highway_watchdog.sh @@ -4,8 +4,8 @@ # Keeps watching lanes while the manager's turn is ended (e.g. it stopped to # talk to the human). Wakes the orchestrator session when attention is needed. # -# Run it DETACHED in its own tmux session (the launcher is the orchestrator): -# tmux new-session -d -s "hw-watchdog__" \ +# Run it DETACHED in its own tmux-session (the launcher is the orchestrator): +# tmux -L "${HIGHWAY_TMUX_SOCKET:-hw}" new-session -d -s "hw-watchdog__" \ # "/scripts/highway_watchdog.sh BATCH_DIR WIDTH SESSION_ID [INTERVAL] [MAX_HOURS]" # # Wake triggers: a lane ended since last poll | live < WIDTH | manager heartbeat stale. @@ -34,6 +34,33 @@ WAKE_GAP=${HIGHWAY_WAKE_GAP:-180} ACTIVE_WINDOW=${HIGHWAY_ACTIVE_WINDOW:-120} # stat -c %Y below is GNU/Linux; adjust for macOS if this ever travels. +# WIDTH PIN: BATCH_DIR/.width (a single integer) overrides the positional WIDTH. +# resolve_width re-reads it EVERY poll, so a mid-run width change is picked up +# live. width_source records which value is in force ('file' or 'arg'). +WIDTH_ARG=$WIDTH +WIDTH_FILE="$BATCH_DIR/.width" +width_source=arg +resolve_width() { + width_source=arg + WIDTH=$WIDTH_ARG + if [ -f "$WIDTH_FILE" ]; then + local w + w=$(head -n1 "$WIDTH_FILE" | tr -d '[:space:]') + if printf '%s' "$w" | grep -qE '^[0-9]+$'; then + WIDTH=$w + width_source=file + fi + fi +} + +# ESCALATION LADDER: after this many consecutive polls where we are under width +# AND live is not recovering (did not increase vs the previous poll), touch +# BATCH_DIR/escalation-needed and switch the wake prompt to the escalation form. +ESCALATE_AFTER=${HIGHWAY_ESCALATE_AFTER:-3} +prev_live=-1 # -1 so the very first poll never looks like a non-increase +ineffective=0 # consecutive ineffective (under-width, non-recovering) polls +escalate=0 # 1 while the ineffective count is at/over the threshold + deadline=$(( $(date +%s) + MAX_HOURS * 3600 )) START_TS=$(date +%s) # Grace after watchdog start during which an ABSENT heartbeat means "manager @@ -48,7 +75,7 @@ live_lanes() { local n=0 lane wt branch base tmuxn rest while IFS=$'\t' read -r lane wt branch base tmuxn rest; do [ "$lane" = "lane" ] && continue - tmux has-session -t "$tmuxn" 2>/dev/null && n=$((n+1)) + tmux -L "${HIGHWAY_TMUX_SOCKET:-hw}" has-session -t "$tmuxn" 2>/dev/null && n=$((n+1)) done < "$MANIFEST" echo "$n" } @@ -57,22 +84,30 @@ ended_list() { local lane wt branch base tmuxn rest while IFS=$'\t' read -r lane wt branch base tmuxn rest; do [ "$lane" = "lane" ] && continue - tmux has-session -t "$tmuxn" 2>/dev/null || echo "$lane" + tmux -L "${HIGHWAY_TMUX_SOCKET:-hw}" has-session -t "$tmuxn" 2>/dev/null || echo "$lane" done < "$MANIFEST" } wake() { - local reason="$1" now + local reason="$1" now prompt now=$(date +%s) if [ $(( now - last_wake )) -lt "$WAKE_GAP" ]; then log "suppress wake (within ${WAKE_GAP}s gap): $reason" return 0 fi printf '%s\t%s\n' "$(date -u +%FT%TZ)" "$reason" >> "$BATCH_DIR/wake-needed" - log "WAKE: $reason" - if amplifier run --resume "$SESSION_ID" --output-format json \ - "HIGHWAY WAKE (watchdog): $reason. Run one steady-state cycle now: highway_status.sh FIRST, verify+merge any landed lane, refill to width, update HIGHWAY.md and the todo lane board, touch the heartbeat, clear wake-needed. Do not end the turn while DEFICIT>0 or the watchdog is dead." \ - >> "$LOGF" 2>&1; then + if [ "${escalate:-0}" = 1 ]; then + # Escalation form: refill has stayed ineffective across multiple polls, so + # the prompt leads with a distinct banner carrying the ineffective-wake + # count, live, and width, and directs a root-cause look rather than a + # reflexive relaunch. + log "ESCALATION WAKE (ineffective=$ineffective live=$live width=$WIDTH): $reason" + prompt="HIGHWAY ESCALATION (watchdog): refill has been ineffective for $ineffective consecutive poll(s) - live=$live is still under width=$WIDTH. $reason. Do NOT just relaunch blindly: run highway_status.sh FIRST, then find WHY lanes are not coming up to width (launch failures, exhausted ready queue, host/resource limits, stuck merges) and fix that, then refill. Do not end the turn while DEFICIT>0 or the watchdog is dead." + else + log "WAKE: $reason" + prompt="HIGHWAY WAKE (watchdog): $reason. Run one steady-state cycle now: highway_status.sh FIRST, verify+merge any landed lane, refill to width, update HIGHWAY.md and the todo lane board, touch the heartbeat, clear wake-needed. Do not end the turn while DEFICIT>0 or the watchdog is dead." + fi + if amplifier run --resume "$SESSION_ID" --output-format json "$prompt" >> "$LOGF" 2>&1; then log "wake delivered (resume ok)" last_wake=$now else @@ -96,6 +131,7 @@ while :; do [ -f "$MANIFEST" ] || { log "no manifest yet at $MANIFEST"; continue; } + resolve_width # WIDTH PIN: re-read BATCH_DIR/.width on every poll live=$(live_lanes) ended_now=$(ended_list | sort) ended_prev=$(sort "$STATE" 2>/dev/null || true) @@ -108,7 +144,7 @@ while :; do hb_age=-1 # heartbeat not created yet fi wd_uptime=$(( $(date +%s) - START_TS )) - log "poll live=$live width=$WIDTH new_ended='${new_ended}' hb_age=${hb_age}s uptime=${wd_uptime}s" + log "poll live=$live width=$WIDTH width_source=$width_source new_ended='${new_ended}' hb_age=${hb_age}s uptime=${wd_uptime}s" # Manager's turn is active -> it handles refill/merge inline; do NOT wake (a # second concurrent --resume instance was the proof-run-02/trial-01 race). @@ -123,6 +159,25 @@ while :; do # treat as stale so the safety-net wakes below still fire. [ "$hb_age" -lt 0 ] && hb_age=$(( HB_MAX + 1 )) + # ESCALATION LADDER: count consecutive polls where live < width AND live did + # not increase vs the previous poll; reset the moment live increases or the + # deficit clears. Computed on non-deferred polls only (a deferred poll means + # the manager is active and refilling inline, so it is not an ineffective wake). + if [ "$live" -ge "$WIDTH" ]; then + ineffective=0 # deficit cleared + elif [ "$live" -gt "$prev_live" ]; then + ineffective=0 # live increased since the previous poll + else + ineffective=$(( ineffective + 1 )) + fi + prev_live=$live + escalate=0 + if [ "$ineffective" -ge "$ESCALATE_AFTER" ]; then + escalate=1 + touch "$BATCH_DIR/escalation-needed" + log "ESCALATION: ineffective=$ineffective >= ${ESCALATE_AFTER} (live=$live width=$WIDTH) - marker touched" + fi + if [ -n "${new_ended// /}" ]; then wake "lane(s) ended: ${new_ended}"; continue; fi if [ "$live" -lt "$WIDTH" ]; then wake "under width: live=$live < width=$WIDTH"; continue; fi if [ "$hb_age" -gt "$HB_MAX" ] && [ "$live" -gt 0 ]; then diff --git a/amplifier_app_cli/data/skills/ten-lane-highway/scripts/infra_ledger.sh b/amplifier_app_cli/data/skills/ten-lane-highway/scripts/infra_ledger.sh new file mode 100755 index 0000000..56802dc --- /dev/null +++ b/amplifier_app_cli/data/skills/ten-lane-highway/scripts/infra_ledger.sh @@ -0,0 +1,87 @@ +#!/usr/bin/env bash +# infra_ledger.sh — a durable ledger of provisioned infrastructure, plus a sweep +# that tears it down. So a highway run can never leak a worktree, container, or +# temp dir it created: every provisioned thing is recorded here with the exact +# command that destroys it, and one `sweep` reclaims them all. +# +# Usage: +# infra_ledger.sh BATCH_DIR add TYPE ID DESTROY_CMD... +# infra_ledger.sh BATCH_DIR list +# infra_ledger.sh BATCH_DIR sweep +# +# Rows live in BATCH_DIR/infra.tsv, one per line, tab-separated: +# ts type id status destroy_cmd +# THIS script is the ONLY writer of infra.tsv — never hand-edit that file. +# +# add Append one row with status=open. DESTROY_CMD... is the (possibly +# multi-word) command that reclaims the resource; it is stored verbatim +# and later run via `bash -c`. +# list Print the open rows; always exit 0. +# sweep Run each OPEN row's destroy_cmd. On rc=0 mark it swept; otherwise +# leave it open and print the failure. Exit nonzero if any row is still +# open afterwards. Idempotent: already-swept rows are never re-run, so +# re-sweeping a fully-swept ledger runs nothing and exits 0. +# +# NOT -e: a failing destroy_cmd during sweep is an expected, handled outcome — +# it must not abort the whole sweep. +set -uo pipefail + +BATCH_DIR=${1:?BATCH_DIR required} +CMD=${2:?command required (add|list|sweep)} +LEDGER="$BATCH_DIR/infra.tsv" + +case "$CMD" in + add) + TYPE=${3:?TYPE required} + ID=${4:?ID required} + shift 4 + [ "$#" -ge 1 ] || { echo "ERROR: add requires DESTROY_CMD..." >&2; exit 1; } + DESTROY="$*" + # A row is one TSV line: tabs/newlines in a field would corrupt the ledger. + if [[ "$TYPE$ID$DESTROY" == *$'\t'* || "$TYPE$ID$DESTROY" == *$'\n'* ]]; then + echo "ERROR: TYPE/ID/DESTROY_CMD must not contain tabs or newlines" >&2; exit 1 + fi + mkdir -p "$BATCH_DIR" + printf '%s\t%s\t%s\t%s\t%s\n' "$(date -u +%FT%TZ)" "$TYPE" "$ID" "open" "$DESTROY" >> "$LEDGER" + echo "ADDED: type=$TYPE id=$ID status=open destroy='$DESTROY'" + ;; + + list) + [ -f "$LEDGER" ] || exit 0 + awk -F'\t' '$4=="open"' "$LEDGER" || true + exit 0 + ;; + + sweep) + [ -f "$LEDGER" ] || { echo "SWEEP: no ledger ($LEDGER) - nothing to do"; exit 0; } + tmp=$(mktemp "$BATCH_DIR/.infra.XXXXXX") + remaining=0 + while IFS=$'\t' read -r ts type id status destroy; do + [ -z "${ts:-}" ] && continue # skip blank lines + if [ "$status" = "open" ]; then + echo ">> sweeping type=$type id=$id: $destroy" + if bash -c "$destroy"; then + status=swept + echo " swept ok" + else + rc=$? + echo " FAILED (rc=$rc) type=$type id=$id: $destroy" >&2 + remaining=$(( remaining + 1 )) + fi + fi + printf '%s\t%s\t%s\t%s\t%s\n' "$ts" "$type" "$id" "$status" "$destroy" >> "$tmp" + done < "$LEDGER" + mv "$tmp" "$LEDGER" + if [ "$remaining" -gt 0 ]; then + echo "SWEEP: $remaining row(s) still open (destroy_cmd failed) - re-run after fixing" + exit 1 + fi + echo "SWEEP: all rows swept" + exit 0 + ;; + + *) + echo "ERROR: unknown command '$CMD' (want: add|list|sweep)" >&2 + exit 1 + ;; +esac diff --git a/amplifier_app_cli/data/skills/ten-lane-highway/scripts/launch_lane.sh b/amplifier_app_cli/data/skills/ten-lane-highway/scripts/launch_lane.sh index 9509314..90f3e6d 100755 --- a/amplifier_app_cli/data/skills/ten-lane-highway/scripts/launch_lane.sh +++ b/amplifier_app_cli/data/skills/ten-lane-highway/scripts/launch_lane.sh @@ -1,15 +1,15 @@ #!/usr/bin/env bash -# launch_lane.sh — start ONE highway lane: worktree + branch + tmux + autonomous /goal session. +# launch_lane.sh — start ONE highway lane: worktree + branch + tmux-session + autonomous /goal session. # # Usage: launch_lane.sh BATCH_DIR LANE REPO_PATH GOAL_FILE [BASE_REF] # BATCH_DIR directory holding this highway's state (manifest, logs, lanes/) -# LANE kebab-case lane name (becomes branch lane/ and tmux hw____) +# LANE kebab-case lane name (becomes branch lane/ and tmux-session hw____) # REPO_PATH path to the repo this lane owns # GOAL_FILE path to the goalify-composed goal file for this lane # BASE_REF ref to branch from (default: main) # # This script is the ONLY writer of manifest.tsv. Never hand-write the manifest. -# Idempotent: re-running skips an existing worktree / running tmux session. +# Idempotent: re-running skips an existing worktree / running tmux-session. set -euo pipefail BATCH_DIR=${1:?BATCH_DIR required} @@ -18,7 +18,7 @@ REPO=${3:?REPO_PATH required} GOAL=${4:?GOAL_FILE required} BASE_REF=${5:-main} -# tmux treats '.' and ':' specially in -t targets: a name built from a raw +# tmux(1) treats '.' and ':' specially in -t targets: a name built from a raw # basename may not round-trip through later has-session checks. Sanitize the # batch identifier and REQUIRE clean lane names, so the name we create is # byte-identical to the name every later check looks up. @@ -78,10 +78,10 @@ merge time. $DONE_MARKER EOF -if tmux has-session -t "$TMUX_NAME" 2>/dev/null; then +if tmux -L "${HIGHWAY_TMUX_SOCKET:-hw}" has-session -t "$TMUX_NAME" 2>/dev/null; then echo "SKIP: $TMUX_NAME already running" else - tmux new-session -d -s "$TMUX_NAME" -c "$WT" \ + tmux -L "${HIGHWAY_TMUX_SOCKET:-hw}" new-session -d -s "$TMUX_NAME" -c "$WT" \ "amplifier run '/goal @GOAL.md' 2>&1 | tee '$LOG'" echo "LAUNCHED: $TMUX_NAME -> $WT ($BRANCH @ ${BASE_SHA:0:8})" fi diff --git a/amplifier_app_cli/data/skills/ten-lane-highway/scripts/verify_lane.sh b/amplifier_app_cli/data/skills/ten-lane-highway/scripts/verify_lane.sh index 216cc1d..88a62d8 100755 --- a/amplifier_app_cli/data/skills/ten-lane-highway/scripts/verify_lane.sh +++ b/amplifier_app_cli/data/skills/ten-lane-highway/scripts/verify_lane.sh @@ -18,7 +18,7 @@ row=$(awk -F'\t' -v l="$LANE" 'NR>1 && $1==l' "$MANIFEST" || true) IFS=$'\t' read -r lane wt branch base tmuxn goal log ts <<< "$row" echo "== lane=$lane branch=$branch base=${base:0:8} wt=$wt" -if tmux has-session -t "$tmuxn" 2>/dev/null; then +if tmux -L "${HIGHWAY_TMUX_SOCKET:-hw}" has-session -t "$tmuxn" 2>/dev/null; then echo "TMUX: LIVE (still running - do NOT merge yet)" else echo "TMUX: ended" @@ -37,6 +37,23 @@ mb=$(git -C "$wt" merge-base "$base" HEAD) echo "-- diffstat merge-base..HEAD (three-dot truth; two-dot lies after base moves):" git -C "$wt" diff --stat "$mb..HEAD" | tail -15 +# TEST-EDIT FLAG: from the three-dot name list, always report how many touched +# paths look like tests, so the orchestrator can weigh a lane's own "I added +# tests" claim against git ground truth. A path counts as a test edit when it +# contains a tests?/ directory component, or a test_/_test filename marker. +names=$(git -C "$wt" diff --name-only "$base...HEAD" || true) +test_edits="" +if [ -n "$names" ]; then + test_edits=$(printf '%s\n' "$names" | grep -E '(^|/)tests?/|test_|_test' || true) +fi +if [ -n "$test_edits" ]; then + n=$(printf '%s\n' "$test_edits" | grep -c .) + echo "TEST-EDITS: $n file(s)" + printf '%s\n' "$test_edits" | sed 's/^/ /' +else + echo "TEST-EDITS: 0 file(s)" +fi + echo "-- last 3 commits:" git -C "$wt" log --oneline -3