Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 43 additions & 9 deletions amplifier_app_cli/data/skills/ten-lane-highway/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,12 +89,35 @@ repo** (a real run left `.amplifier/bin/` behind as untracked pollution).
| `launch_lane.sh BATCH_DIR LANE REPO GOAL [BASE_REF]` | Worktree + branch + tmux + `/goal` session, idempotent; the ONLY writer of `manifest.tsv` | Hand-written manifests diverged on column count and broke a real batch |
| `verify_lane.sh BATCH_DIR LANE` | Git-facts probe for one landed lane (DONE.json, ahead-count, three-dot diffstat, uncommitted work) | "Ground truth from git and the filesystem, not from what any session said about itself" |
| `highway_watchdog.sh BATCH_DIR WIDTH SESSION_ID [INTERVAL] [MAX_HOURS]` | Detached tmux loop that re-wakes THIS session (`amplifier run --resume`) on lane-end / under-width / stale heartbeat | The highway once froze overnight because the manager stopped monitoring the moment it reported status |
| `infra_ledger.sh BATCH_DIR add TYPE ID DESTROY_CMD...` / `infra_ledger.sh BATCH_DIR sweep` | Records any infrastructure a lane OR the manager stands up (DTU, gitea instance, container, service, background process) into `infra.tsv` at creation, each with its teardown command; `sweep` runs those commands and exits non-zero until nothing is left standing | A run closed with a DTU and a gitea container still live — nothing the highway stands up should outlive it (Rule 14) |
| `infra_ledger.sh BATCH_DIR add TYPE ID DESTROY_CMD...` / `infra_ledger.sh BATCH_DIR sweep --all-owners` | Records any infrastructure a lane OR the manager stands up (DTU, gitea instance, container, service, background process) into `infra.tsv` at creation, each with its teardown command; `sweep` runs those commands and exits non-zero until nothing is left standing | A run closed with a DTU and a gitea container still live — nothing the highway stands up should outlive it (Rule 14) |

**`sweep` is the MANAGER's batch-close verb, never a lane's.** It runs EVERY
open row's destroy command, so one lane calling it destroys every other lane's
live infrastructure — that is not hypothetical: on 2026-09-02 a single foreign
`sweep` took lane l1's three DTUs and lane 161's three, 35 minutes into their
measurements. The script now **refuses with exit 3, having run nothing**, when
the open rows span more than one owner or any row is unattributable, so:

- **A lane tearing down its OWN rows uses the batch's lane-scoped teardown
tool** (`lane_teardown.sh BATCH_DIR <lane> teardown --yes`; omit `--yes` for a
dry run), which touches only the rows that lane claimed. A lane never calls
`sweep`.
- **The manager closing the batch passes `--all-owners`** — the deliberate
batch-close override. Every close instruction below says
`sweep --all-owners` for exactly this reason: a guard that deadlocks the
documented close is a regression, not a fix.

A destroy command for infrastructure that is **already gone** closes its row as
`swept:already-absent` — distinct from `swept`, because the sweep did not
perform that teardown. A REAL teardown failure still exits non-zero and leaves
the row open; the already-gone signature is deliberately narrow, so the signal
that a teardown genuinely failed is never lost.

State lives in `BATCH_DIR` (create one per highway, e.g. `~/dev/hw-<name>`):
`manifest.tsv` (scripts write), `HIGHWAY.md` (you write), `goals/` (pre-composed
goal files), `lanes/` (worktrees), `.width` (authoritative width), `infra.tsv`
(the infra ledger), `.manager-heartbeat`, `wake-needed`, `watchdog.log`.
(the infra ledger), `infra.owners.tsv` (which lane claimed which row),
`.manager-heartbeat`, `wake-needed`, `watchdog.log`.

## Phase 1 — Intake

Expand Down Expand Up @@ -143,7 +166,10 @@ Phase 5 invariant); an item arriving at weave-in gets its goal file composed
when it enters the queue, not at refill. Each goal file MUST instruct its lane to
register any infrastructure it stands up — DTU, gitea instance, container,
service, background process — with `infra_ledger.sh <BATCH_DIR> add <type> <id>
<destroy-cmd…>` at creation (Rule 14).
<destroy-cmd…>` at creation (Rule 14), and to tear down **only its own rows**
via the batch's lane-scoped teardown tool. A goal file must never tell a lane to
run `sweep`: that is the manager's batch-close verb and it destroys every other
lane's live infrastructure.

**Success criteria**: a priority queue in `HIGHWAY.md` with a one-line
rationale per item tied to the outcome/constraints, and a pre-composed goal file
Expand Down Expand Up @@ -310,9 +336,13 @@ outcome verified with nothing pending.

When you close: final Phase 5 pass; merge
or honestly disposition every open lane; then **run `infra_ledger.sh BATCH_DIR
sweep` and do not treat the highway as closed until it exits clean** — it tears
sweep --all-owners` and do not treat the highway as closed until it exits
clean** — it tears
down every DTU, gitea instance, container, service, and background process the
run ledgered, whether a lane or the manager stood it up (Rule 14). Kill the
run ledgered, whether a lane or the manager stood it up (Rule 14). `--all-owners`
is the manager's batch-close override; without it `sweep` refuses (exit 3) the
moment the open rows span more than one lane, which is the guard that stops a
lane from destroying its neighbours' infrastructure. Kill the
watchdog by the exact name `highway_status.sh` reports
(`tmux -L hw kill-session -t <wd_name>`). **Archive
the per-lane evidence BEFORE pruning** — pruning the lane dirs otherwise deletes
Expand All @@ -328,8 +358,8 @@ status, landed list from `landed_from_git.sh`, residuals with named reasons);
report with `DONE:` or `GAVE UP:` leading.

**Success criteria**: no `hw__` tmux sessions, no stray worktrees/branches,
`infra_ledger.sh BATCH_DIR sweep` exits clean (nothing ledgered still standing),
final report matches git facts.
`infra_ledger.sh BATCH_DIR sweep --all-owners` exits clean (nothing ledgered
still standing), final report matches git facts.

## Rules — each bought with a documented failure

Expand Down Expand Up @@ -367,8 +397,12 @@ final report matches git facts.
highway stands up outlives it.** Any DTU, gitea instance, container,
service, or background process a lane OR the manager stands up is recorded
with `infra_ledger.sh BATCH_DIR add ...` at creation, and Phase 7 does not
close until `infra_ledger.sh BATCH_DIR sweep` exits clean. (A run closed
leaving a DTU and a gitea container running.)
close until `infra_ledger.sh BATCH_DIR sweep --all-owners` exits clean.
(A run closed leaving a DTU and a gitea container running.) **`sweep` is
the manager's verb; a lane tears down only its own rows via the batch's
lane-scoped teardown tool.** (One foreign `sweep` destroyed two other
lanes' DTUs mid-measurement; `sweep` now refuses a multi-owner ledger with
exit 3 unless `--all-owners` is passed.)

## Known limits (still not built)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,32 @@
# Usage:
# infra_ledger.sh BATCH_DIR add TYPE ID DESTROY_CMD...
# infra_ledger.sh BATCH_DIR list
# infra_ledger.sh BATCH_DIR sweep
# infra_ledger.sh BATCH_DIR sweep [--all-owners]
#
# Rows live in BATCH_DIR/infra.tsv, one per line, tab-separated:
# ts <TAB> type <TAB> id <TAB> status <TAB> destroy_cmd
# THIS script is the ONLY writer of infra.tsv — never hand-edit that file.
# Rows are attributed to a lane by a sibling BATCH_DIR/infra.owners.tsv
# ts <TAB> id <TAB> lane
# written by the batch's lane-scoped teardown tool, not by this script.
#
# add Append one row with status=open. DESTROY_CMD... is the (possibly
# multi-word) command that reclaims the resource; it is stored verbatim
# and later run via `bash -c`.
# list Print the open rows; always exit 0.
# sweep Run each OPEN row's destroy_cmd. On rc=0 mark it swept; otherwise
# leave it open and print the failure. Exit nonzero if any row is still
# open afterwards. Idempotent: already-swept rows are never re-run, so
# sweep THE MANAGER'S BATCH-CLOSE VERB, NEVER A LANE'S. It runs EVERY open
# row's destroy_cmd, so a lane calling it destroys other lanes' live
# infrastructure. A lane tearing down its OWN rows uses the batch's
# lane-scoped teardown tool (lane_teardown.sh) instead.
# Refuses with exit 3, having run NOTHING, when the open rows span
# more than one owner or any row is unattributable; the manager
# closing the batch passes --all-owners to proceed anyway.
# Otherwise, per open row: rc=0 marks it `swept`; a rc!=0 whose output
# matches the NARROW already-gone signature marks it
# `swept:already-absent` (recorded distinctly — the row is closed but
# this sweep did not perform the teardown); any other failure leaves
# the row open and is printed. Exit nonzero if any row is still open
# afterwards. Idempotent: already-swept rows are never re-run, so
# re-sweeping a fully-swept ledger runs nothing and exits 0.
#
# NOT -e: a failing destroy_cmd during sweep is an expected, handled outcome —
Expand All @@ -30,6 +43,12 @@ BATCH_DIR=${1:?BATCH_DIR required}
CMD=${2:?command required (add|list|sweep)}
LEDGER="$BATCH_DIR/infra.tsv"

# Signatures meaning "the thing you asked me to destroy does not exist".
# Deliberately NARROW: a blanket exit-code amnesty would destroy the signal
# that a REAL teardown failed, which is the whole reason sweep checks rc
# (Rule 14: nothing the highway stands up should outlive it).
ALREADY_GONE_RE=${ALREADY_GONE_RE:-"environment not found|not found|no such|does not exist|doesn't exist|unknown (environment|container|project)"}

case "$CMD" in
add)
TYPE=${3:?TYPE required}
Expand All @@ -54,17 +73,60 @@ case "$CMD" in

sweep)
[ -f "$LEDGER" ] || { echo "SWEEP: no ledger ($LEDGER) - nothing to do"; exit 0; }

# ---- MULTI-LANE GUARD (model_performance-0rg) -------------------------
# sweep is the MANAGER's BATCH-CLOSE verb. Its only predicate used to be
# `status == open`, so the FIRST caller destroyed EVERY lane's live
# infrastructure. Observed 2026-09-02: one foreign sweep took lane l1's
# three DTUs and lane 161's three, 35 minutes into their measurements.
# A lane tearing down its OWN rows uses the lane-scoped tool:
# .amplifier/evaluation/tools/lane_teardown.sh BATCH_DIR LANE teardown
ALL_OWNERS=0
for a in "$@"; do [ "$a" = "--all-owners" ] && ALL_OWNERS=1; done
if [ "$ALL_OWNERS" != 1 ]; then
OWNERS_FILE="$BATCH_DIR/infra.owners.tsv"
owners=$(awk -F'\t' -v of="$OWNERS_FILE" '
BEGIN { while ((getline line < of) > 0) { split(line, f, "\t"); own[f[2]] = f[3] } }
$4 ~ /^open/ { print ($3 in own) ? own[$3] : "<unattributed:" $3 ">" }
' "$LEDGER" | sort -u)
n=$(printf '%s\n' "$owners" | grep -c . || true)
if [ "${n:-0}" -gt 1 ] || printf '%s' "$owners" | grep -q '^<unattributed:'; then
echo "REFUSING to sweep: open rows span more than one owner, or are unattributable." >&2
printf '%s\n' "$owners" | sed 's/^/ owner: /' >&2
echo "" >&2
echo "sweep is the MANAGER's batch-close verb and destroys EVERY open row." >&2
echo "A lane tearing down its OWN rows must use the lane-scoped tool:" >&2
echo " .amplifier/evaluation/tools/lane_teardown.sh $BATCH_DIR <lane> teardown --yes" >&2
echo "The manager closing the batch passes --all-owners." >&2
exit 3
fi
fi
# ---- end guard --------------------------------------------------------

tmp=$(mktemp "$BATCH_DIR/.infra.XXXXXX")
remaining=0
while IFS=$'\t' read -r ts type id status destroy; do
[ -z "${ts:-}" ] && continue # skip blank lines
if [ "$status" = "open" ]; then
echo ">> sweeping type=$type id=$id: $destroy"
if bash -c "$destroy"; then
# Capture output so an "already gone" refusal can be recognised.
out=$(bash -c "$destroy" 2>&1); rc=$?
[ -n "$out" ] && printf '%s\n' "$out"
if [ "$rc" -eq 0 ]; then
status=swept
echo " swept ok"
elif printf '%s' "$out" | grep -qiE "$ALREADY_GONE_RE"; then
# model_performance-bqu: ALREADY GONE is the DESIRED end state of a
# destroy, not a failure. Before this, infrastructure torn down by any
# other path (a lane by hand, a manager recovery, a crash cleanup)
# left a row that could NEVER be closed: the destroy_cmd failed
# forever, so `sweep` never exited clean and SKILL.md's "do not treat
# the highway as closed until it exits clean" became unsatisfiable.
# Recorded DISTINCTLY from a real teardown so the two are never
# confused in the ledger.
status=swept:already-absent
echo " already absent - closing row (not a teardown this sweep performed)"
else
rc=$?
echo " FAILED (rc=$rc) type=$type id=$id: $destroy" >&2
remaining=$(( remaining + 1 ))
fi
Expand Down
Loading
Loading