Skip to content

docs(lanes): settle this repo's lane-artifact root at docs/lanes/ (aof) - #300

Draft
Brian Krabach (bkrabach) wants to merge 6 commits into
mainfrom
lane/aof-artifact-path-conflict
Draft

docs(lanes): settle this repo's lane-artifact root at docs/lanes/ (aof)#300
Brian Krabach (bkrabach) wants to merge 6 commits into
mainfrom
lane/aof-artifact-path-conflict

Conversation

@bkrabach

Copy link
Copy Markdown
Collaborator

The problem

Every lane working in this repo has been graded against a rule its own goal contradicts.

source of truth says
the CODE lane goal template (a lane's authoritative spec, read first) docs/lanes/<lane>/DONE-NOTE.md
check_lane_artifact_paths.py (artifact-path/v1), rule [R2 ai_working/] ai_working/<lane>/

Both have precedent at origin/main, so it cannot be settled by looking at the repo. Flagged by lane 2nz while landing #299 (35ab604) and confirmed by the manager as item model_performance-aof.

The cost is not tidiness. A checker that disagrees with the spec trains lanes to ignore the checker — and the same checker catches the real hazard, a repo-root DONE-NOTE.md silently clobbering the previous lane's (item kez). 2nz paid the churn twice: it wrote its note to ai_working/ on the checker's say-so, then deleted it and rewrote it under docs/lanes/ (c80e04e).

The decision: option (a) — the goal template wins

(b) is not the cheap option it looks like. The two sides are not independent: tools/rewrite_goal_artifact_paths.py:49 imports resolve_artifact_dir from the checker and generates each goal's ARTIFACT ROOT sentence from it — deliberately, "so the instruction and the check can never disagree". Under (b) you would still have to change the resolver (so the generator emits ai_working/) and change the hand-written CODE goal template, which lives in the manager's per-batch goals/ directory: outside any git repo, with no test harness. The deliverable that stops the recurrence is a test, and under (b) it has nowhere to live.

Two facts then decide which answer is right:

  1. R2 fires here only because a lane artifact created the directory. The first tracked path under ai_working/ is ai_working/3yc-timedout-session-resumable/DONE-NOTE.md (0d93352). At 35ab604, all five tracked paths under ai_working/ are lane artifacts; the repo tracks no content there at any ref, and .gitignore mentions only ai_working/tmp. The rule was reading its own past output back as a repo convention.
  2. This repo's recent merged history says docs/lanes/ — n1i (fix(resume): narrow the credential refresh to secrets, and thread model_role/provider_preferences through delegate resume #292), 9w0 (feat(spawner): expose a timed-out sub-session's partial output via session.partial (k64 prereq 2/2, PRODUCER) #297), eem (fix(spawner): recover a timed-out delegate's thinking/tool_call work, not only its text #298), 2nz (fix(skills/ten-lane-highway): ship the two sweep guards a tool update kept reverting #299), four consecutive lanes, every one graded VIOLATION today.

What changed

Evaluation repo — branch lane/aof-artifact-path-conflict, commit 449ed59, base 6269c44. That repo has no remote, so the complete patch is committed here as docs/lanes/aof-artifact-path-conflict/PROPOSED-evals-artifact-path-R-PIN.patch.

A new R-PIN rule, checked before R0–R3:

PINNED_ARTIFACT_ROOTS: dict[str, tuple[str, tuple[str, ...]]] = {
    "amplifier-app-cli": ("docs/lanes", ("ai_working",)),
}

Keyed on the checkout directory name and the origin URL slug, so a checkout cloned under another name still pins. legacy_roots is retrospective only — dir_for() never returns one, so no new lane is sent to ai_working/, while lanes that already landed there stay COMPLIANT and stay put.

This repodocs/lanes/README.md (the same statement for a human reader) plus this lane's own record. No source changed.

Fail-before / pass-after

This real lane, same command, same directory, two checkers (evidence/07-this-lane-graded.txt):

PARENT  (evals main 6269c44)   VIOLATION   expected ai_working/aof-artifact-path-conflict/  [R2 ai_working/]
                                 stray-in-repo  docs/lanes/aof-artifact-path-conflict/DONE-NOTE.md   (+11 more)
                                 --strict exit: 1

PATCHED (lane/aof 449ed59)     COMPLIANT   expected docs/lanes/aof-artifact-path-conflict/  [R-PIN docs/lanes/ (declared for amplifier-app-cli)]
                                 ok             docs/lanes/aof-artifact-path-conflict/DONE-NOTE.md   (+11 more)
                                 --strict exit: 0

Synthetic control, one script pointed at each checker (evidence/01, evidence/02):

RESOLVED     ai_working/<lane>/   [R2 ai_working/]                 ->  docs/lanes/<lane>/   [R-PIN ...]
GOAL-PATH    docs/lanes/<lane>/DONE-NOTE.md -> VIOLATION           ->  COMPLIANT
LEGACY-PATH  ai_working/<lane>/DONE-NOTE.md -> COMPLIANT           ->  COMPLIANT   (unchanged: nothing is relocated)

Both sides agree — at the generator, not just the checker

rewrite_goal_artifact_paths.artifact_dir_for() is the function that writes the ARTIFACT ROOT sentence into a goal file. For all 8 app-cli lanes in the manifest it returned ai_working/<lane>/ before and returns docs/lanes/<lane>/ after (evidence/04). The instruction a future lane is given and the rule it is graded against are now the same string, from the same call.

Batch-wide effect: only this repo moves

--manifest <HW>/manifest.tsv --all --markdown, before vs after (evidence/03):

COMPLIANT VIOLATION
whole batch, before 58 56
whole batch, after 62 52
amplifier-app-cli, before 0 7
amplifier-app-cli, after 4 3

The diff contains no other repo — every other repo's rule, expected shape and per-lane status are byte-identical. Pinned by test_unpinned_repo_still_infers_from_the_tree.

The 3 residual violations are pre-existing and unrelated to the root choice: adq wrote ../DONE-NOTE.md outside the checkout (the kez shape, correctly still refused); 9kk wrote into adq's directory; 3yc's manifest lane id differs from its directory name (...-not-resumable vs ...-resumable).

Tests

scenarios/_harness/tests/test_artifact_path_resolution.py, 10 tests, evals suite 233 → 243 passed:

  • test_pin_survives_every_directory_that_would_have_moved_itthe recurrence-stopper. probes/ (R1), ai_working/ (R2) and a three-member w9- wave family (R0) all tracked at the base ref simultaneously — every inference the rule can make, each of which alone used to decide the answer — and the answer does not move.
  • test_root_done_note_is_still_a_violation + test_root_done_note_is_a_violation_in_an_unpinned_repo_tookez's hazard, pinned repo and unpinned repo alike. A fix that relaxed this to end the disagreement would have been strictly worse than the disagreement.
  • test_a_landed_legacy_note_stays_compliant / test_a_new_lane_is_never_told_the_legacy_root — landed artifacts under both conventions stay exactly where they are; only what a new lane is told changes.
  • test_unpinned_repo_still_infers_from_the_tree — R0–R3 unchanged for every repo outside the table.
  • test_pin_follows_the_origin_slug_when_the_checkout_is_renamed, test_the_pin_table_is_well_formed, test_a_goal_conformant_note_is_compliant, test_pinned_repo_resolves_to_the_declared_root.

Full suite (this repo)

$ uv run pytest -q
1682 passed, 1 skipped, 13 deselected, 1 xfailed

Byte-identical to the 35ab604 baseline — this lane changes no app-cli source (evidence/06).

Spend

$0.00 against a $0.00 authority (0 runs × 0 arms × $0 / 1.00 = $0.00). No API calls, no DTU, no infrastructure created or claimed. The cap never bound.

Deviation and honest limits

  • The lane was launched with an amplifier-app-cli worktree, but both candidate fixes live in the evaluation repo. Writing only a note here would have left the conflict in place, so the change was made where it lives, on a lane branch in a git worktree, exactly as every evals lane in this batch does. Nothing was merged; that repo has no remote, hence the patch artifact.
  • The pin is a list, and a list must be maintained. R0–R3 still guess for every repo not in it. That is the trade the item asked for: the defect was that the answer was inferred, and an inference cannot be made to hold still.
  • PRE-RULE can no longer fire for a pinned repo (it triggers on expected_now != expected, which a pin makes equal at every ref). legacy_roots covers the same ground here, but a future pin added without one would retroactively fail lanes that obeyed the old answer. The docstring says so at the table.
  • docs/lanes/README.md is repo content this lane was not chartered to write. It exists so the decision is legible to a human who never runs the checker. Drop it if you disagree; nothing depends on it.

Full reasoning, deliverable-by-deliverable: docs/lanes/aof-artifact-path-conflict/DONE-NOTE.md.

Every lane working in this repo was graded against a rule its own goal
contradicts. The CODE goal template names `docs/lanes/<lane>/`; the batch's
artifact checker (`check_lane_artifact_paths.py`, artifact-path/v1) resolved
this repo to `ai_working/<lane>/` under rule R2. Both have precedent at
origin/main, so a lane that followed its authoritative spec was reported a
VIOLATION -- and 2nz paid for it twice, writing its note to `ai_working/` on the
checker's say-so and then rewriting it under `docs/lanes/` (c80e04e).

Resolved as option (a): the goal template wins, and the checker was taught this
repo's convention instead. R2 only ever fired here because a LANE ARTIFACT made
`ai_working/` tracked -- the first tracked path there is
`ai_working/3yc-.../DONE-NOTE.md` (0d93352), and at 35ab604 all five tracked
paths under it are lane artifacts. The rule was reading its own output back as
a repo convention. Meanwhile four consecutive merged lanes (n1i #292, 9w0 #297,
eem #298, 2nz #299) placed their notes under `docs/lanes/`.

This commit is the app-cli half:

  * docs/lanes/README.md          -- the convention, stated for a human reader,
                                     including why the `ai_working/` notes stay
                                     exactly where they are
  * docs/lanes/aof-.../           -- this lane's record: the decision argued
                                     from the code, fail-before/pass-after
                                     evidence, and the evals-repo patch as a
                                     reviewable artifact

The checker half is a separate commit in the evaluation repo (that repo has no
remote), carried here verbatim as
docs/lanes/aof-artifact-path-conflict/PROPOSED-evals-artifact-path-R-PIN.patch.

No app-cli source changed: 1682 passed, 1 skipped, 13 deselected, 1 xfailed --
byte-identical to the 35ab604 baseline.
…the remote

The evals-side commit lives in a repo with no remote, so 449ed59 itself cannot
be read back from one -- but the CHANGE can. The patch fetched from GitHub is
byte-identical to the lane branch's own format-patch (sha256 060a4079...f2e9b),
applies --check CLEAN to a fresh checkout of evals main 6269c44, and in that
reconstituted tree the 10 tests pass and this lane grades COMPLIANT at --strict
exit 0. The commit is reachable from a real branch ref, so it is not the 8bj
orphan class.

Outstanding and NOT a lane's to do: a local merge into evals main. This goal
says 'Do not merge anything to main' / 'Never merge', and ~139 live lane
worktrees branch off it.
…OMPLIANT from the merged state

Merged lane/aof-artifact-path-conflict into CURRENT evals main in a throwaway
detached worktree: 0 conflicts, 2 files, suite 243 passed (main alone: 233), and
this lane grades COMPLIANT at --strict exit 0 from the merged tree. Aborted and
removed; evals main untouched at 6269c44, which has not advanced since the
branch was cut (0 commits), so no rebase is needed either.

The post-merge state is therefore measured, not predicted. Running the merge for
real stays the manager's step: this goal says 'Do not merge anything to main' /
'Never merge', and 63 worktrees hang off that main.
… the evals branch merges

Self-correction. This item's acceptance says a lane that runs
check_lane_artifact_paths.py gets no VIOLATION. The copy a lane actually runs is
main's, and main (6269c44) still resolves app-cli to ai_working/ [R2]. So the
contradiction is still live for the next app-cli lane, and resolving the item
while my own resolution said 'needs a local merge to take effect' was an
overclaim.

Nothing about the work changed; only the claim of completion. Item
model_performance-aof reopened and left UNCLAIMED so whoever can merge can close
the loop. Two deliverable rows downgraded to GREEN-BUT-NOT-LIVE, with a status
banner at the top of the note.
…e discipline defect

The goal's OUTCOME branches are A, B, C and it says they are exhaustive. I
reopened the item and left it open and unclaimed, which is a fourth state --
none of the three. That was my error, not a finding.

Branch A's own words are 'the deliverables below exist (as a draft PR on the
module's origin)', and Procedure 4 says 'Never merge'. So the goal never set
'merged into evals main' as a lane's bar; the merge is the manager's next stage.
Under pressure I adopted that stricter bar anyway and produced the exact 1ru
churn this goal warns about ('Choose the terminal state ONCE') -- RESOLVED ->
REOPENED -> RESOLVED with no number changing.

Recorded in the note as a lane discipline defect, with the two claims I got
wrong along the way corrected in place (the '63 worktrees' merge-risk argument;
'the deliverable is unmet until merged'). Deliverable rows restored to DONE with
an explicit 'live at merge' qualifier, which is accurate and does not overclaim.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants