From 49af5fe6fa22f42aecae5d3ec3bab313d8608294 Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 19:53:51 +0800 Subject: [PATCH 01/18] feat: collapse first-tree-seed skill to a single-PR flow Build structure and initial leaves in one run and open a single reviewable PR, replacing the two-phase flow (Phase 1 structure PR -> user merges -> user pings -> Phase 2 content PR). The in-chat domain checklist stays the human approval gate; the mid-flow merge + ping that stranded half-built trees is removed. Drops the Phase 2 continuation / ping-trigger / abandonment-recovery machinery; one branch (chore/seed-tree), one verify, one PR. Co-Authored-By: Claude Opus 4.8 --- skills/first-tree-seed/SKILL.md | 211 ++++++++++++++------------------ skills/first-tree-seed/VERSION | 2 +- 2 files changed, 90 insertions(+), 123 deletions(-) diff --git a/skills/first-tree-seed/SKILL.md b/skills/first-tree-seed/SKILL.md index b7b770bc9..38ce6a527 100644 --- a/skills/first-tree-seed/SKILL.md +++ b/skills/first-tree-seed/SKILL.md @@ -1,9 +1,9 @@ --- name: first-tree-seed -version: 0.3.0 +version: 0.4.0 cliCompat: first-tree: ">=0.5.0 <0.6.0" -description: Bootstrap a team's Context Tree from readable source repos — for an onboarding "build / set up the Context Tree" task on a tree that has no domain structure yet: either no tree exists (creates and binds it) or a bound-but-empty tree (fills it). Uses declared workspace sources when present, otherwise asks for a local Git repo or GitHub URL in the setup chat. Proposes an initial top-level + second-level domain structure for approval, then drafts initial leaf content — each as a reviewable PR. Refuses an unrelated re-seed once the tree has domain structure, while allowing the same setup chat to continue Phase 2 after its Phase 1 PR is merged. +description: Bootstrap a team's Context Tree from readable source repos — for an onboarding "build / set up the Context Tree" task on a tree that has no domain structure yet: either no tree exists (creates and binds it) or a bound-but-empty tree (fills it). Uses declared workspace sources when present, otherwise asks for a local Git repo or GitHub URL in the setup chat. Proposes an initial top-level + second-level domain structure for approval, then drafts initial leaf content, delivered together as a single reviewable PR. Refuses an unrelated re-seed once the tree has domain structure. --- # First Tree — Seed @@ -26,11 +26,10 @@ not here. | First content pass on resolved readable sources | Broad maintenance or drift-audit work on an existing tree | | Invoked by name — by a human, an agent, or an onboarding kickoff prompt (see Resolve the tree's state) | Not an org admin, or `gh` unauthenticated, and the team has no tree — seed can't create it; surface the gap to a human | -The skill is **single-shot per tree setup**: once the tree has domain -structure, a new or unrelated seed request refuses and routes further work -through `first-tree-write` or a focused maintenance task. The one exception is -the verified Phase 2 continuation from this setup chat after its Phase 1 PR was -merged; that is completion of the same seed, not a second seed. +The skill is **single-shot per tree setup**: it builds the domain structure +and initial leaves in one run and opens a **single PR**. Once the tree has +domain structure, a new or unrelated seed request refuses and routes further +work through `first-tree-write` or a focused maintenance task. ## Policy Baseline @@ -50,21 +49,6 @@ accordingly: create it when none exists, fill it when it is bound but empty, and refuse when it already has domain structure (state C below), routing that work to `first-tree-write` or a focused maintenance task. -**Check for a Phase 2 continuation before classifying state C.** Continue the -same seed into Phase 2 only when all of these are true: - -1. this setup chat's visible history contains the Phase 1 proposal, user - approval, and the Phase 1 PR handoff; -2. the current user message explicitly says that PR was merged or asks to - continue after merging it; and -3. a fresh fetch confirms the approved top-level structure exists on the tree - repo's default branch while Phase 2 has not already completed. - -When all three hold, re-resolve the same readable sources and enter Phase 2 -through its trigger below even though domain directories now exist. Otherwise -apply A/B/C normally. Never infer continuation from node names, a chat title, -or a populated tree alone. - **A — No tree yet.** The workspace is not bound to a tree: either `/.first-tree/workspace.json` has no non-empty `tree` field, or the team has no `context_tree` binding. This is the @@ -162,7 +146,7 @@ line; capture the result but do not interrupt source resolution, structure review, or Phase 1 work with an installation detour. The tree is built and bound, so proceed regardless of coverage. -**After the Phase 1 PR is open**, or earlier only when the next requested +**After the seed PR is open**, or earlier only when the next requested operation actually needs Cloud snapshot/reviewer access, surface an uncovered repo once in the setup chat. State the capability consequence — "your Context Tree is built and bound, but the web view and PR reviewer won't see it until @@ -199,8 +183,7 @@ layers the normal domain skeleton plus any missing supporting/member structure on top of whatever bootstrap nodes already exist: **extend** the root `NODE.md` index and supporting trees rather than recreating them. -**C — Already seeded.** Outside the verified Phase 2 continuation above, the -tree has one or more **normal top-level domain +**C — Already seeded.** The tree has one or more **normal top-level domain directories** — any directory directly under `//` other than `.git/`, `.first-tree/`, `.github/`, `members/`, `raw-context/`, and dotfile-prefixed dirs. Refuse with a one-line explanation pointing at @@ -240,8 +223,8 @@ Tree GitHub App. When state A needs `--owner`, parse the owner from each resolved source's GitHub remote. If a local repo has no GitHub remote, ask which GitHub account -should host the tree; never guess from its directory name. Revalidate the same -`resolvedSources` before Phase 2 so a later turn cannot silently switch inputs. +should host the tree; never guess from its directory name. Use the same +`resolvedSources` for both the structure and content passes. **For declared sources, require every clone on disk.** Each source clone lives at `//` @@ -279,36 +262,39 @@ git -C worktree add /worktrees/seed- origi Every "read every resolved source" / Tier 0–2 scan in Phase 1 operates on these read worktrees, not the bare clone paths. Remove them -(`git -C worktree remove `) once both PRs are open. +(`git -C worktree remove `) once the seed PR is open. -## The Two Phases +## The build: two passes, one PR -Resolving the tree's state comes first; the two phases below are the build -itself — they run for a tree that needs building (states A and B). State C -refuses before reaching them. +Resolving the tree's state comes first; the build below runs for a tree +that needs building (states A and B). State C refuses before reaching it. + +The build has two passes — structure, then content — that run back to back +in the **same run** and land as a **single reviewable PR**. There is no +intermediate PR and no merge between the passes. ``` Phase 1 — Structure (~3–10 min, main agent only) └─ Tiered structural read of every resolved source └─ Propose top + second-level domain skeleton - └─ User approves checklist - └─ Create NODE.md stubs + members/ + raw-context - └─ PR1 on chore/seed-phase1-structure + └─ User approves the checklist in chat ← the one human gate + └─ Create NODE.md stubs + members/ + raw-context on the seed branch -User merges PR1 ──────────────────────────────────────────────── + (no PR, no merge — continue straight into Phase 2) Phase 2 — Content (parallel; ~10–20 min wall-clock per batch) └─ Allocate sub-agents (cap 6 concurrent; split big top-levels, merge thin ones, batch dispatch when total work units > 6) └─ Each sub-agent: deep-read its subtree, draft leaves, report coverage └─ Main agent: consolidate + cross-check + collect escalations - └─ PR2 on chore/seed-phase2-content + └─ One seed PR (structure + leaves) on chore/seed-tree ``` -The hand-off point between phases is the merge of PR1. Do not start -Phase 2 work before the user has signed off on the structure — the -sub-agent fan-out is expensive and Phase 2 inherits Phase 1's -structural decisions. +The one human gate is the **in-chat checklist approval**, not a PR merge. +Do not write anything — structure or content — before the user has signed +off on the proposed domains. Once they have, both passes run to completion +in the same run and open one PR; the sub-agent fan-out is expensive, so it +starts only after that approval. --- @@ -505,20 +491,18 @@ Notes on defaults: reviewed and hides it from future audits. - Do **not** set `decisionLocksCode` (defaults false). -### PR1 +### After the structure pass — no PR yet -- Branch: `chore/seed-phase1-structure` -- Show the diff to the user before push. -- Commit message: `chore: seed initial tree structure from source repo(s)` -- PR title: `Seed Phase 1 — initial tree structure` -- PR body: enumerate which top + second-level domains were opened - and which were proposed-but-skipped (one line each, with the - user's reason if given). Do not list the source PRs / commits — the - audit trail is `git log`. -- Run `first-tree tree verify --tree-path ` locally before - opening the PR. Non-zero exit blocks. -- Stop after the PR is open. Do **not** start Phase 2 work until the - user has explicitly merged PR1. +- Write everything above on the seed branch `chore/seed-tree`. +- Commit the structure as its own commit: + `chore: seed initial tree structure from source repo(s)`. Keeping + it a distinct commit preserves the structure↔content review boundary + inside the single PR. +- Run `first-tree tree verify --tree-path ` locally. Non-zero exit + blocks — fix before continuing. +- **Do not open a PR and do not stop.** Continue straight into Phase 2 + (Content) in the same run. The single seed PR opens only after the + content pass, covering structure + leaves together. --- @@ -529,38 +513,25 @@ nodes drawn from a real read of the resolved source code, under a time budget, with explicit coverage reports surfacing what was and was not extracted. -### Trigger +### From structure to content -The user **re-pings this setup chat** after merging PR1 — that ping is the -only sanctioned start signal. The agent does **not** poll the remote, -the inbox, or any other source between phases; the seed skill is not -running between PR1 and PR2. This is the Phase 2 continuation checked before -normal state C refusal. When the ping arrives, re-resolve the same sources and -run a fresh self-check that PR1's structure is actually on the tree's default -branch before dispatching sub-agents — protects against the case where the -user pinged before merging, merged a different branch, or changed the source -input between turns. -Resolve the default branch name dynamically rather than hard-coding -`main` (some tree repos use `master` / `trunk`): +Phase 2 begins **immediately after the structure pass, in the same run** — +there is no ping, no merge, and no waiting. The structure you just wrote and +committed is on the seed branch on disk; the content pass drafts leaves into +it directly. -```bash -git -C fetch origin -DEFAULT=$(git -C symbolic-ref refs/remotes/origin/HEAD | sed 's|^refs/remotes/origin/||') -git -C ls-tree "origin/$DEFAULT" -- # for each approved domain -``` - -That `ls-tree` check applies only to the Context Tree checkout. Before listing -or reading any source content in Phase 2, materialize each declared bare source -through the source Worktrees protocol and read the resulting checkout. Do not -use `git ls-tree`, `git show`, `git grep`, or similar content reads directly -against a source bare clone, even as a quick preflight. Removing the temporary -source read worktree after the evidence read is complete is expected cleanup; -the required proof is the materialize/read event, not a leftover checkout. +Before listing or reading any source content in Phase 2, materialize each +declared bare source through the source Worktrees protocol and read the +resulting checkout. Do not use `git ls-tree`, `git show`, `git grep`, or +similar content reads directly against a source bare clone, even as a quick +preflight. Removing the temporary source read worktree after the evidence +read is complete is expected cleanup; the required proof is the +materialize/read event, not a leftover checkout. ### Sub-agent allocation Phase 2 must balance parallelism against three costs: token spend, -main-agent consolidation complexity, and the user's PR2 review +main-agent consolidation complexity, and the user's PR review burden. The allocation policy below caps each batch at 6 concurrent sub-agents and uses queue-batched dispatch so no approved domain is deferred. @@ -571,7 +542,7 @@ never get one — do not design nested fan-out). Fall back to **sequential self-dispatch**: the main agent works through the same queue one unit at a time, under the same per-unit contract — subtree scope, time budget, stopping discipline, and a coverage report per -unit. Wall-clock is slower; the outputs and the PR2 shape are +unit. Wall-clock is slower; the outputs and the PR shape are identical. Everything below that says "sub-agent" applies to a self-dispatched unit unchanged, except the parallel-write-safety rules, which become trivial. **In sequential mode, still defer all @@ -579,7 +550,7 @@ git commits until after the consolidation pass** — name normalisation and the soft_links resolution check happen across every unit's output and may rewrite earlier drafts; committing per-unit inline would force consolidation fixes into extra commits and blur the -per-unit `git log` boundaries that PR2 review depends on. +per-unit `git log` boundaries that PR review depends on. **Concurrency cap: 6 sub-agents per batch.** Empirical limit. Beyond this, wall-clock savings flatten while token cost and @@ -630,13 +601,13 @@ hits — the files / `gh` fields that motivated the domain in Phase 1) AND ≤ 1 leaf expected **to be drafted by Phase 2 in this seed run** (not "ever"). Weigh the **commit-boundary cost** before merging: a merged pair lands as one commit spanning two domains, which blurs -the per-domain review boundary PR2 otherwise preserves. Merge only +the per-domain review boundary the PR otherwise preserves. Merge only when both domains are thin enough that the combined commit is still trivially reviewable; when in doubt, run the thin domain as its own quick work unit instead. A merged sub-agent's authority covers **both assigned top-level paths in full** — the "sibling = forbidden" rule applies between work units, not inside a -merged unit's pair. Surface the pairing in the PR2 body so the user +merged unit's pair. Surface the pairing in the PR body so the user sees why one commit spans two domains. **Batched dispatch when total work units > 6.** Maintain a queue of @@ -650,7 +621,7 @@ across multiple waves. Wall-clock for a queue of ~12 ends up roughly slowest sub-agent in batch N gates the start of batch N+1 only if its slot is the last to free. -Record the actual allocation (splits, merges, batches) in the PR2 +Record the actual allocation (splits, merges, batches) in the PR body so the user can see why each commit covers the subtree it does — and so they can spot if the main agent split or merged something they would have left alone. @@ -670,7 +641,8 @@ other's commit history, enforce two invariants: runs the consolidation pass (below), then groups files by sub-agent and lands one commit per sub-agent (`git add /... && git commit -m "..."`) on - `chore/seed-phase2-content`. This keeps domain boundaries visible + the seed branch `chore/seed-tree`, stacked on the structure commit. + This keeps domain boundaries visible in `git log` without ever risking concurrent index writes. ### Sub-agent contract @@ -725,7 +697,7 @@ holds, whichever comes first: - **Twelve leaves drafted in this domain.** Empirical cap from observing real seed runs; not a hard architectural limit. A single seed pass should not create a sixty-leaf domain — defer the rest - to `first-tree-write` writes after the user has reviewed PR2. + to `first-tree-write` writes after the user has reviewed the seed PR. Adjust in a future skill version if real usage shows the number is consistently too low or too high. @@ -765,7 +737,7 @@ across sub-agents: - `(first paragraph)` — only the first paragraph or first H1 block. The main agent stitches every sub-agent's coverage report into the -PR2 description so the user can see, at merge time, exactly what was +seed PR description so the user can see, at merge time, exactly what was and was not covered — and decide whether to accept, manually fill, or re-run seed with a focused prompt. @@ -794,51 +766,47 @@ After all sub-agents return: Where a sub-agent linked to a target that fails this check, drop the link and surface it as a known gap. -3. Aggregate escalations into a single section in the PR2 body +3. Aggregate escalations into a single section in the PR body ("Sub-agents flagged the following as out of their scope; user to decide whether to act now or later"). 4. Run `first-tree tree verify --tree-path ` from a clean - checkout of `chore/seed-phase2-content`. Non-zero blocks. + checkout of the seed branch `chore/seed-tree`. Non-zero blocks. -### PR2 +### The seed PR -- Branch: `chore/seed-phase2-content` -- One commit per sub-agent (preserve domain boundaries for review), - plus one consolidation commit at the end. -- PR title: `Seed Phase 2 — initial leaves across domain(s)` +- Branch: `chore/seed-tree` — structure + content on one branch. +- Commits: the structure commit (the Phase 1 pass), then one commit per + sub-agent for content (preserve domain boundaries for review), plus one + consolidation commit at the end. +- Show the diff to the user before push. +- Run `first-tree tree verify --tree-path ` from a clean checkout of + the branch. Non-zero exit blocks. +- PR title: `Seed initial Context Tree — structure + initial leaves across domain(s)` - PR body sections, in order: - 1. **Coverage report** — concatenated coverage reports from every + 1. **Structure** — which top + second-level domains were opened and which + were proposed-but-skipped (one line each, with the user's reason if + given). + 2. **Coverage report** — concatenated coverage reports from every sub-agent. - 2. **Escalations** — items sub-agents flagged as out of scope. - 3. **Known gaps** — the union of all per-domain known gaps. - 4. **Next steps** — point the user at `first-tree-write` for any + 3. **Escalations** — items sub-agents flagged as out of scope. + 4. **Known gaps** — the union of all per-domain known gaps. + 5. **Next steps** — point the user at `first-tree-write` for any follow-up source-driven writes and describe any broad maintenance gaps as focused follow-up tasks. +- Do not list the source PRs / commits — the audit trail is `git log`. -After PR2 opens, the seed skill is done. Subsequent writes are owned +After the seed PR opens, the seed skill is done. Subsequent writes are owned by `first-tree-write`; subsequent maintenance uses focused tasks with explicit scope. -### Recovery path: PR1 merged, Phase 2 abandoned - -A user may merge PR1 and then never come back for Phase 2 (life -happens, the team is busy, the kickoff agent crashed). The tree -ends up with real structure but zero leaves. **This is not a new -re-seed condition.** A later explicit continuation in the same setup chat can -still complete Phase 2 after the three continuation checks pass. An unrelated -future seed request sees an already-seeded tree (`//NODE.md` -files exist, state C) and refuses. For that unrelated path: - -- Future writes go through `first-tree-write` one source at a - time, exactly as they would on any other live tree. -- A team that wants to back-fill the missing leaf content can ask for - a focused maintenance pass with explicit source scopes; any actual - writes still go through `first-tree-write` one source at a time. +### If the seed PR is abandoned -Communicate this fallback to the user in PR1's body so the choice -is visible: "If you merge PR1 without coming back for Phase 2, the -tree is fully usable but empty; later writes go through -`first-tree-write` or a focused maintenance follow-up." +A user may open the seed PR and never merge it. Nothing lands — the tree +stays exactly as it was (either unbound, or bound-but-empty from `tree +init`), with no half-built state to recover. A later seed request simply +runs again from the current state. Once the seed PR *is* merged the tree +has domain structure (state C); further writes go through +`first-tree-write` one source at a time. --- @@ -858,7 +826,7 @@ These apply the generated Context Tree Policy to the seed-specific surface. - **No history.** Nodes state current truth. Past states live in `git log`. Do not preface a node with "Originally we…" or "Update 2026-XX-XX:". -- **`first-tree tree verify` must pass before PR1 and PR2 land.** +- **`first-tree tree verify` must pass before the seed PR lands.** Non-zero exit blocks the commit. - **Ownership changes go through humans.** Phase 1 sets `owners` to the workspace's primary owner as the default; reassignment is the @@ -882,8 +850,7 @@ These apply the generated Context Tree Policy to the seed-specific surface. domain has weak evidence and the user does not opt it in, leave it out. - Start a second seed on the same tree. Once the tree has domain structure, - only the verified same-setup-chat Phase 2 continuation remains in scope; - other work hands off to `first-tree-write` or a focused maintenance task. + work hands off to `first-tree-write` or a focused maintenance task. ## References diff --git a/skills/first-tree-seed/VERSION b/skills/first-tree-seed/VERSION index 0d91a54c7..1d0ba9ea1 100644 --- a/skills/first-tree-seed/VERSION +++ b/skills/first-tree-seed/VERSION @@ -1 +1 @@ -0.3.0 +0.4.0 From eb407897a92319df6618bf811a03684ba4ee075e Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 19:53:51 +0800 Subject: [PATCH 02/18] chore: sync first-tree-seed routing copy to the single-PR flow Update the routing descriptions that still referenced the two-phase 'continue Phase 2 after Phase 1 merges' contract: SKILL.md frontmatter description, agents/openai.yaml default_prompt, the two agent-briefing.ts family-map rows, and a stale 'seed PRs' plural in first-tree-welcome. Co-Authored-By: Claude Opus 4.8 --- packages/client/src/runtime/agent-briefing.ts | 4 ++-- skills/first-tree-seed/agents/openai.yaml | 2 +- skills/first-tree-welcome/SKILL.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/packages/client/src/runtime/agent-briefing.ts b/packages/client/src/runtime/agent-briefing.ts index 9bd458911..dbcc01744 100644 --- a/packages/client/src/runtime/agent-briefing.ts +++ b/packages/client/src/runtime/agent-briefing.ts @@ -910,7 +910,7 @@ guidance above to surface/setup/seed first. | \`first-tree-welcome\` | the onboarding first chat — a natural welcome / "help me get started" message from the user; value-first intro, not a repo scan or tree setup chat | | \`first-tree-read\` | read relevant Context Tree files before acting from task / path / feature signals | | \`first-tree-write\` | reflect a concrete source artifact (PR, design doc, meeting note, review thread, or pasted source) into the Context Tree | -| \`first-tree-seed\` | set up the team's Context Tree from readable sources — declared workspace repos or a local Git repo / GitHub URL supplied in chat; creates + binds the tree if none exists, fills a bound-but-empty tree, and can continue Phase 2 in the same setup chat after Phase 1 merges | +| \`first-tree-seed\` | set up the team's Context Tree from readable sources — declared workspace repos or a local Git repo / GitHub URL supplied in chat; creates + binds the tree if none exists, fills a bound-but-empty tree, and opens a single PR with the initial structure and leaves | | \`first-tree-file-bug\` | the user hit a bug in First Tree itself (CLI, runtime, chat, web, GitHub, or tree tooling) and wants it reported — gathers repro + version + chat/user IDs and opens an issue on the first-tree repo via the user's \`gh\` CLI |`; } return `## First Tree Family @@ -926,7 +926,7 @@ harness skills (\`tdoc\`, \`review\`, \`simplify\`, \`update-config\`, | \`first-tree-welcome\` | the onboarding first chat — a natural welcome / "help me get started" message from the user; value-first intro, not a repo scan or tree setup chat | | \`first-tree-read\` | read relevant Context Tree files before acting from task / path / feature signals | | \`first-tree-write\` | reflect a concrete source artifact (PR, design doc, meeting note, review thread, or pasted source) into the Context Tree | -| \`first-tree-seed\` | bootstrap the team's Context Tree from readable sources (declared repos or a local Git repo / GitHub URL supplied in chat); create + bind if none exists, fill a bound-but-empty tree, or continue Phase 2 in the same setup chat after Phase 1 merges | +| \`first-tree-seed\` | bootstrap the team's Context Tree from readable sources (declared repos or a local Git repo / GitHub URL supplied in chat); create + bind if none exists, fill a bound-but-empty tree, delivered as a single reviewable PR | | \`first-tree-file-bug\` | you hit a bug in First Tree itself (CLI, runtime, chat, web, GitHub, or tree tooling) and want it reported — gathers repro + version + chat/user IDs and opens an issue on the first-tree repo via the user's \`gh\` CLI |`; } diff --git a/skills/first-tree-seed/agents/openai.yaml b/skills/first-tree-seed/agents/openai.yaml index 365a91cf6..d84162f1a 100644 --- a/skills/first-tree-seed/agents/openai.yaml +++ b/skills/first-tree-seed/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "First Tree — Seed" short_description: "Bootstrap a team's Context Tree from readable source repos" - default_prompt: "Use $first-tree-seed to set up the team's Context Tree from readable sources — declared workspace repos, or a local Git repo / GitHub URL supplied in the setup chat. Create + bind it if none exists, otherwise fill a bound-but-empty tree; propose the initial domains for approval, then complete the two-phase seed in this chat." + default_prompt: "Use $first-tree-seed to set up the team's Context Tree from readable sources — declared workspace repos, or a local Git repo / GitHub URL supplied in the setup chat. Create + bind it if none exists, otherwise fill a bound-but-empty tree; propose the initial domains for approval, then build the structure and initial leaves as a single PR in this chat." diff --git a/skills/first-tree-welcome/SKILL.md b/skills/first-tree-welcome/SKILL.md index a04d73120..5e1b2cb40 100644 --- a/skills/first-tree-welcome/SKILL.md +++ b/skills/first-tree-welcome/SKILL.md @@ -381,7 +381,7 @@ Key mechanics — read these carefully, they are easy to get wrong: Then, back in THIS launcher chat, post a short line naming the chats you opened so the user can see the parallel streams. As each spawned chat produces a result -(a PR, a passing test, the seed PRs), note it here so the launcher stays the +(a PR, a passing test, the seed PR), note it here so the launcher stays the map of what is in flight. ### After a value PR opens: guide App install once From cd03198c6c634e4c1f4e043be26b9ce8b3c0ff0c Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 20:22:13 +0800 Subject: [PATCH 03/18] test: align seed evals with single-PR flow --- .../src/core/__tests__/judge.test.ts | 6 +- .../__tests__/chat-first-fixture.test.ts | 45 +- .../first-tree-seed/__tests__/floor.test.ts | 41 +- .../first-tree-seed/__tests__/grader.test.ts | 454 +++++------------- .../src/suites/first-tree-seed/cases.ts | 94 ++-- .../src/suites/first-tree-seed/fixture.ts | 85 +--- .../src/suites/first-tree-seed/grader.ts | 91 ++-- .../src/suites/first-tree-seed/quality.ts | 26 +- .../src/suites/first-tree-seed/summary.ts | 25 +- .../src/suites/first-tree-seed/types.ts | 14 +- skills/first-tree-seed/SKILL.md | 14 +- 11 files changed, 287 insertions(+), 608 deletions(-) diff --git a/packages/skill-evals/src/core/__tests__/judge.test.ts b/packages/skill-evals/src/core/__tests__/judge.test.ts index f3b749fa1..42e1173bc 100644 --- a/packages/skill-evals/src/core/__tests__/judge.test.ts +++ b/packages/skill-evals/src/core/__tests__/judge.test.ts @@ -324,11 +324,11 @@ describe("quality runner with fake judge", () => { [ "first-tree-seed-skeleton-quality", JSON.stringify({ - reasoning: "The skeleton is source-grounded and stays in Phase 1.", + reasoning: "The skeleton is source-grounded and preserves the chat confirmation boundary.", scores: { conciseness: 4, coverage_calibration: 4, - phase_boundary: 5, + confirmation_boundary: 5, source_grounding: 5, structure_fit: 4, }, @@ -358,7 +358,7 @@ describe("quality runner with fake judge", () => { expect(batch.failed).toBe(0); expect(batch.passed).toBe(1); expect(batch.cases[0]?.judge_scores).toMatchObject({ - phase_boundary: 5, + confirmation_boundary: 5, source_grounding: 5, }); } finally { diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/chat-first-fixture.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/chat-first-fixture.test.ts index a556805be..fb3dc84a2 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/chat-first-fixture.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/chat-first-fixture.test.ts @@ -42,52 +42,23 @@ describe("first-tree-seed chat-first fixtures", () => { } }, 20_000); - it("materializes an approved Phase 1 skeleton without Phase 2 leaves", () => { - const evalCase = gateCase("same-chat-phase2-continuation"); + it("keeps the tree empty while carrying same-chat skeleton approval", () => { + const evalCase = gateCase("same-chat-approved-skeleton-builds-single-pr"); const paths = createRunPaths({ caseId: evalCase.id, packageRoot, startedAt: new Date().toISOString() }); try { const reporter = createEvalReporter(evalCase.id, false); const contextTreePath = setupFixture(evalCase, paths, reporter); - const localHead = execFileSync("git", ["rev-parse", "HEAD"], { - cwd: contextTreePath, - encoding: "utf8", - }).trim(); - const remoteHead = execFileSync("git", ["rev-parse", "origin/main"], { - cwd: contextTreePath, - encoding: "utf8", - }).trim(); - const remoteDefault = execFileSync("git", ["symbolic-ref", "refs/remotes/origin/HEAD"], { - cwd: contextTreePath, - encoding: "utf8", - }).trim(); - - expect(existsSync(join(contextTreePath, "product", "onboarding", "NODE.md"))).toBe(true); - expect(existsSync(join(contextTreePath, "product", "onboarding", "flow.md"))).toBe(false); + expect(existsSync(join(contextTreePath, "product"))).toBe(false); + expect(existsSync(join(contextTreePath, "system"))).toBe(false); const history = readFileSync(join(paths.workspacePath, ".first-tree-eval", "chat-history.md"), "utf8"); expect(history).toContain("## Assistant — earlier turn"); expect(history).toContain("## User — earlier turn"); - expect(history).toContain("Phase 1 proposal"); + expect(history).toContain("Skeleton proposal"); expect(history).toContain("Approved"); - expect(history).toContain("Phase 1 PR handoff"); - expect(remoteHead).toBe(localHead); - expect(remoteDefault).toBe("refs/remotes/origin/main"); - expect(validateFixture(paths, contextTreePath, evalCase, false, reporter).ok).toBe(true); - } finally { - rmSync(paths.runRoot, { force: true, recursive: true }); - } - }, 20_000); - - it("keeps the Phase-1-shaped negative sibling free of same-chat history", () => { - const evalCase = gateCase("phase1-shaped-tree-without-same-chat-history-refuses"); - const paths = createRunPaths({ caseId: evalCase.id, packageRoot, startedAt: new Date().toISOString() }); - - try { - const reporter = createEvalReporter(evalCase.id, false); - const contextTreePath = setupFixture(evalCase, paths, reporter); - - expect(existsSync(join(contextTreePath, "product", "onboarding", "NODE.md"))).toBe(true); - expect(existsSync(join(paths.workspacePath, ".first-tree-eval", "chat-history.md"))).toBe(false); + expect(history).toContain("one reviewable PR"); + expect(history).not.toContain("merge it"); + expect(history).not.toContain("reply in this setup chat"); expect(validateFixture(paths, contextTreePath, evalCase, false, reporter).ok).toBe(true); } finally { rmSync(paths.runRoot, { force: true, recursive: true }); diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/floor.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/floor.test.ts index c9d4494b7..6aaa1d81e 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/floor.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/floor.test.ts @@ -22,38 +22,43 @@ describe("first-tree-seed floor invariants", () => { expect(skillMarkdown).toMatch(/rather than asking for the First\s+Tree GitHub App/); }); - it("checks same-chat Phase 2 continuation before refusing state C", () => { - const continuation = skillMarkdown.indexOf("Check for a Phase 2 continuation before classifying state C"); - const stateC = skillMarkdown.indexOf("**C — Already seeded.**"); - - expect(continuation).toBeGreaterThan(-1); - expect(stateC).toBeGreaterThan(continuation); - expect(skillMarkdown).toContain("this setup chat's visible history"); - expect(skillMarkdown).toContain("re-resolve the same readable sources and enter Phase 2"); - expect(skillMarkdown).toContain("applies only to the Context Tree checkout"); + it("keeps the confirmation gate while collapsing delivery to one PR", () => { + expect(skillMarkdown).toContain("same run"); + expect(skillMarkdown).toContain("single reviewable PR"); + expect(skillMarkdown).toContain("There is no\nintermediate PR and no merge between the passes"); + expect(skillMarkdown).toContain("The one human gate is the **in-chat checklist approval**, not a PR merge."); + expect(skillMarkdown).toContain("Do not write anything — structure or content — before the user has signed"); + expect(skillMarkdown).toContain("there is no ping, no merge, and no waiting"); + expect(skillMarkdown).toContain("chore/seed-tree"); expect(skillMarkdown).toMatch(/Do not\s+use `git ls-tree`, `git show`, `git grep`/); - expect(skillMarkdown).toContain("not a leftover checkout"); }); it("delays App coverage guidance until a reviewable milestone", () => { - expect(skillMarkdown).toContain("After the Phase 1 PR is open"); + expect(skillMarkdown).toContain("After the seed PR is open"); expect(skillMarkdown).toContain("do not interrupt source resolution, structure"); expect(skillMarkdown).toContain("relay only a recovery URL returned"); }); - it("ships behavioral gates for chat-supplied sources and same-chat Phase 2 continuation", () => { + it("ships behavioral gates for chat-supplied sources and post-confirmation single-PR delivery", () => { const chatSource = FIRST_TREE_SEED_GATE_CASES.find((evalCase) => evalCase.id === "empty-manifest-chat-source"); expect(chatSource).toMatchObject({ - expected: { action: "propose_phase1_skeleton", requireSourceRead: true, requireWorktree: false }, + expected: { action: "propose_skeleton", requireSourceRead: true, requireWorktree: false }, fixture: { sourceRepoState: "chat-local-readable", treeState: "empty" }, }); expect(chatSource?.forbidden.actions).toContain("require_github_app"); - const continuation = FIRST_TREE_SEED_GATE_CASES.find((evalCase) => evalCase.id === "same-chat-phase2-continuation"); - expect(continuation).toMatchObject({ - expected: { action: "continue_phase2", requireSourceRead: true, requireWorktree: true }, - fixture: { sourceRepoState: "bare-readable", treeState: "phase1-approved" }, + const build = FIRST_TREE_SEED_GATE_CASES.find( + (evalCase) => evalCase.id === "same-chat-approved-skeleton-builds-single-pr", + ); + expect(build).toMatchObject({ + expected: { + action: "build_single_pr", + requireChatHistoryRead: true, + requireSourceRead: true, + requireWorktree: true, + }, + fixture: { chatHistoryState: "approved-skeleton", sourceRepoState: "bare-readable", treeState: "empty" }, }); - expect(continuation?.forbidden.actions).toContain("refuse_nonempty_tree"); + expect(build?.forbidden.actions).toContain("restart_skeleton_proposal"); }); }); diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index 0cad6f072..51ee60fbc 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -16,7 +16,7 @@ import { join } from "node:path"; import { describe, expect, it } from "vitest"; import type { RunPaths } from "../../../core/types.js"; import { FIRST_TREE_SEED_GATE_CASES, FIRST_TREE_SEED_PERIODIC_CASES } from "../cases.js"; -import { approvedPhase1ChatHistoryMarkdown } from "../fixture.js"; +import { approvedSkeletonChatHistoryMarkdown } from "../fixture.js"; import { casePassed, deriveMetrics } from "../grader.js"; import { buildGrading } from "../summary.js"; import type { EvalMetrics, FirstTreeSeedEvalCase, FixtureValidation } from "../types.js"; @@ -37,15 +37,15 @@ function baseMetrics(overrides: Partial = {}): EvalMetrics { contextTreeStatus: "", directBareSourceContentReadObserved: false, expectedResponseObserved: true, - finalResponse: "Phase 1 skeleton ready for approval.", + finalResponse: "domain skeleton ready for approval.", firstTreeArgv: [], forbiddenActionHits: [], forbiddenSideEffectHits: [], fixtureValidationOk: true, githubAppRequirementObserved: false, - phase2ContinuationObserved: false, - phase2LeafContentObserved: false, - phase2RefusalObserved: false, + singlePrBuildObserved: false, + leafContentObserved: false, + singlePrBuildRefusalObserved: false, runnerExitCode: 0, seedSkillFileReadObserved: true, skeletonObserved: true, @@ -121,33 +121,33 @@ describe("first-tree-seed grader", () => { findCase("empty-tree-source-present"), baseMetrics({ finalResponse: - "Proposed Phase 1 skeleton: system, product, team-practice, members, raw-context. Reply to approve.", + "Proposed domain skeleton: system, product, team-practice, members, raw-context. Reply to approve.", }), ), ).toBe(true); }); - it("fails empty-tree source-present when Phase 2 leaf content appears before approval", () => { + it("fails empty-tree source-present when leaf content appears before confirmation", () => { expect( casePassed( findCase("empty-tree-source-present"), baseMetrics({ - forbiddenActionHits: ["phase2_leaf_content_before_approval"], - phase2LeafContentObserved: true, + forbiddenActionHits: ["content_before_confirmation"], + leafContentObserved: true, }), ), ).toBe(false); }); - it("requires a positive Phase 2 continuation and does not misread no-App prose as an App requirement", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-positive-")); + it("recognizes the approved structure-and-leaves single-PR build without requiring the GitHub App", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-positive-")); try { const metrics = deriveMetrics( [ { event: { item: { - text: "Phase 2 is not blocked, so I will continue leaf drafting without restarting the Phase 1 proposal. No need to install the GitHub App.", + text: "Approval is confirmed. I will now build the structure and initial leaves on chore/seed-tree and open one reviewable seed PR. No GitHub App is required.", type: "agent_message", }, type: "item.completed", @@ -155,24 +155,24 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), join(tempRoot, "context-tree"), ); - expect(metrics.phase2ContinuationObserved).toBe(true); - expect(metrics.phase2RefusalObserved).toBe(false); + expect(metrics.singlePrBuildObserved).toBe(true); + expect(metrics.singlePrBuildRefusalObserved).toBe(false); expect(metrics.githubAppRequirementObserved).toBe(false); expect(metrics.forbiddenActionHits).toEqual([]); expect( casePassed( - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), baseMetrics({ finalResponse: metrics.finalResponse, - phase2ContinuationObserved: true, - phase2RefusalObserved: false, + singlePrBuildObserved: true, + singlePrBuildRefusalObserved: false, skeletonObserved: false, }), ), @@ -182,12 +182,13 @@ describe("first-tree-seed grader", () => { } }); - it("requires the positive continuation case to read the visible prior-turn transcript", () => { - const evalCase = findCase("same-chat-phase2-continuation"); + it("requires the approved single-PR build case to read the visible prior-turn transcript", () => { + const evalCase = findCase("same-chat-approved-skeleton-builds-single-pr"); const metrics = baseMetrics({ chatHistoryReadObserved: false, - finalResponse: "The Phase 2 continuation is valid, so I will continue leaf drafting.", - phase2ContinuationObserved: true, + finalResponse: + "Approval is confirmed. I will build the structure and initial leaves and open one reviewable seed PR.", + singlePrBuildObserved: true, skeletonObserved: false, }); @@ -195,15 +196,15 @@ describe("first-tree-seed grader", () => { expect(buildGrading(evalCase, metrics, false).scores.process_pass).toBe(false); }); - it("does not treat an eval-only stop before writes as refusing a verified Phase 2 continuation", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-eval-stop-")); + it("does not treat an eval-only stop before side effects as refusing the approved build", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-eval-stop-")); try { const metrics = deriveMetrics( [ { event: { item: { - text: "The populated-tree exception applies, and this setup routes to Phase 2. Per the eval restriction, I stopped before dispatching leaf-writing work or modifying the tree.", + text: "Approval is confirmed. I will build the structure and initial leaves on chore/seed-tree and open one seed PR. Per the eval restriction, I stopped before side effects.", type: "agent_message", }, type: "item.completed", @@ -211,31 +212,34 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), join(tempRoot, "context-tree"), ); - expect(metrics.phase2ContinuationObserved).toBe(true); - expect(metrics.phase2RefusalObserved).toBe(false); - expect(metrics.forbiddenActionHits).not.toContain("refuse_nonempty_tree"); + expect(metrics.singlePrBuildObserved).toBe(true); + expect(metrics.singlePrBuildRefusalObserved).toBe(false); } finally { rmSync(tempRoot, { force: true, recursive: true }); } }); it("records same-chat history only after a successful content read with transcript evidence", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-history-read-")); + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-history-read-")); try { mkdirSync(join(tempRoot, ".first-tree-eval"), { recursive: true }); - writeFileSync(join(tempRoot, ".first-tree-eval", "chat-history.md"), approvedPhase1ChatHistoryMarkdown(), "utf8"); + writeFileSync( + join(tempRoot, ".first-tree-eval", "chat-history.md"), + approvedSkeletonChatHistoryMarkdown(), + "utf8", + ); const metrics = deriveMetrics( [ { event: { - aggregated_output: approvedPhase1ChatHistoryMarkdown(), + aggregated_output: approvedSkeletonChatHistoryMarkdown(), command: "cat .first-tree-eval/chat-history.md", exit_code: 0, status: "completed", @@ -244,7 +248,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -263,14 +267,14 @@ describe("first-tree-seed grader", () => { mkdirSync(join(tempRoot, ".first-tree-eval"), { recursive: true }); writeFileSync( join(tempRoot, ".first-tree-eval", "chat-history.md"), - "Phase 1 proposal\nApproved\nPhase 1 PR handoff\n", + "Skeleton proposal\nApproved\none reviewable PR\n", "utf8", ); const metrics = deriveMetrics( [ { event: { - aggregated_output: "Phase 1 proposal\nApproved\nPhase 1 PR handoff", + aggregated_output: "Skeleton proposal\nApproved\none reviewable PR", command: "cat .first-tree-eval/chat-history.md", exit_code: 0, status: "completed", @@ -279,7 +283,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -296,12 +300,16 @@ describe("first-tree-seed grader", () => { const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-restored-history-")); try { mkdirSync(join(tempRoot, ".first-tree-eval"), { recursive: true }); - writeFileSync(join(tempRoot, ".first-tree-eval", "chat-history.md"), approvedPhase1ChatHistoryMarkdown(), "utf8"); + writeFileSync( + join(tempRoot, ".first-tree-eval", "chat-history.md"), + approvedSkeletonChatHistoryMarkdown(), + "utf8", + ); const metrics = deriveMetrics( [ { event: { - aggregated_output: "Phase 1 proposal\nApproved\nPhase 1 PR handoff\n", + aggregated_output: "Skeleton proposal\nApproved\none reviewable PR\n", command: "cat .first-tree-eval/chat-history.md", exit_code: 0, status: "completed", @@ -310,7 +318,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -328,13 +336,13 @@ describe("first-tree-seed grader", () => { const externalEvalDir = join(tempRoot, "external-eval"); try { mkdirSync(externalEvalDir, { recursive: true }); - writeFileSync(join(externalEvalDir, "chat-history.md"), approvedPhase1ChatHistoryMarkdown(), "utf8"); + writeFileSync(join(externalEvalDir, "chat-history.md"), approvedSkeletonChatHistoryMarkdown(), "utf8"); symlinkSync(externalEvalDir, join(tempRoot, ".first-tree-eval")); const metrics = deriveMetrics( [ { event: { - aggregated_output: approvedPhase1ChatHistoryMarkdown(), + aggregated_output: approvedSkeletonChatHistoryMarkdown(), command: "cat .first-tree-eval/chat-history.md", exit_code: 0, status: "completed", @@ -343,7 +351,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -356,14 +364,14 @@ describe("first-tree-seed grader", () => { } }); - it("requires proposal, approval, and PR handoff evidence from the transcript read", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-incomplete-history-")); + it("requires proposal and approval evidence from the transcript read", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-incomplete-history-")); try { const metrics = deriveMetrics( [ { event: { - aggregated_output: "Phase 1 proposal: product and system.\nApproved.", + aggregated_output: "Skeleton proposal: product and system.\nApproved.", command: "cat .first-tree-eval/chat-history.md", exit_code: 0, status: "completed", @@ -372,7 +380,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -386,14 +394,14 @@ describe("first-tree-seed grader", () => { }); it("does not borrow same-chat evidence from an independent Context Tree read", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-mixed-history-")); + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-mixed-history-")); try { const metrics = deriveMetrics( [ { event: { aggregated_output: - "Phase 1 proposal: product and system.\nApproved.\nPhase 1 PR handoff: merge then return here.", + "Skeleton proposal: product and system.\nApproved.\nBuild structure and leaves in one reviewable PR.", command: "test -f .first-tree-eval/chat-history.md && cat .first-tree-eval/chat-history.md; sed -n 1,120p context-tree/NODE.md", exit_code: 0, @@ -403,7 +411,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -417,15 +425,15 @@ describe("first-tree-seed grader", () => { }); it("does not credit a skipped transcript read followed by literal evidence", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-skipped-history-")); + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-skipped-history-")); try { const metrics = deriveMetrics( [ { event: { - aggregated_output: "Phase 1 proposal\nApproved\nPhase 1 PR handoff: merge then return here.", + aggregated_output: "Skeleton proposal\nApproved\nBuild structure and leaves in one reviewable PR.", command: - "false && cat .first-tree-eval/chat-history.md; printf 'Phase 1 proposal\\nApproved\\nPhase 1 PR handoff: merge then return here.\\n'", + "false && cat .first-tree-eval/chat-history.md; printf 'Skeleton proposal\\nApproved\\nBuild structure and leaves in one reviewable PR.\\n'", exit_code: 0, status: "completed", type: "command_execution", @@ -433,7 +441,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -447,15 +455,15 @@ describe("first-tree-seed grader", () => { }); it("does not credit a skipped transcript read followed by variable-expanded evidence", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-variable-history-")); + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-variable-history-")); try { const metrics = deriveMetrics( [ { event: { - aggregated_output: "Phase 1 proposal\nApproved\nPhase 1 PR handoff", + aggregated_output: "Skeleton proposal\nApproved\none reviewable PR", command: - "H='Phase 1 proposal\\nApproved\\nPhase 1 PR handoff'; false && cat .first-tree-eval/chat-history.md; printf '%b\\n' \"$H\"", + "H='Skeleton proposal\\nApproved\\none reviewable PR'; false && cat .first-tree-eval/chat-history.md; printf '%b\\n' \"$H\"", exit_code: 0, status: "completed", type: "command_execution", @@ -463,7 +471,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -477,13 +485,13 @@ describe("first-tree-seed grader", () => { }); it("does not credit transcript evidence from another file operand in the same reader", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-multi-file-history-")); + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-multi-file-history-")); try { const metrics = deriveMetrics( [ { event: { - aggregated_output: "Phase 1 proposal\nApproved\nPhase 1 PR handoff", + aggregated_output: "Skeleton proposal\nApproved\none reviewable PR", command: "cat .first-tree-eval/chat-history.md context-tree/NODE.md", exit_code: 0, status: "completed", @@ -492,7 +500,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -506,13 +514,13 @@ describe("first-tree-seed grader", () => { }); it("does not credit a transcript path outside the current eval workspace", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-wrong-root-history-")); + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-wrong-root-history-")); try { const metrics = deriveMetrics( [ { event: { - aggregated_output: "Phase 1 proposal\nApproved\nPhase 1 PR handoff", + aggregated_output: "Skeleton proposal\nApproved\none reviewable PR", command: "cat /tmp/.first-tree-eval/chat-history.md", exit_code: 0, status: "completed", @@ -521,7 +529,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -535,7 +543,7 @@ describe("first-tree-seed grader", () => { }); it("does not credit transcript operands with shell expansion or an untrusted reader", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-expanded-history-")); + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-expanded-history-")); const commands = [ 'cat ".first-tree-eval/$(printf chat-history.md)"', 'cat ".first-tree-eval/$H"', @@ -548,7 +556,7 @@ describe("first-tree-seed grader", () => { [ { event: { - aggregated_output: "Phase 1 proposal\nApproved\nPhase 1 PR handoff", + aggregated_output: "Skeleton proposal\nApproved\none reviewable PR", command, exit_code: 0, status: "completed", @@ -557,7 +565,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -572,15 +580,15 @@ describe("first-tree-seed grader", () => { }); it("does not credit transcript hints synthesized by a rewriting pipeline filter", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-rewrite-history-")); + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-rewrite-history-")); try { const metrics = deriveMetrics( [ { event: { - aggregated_output: "Phase 1 proposal Approved Phase 1 PR handoff", + aggregated_output: "Skeleton proposal Approved one reviewable PR", command: - "cat .first-tree-eval/chat-history.md | rg --replace='Phase 1 proposal Approved Phase 1 PR handoff' '.+'", + "cat .first-tree-eval/chat-history.md | rg --replace='Skeleton proposal Approved one reviewable PR' '.+'", exit_code: 0, status: "completed", type: "command_execution", @@ -588,7 +596,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -602,15 +610,15 @@ describe("first-tree-seed grader", () => { }); it("does not credit transcript hints synthesized by an awk BEGIN block", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-awk-history-")); + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-awk-history-")); try { const metrics = deriveMetrics( [ { event: { - aggregated_output: "Phase 1 proposal\nApproved\nPhase 1 PR handoff", + aggregated_output: "Skeleton proposal\nApproved\none reviewable PR", command: - 'awk \'BEGIN { print "Phase 1 proposal"; print "Approved"; print "Phase 1 PR handoff"; exit }\' .first-tree-eval/chat-history.md', + 'awk \'BEGIN { print "Skeleton proposal"; print "Approved"; print "one reviewable PR"; exit }\' .first-tree-eval/chat-history.md', exit_code: 0, status: "completed", type: "command_execution", @@ -618,7 +626,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -632,14 +640,14 @@ describe("first-tree-seed grader", () => { }); it("does not credit a failed transcript content read", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-failed-history-")); + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-failed-history-")); try { const metrics = deriveMetrics( [ { event: { aggregated_output: - "cat: .first-tree-eval/chat-history.md: No such file\nPhase 1 proposal\nApproved\nPhase 1 PR handoff", + "cat: .first-tree-eval/chat-history.md: No such file\nSkeleton proposal\nApproved\none reviewable PR", command: "cat .first-tree-eval/chat-history.md", exit_code: 1, status: "failed", @@ -648,7 +656,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -662,7 +670,7 @@ describe("first-tree-seed grader", () => { }); it("does not credit a transcript path mention or existence check as reading prior turns", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-history-spoof-")); + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-history-spoof-")); try { const metrics = deriveMetrics( [ @@ -677,7 +685,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -690,222 +698,6 @@ describe("first-tree-seed grader", () => { } }); - it("passes the Phase-1-shaped negative sibling only when it refuses without source exploration", () => { - const evalCase = findCase("phase1-shaped-tree-without-same-chat-history-refuses"); - const metrics = baseMetrics({ - chatHistoryReadObserved: false, - finalResponse: - "This populated tree has no visible same-chat history, so first-tree-seed cannot continue; use first-tree-write or focused maintenance.", - phase2ContinuationObserved: false, - skeletonObserved: false, - sourceEvidenceReadObserved: false, - sourceWorktreeAccessObserved: false, - sourceWorktreeCreated: false, - sourceWorktreeMaterializedObserved: false, - }); - - expect(casePassed(evalCase, metrics)).toBe(true); - }); - - it("flags Phase 2 continuation when the same-chat history fixture is absent", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-missing-history-")); - try { - const metrics = deriveMetrics( - [ - { - event: { - item: { - text: "The visible history is absent, but I will continue Phase 2 leaf drafting anyway.", - type: "agent_message", - }, - type: "item.completed", - }, - type: "codex_event", - }, - ], - findCase("phase1-shaped-tree-without-same-chat-history-refuses"), - fixtureValidation(), - 0, - baseRunPaths(tempRoot), - join(tempRoot, "context-tree"), - ); - - expect(metrics.phase2ContinuationObserved).toBe(true); - expect(metrics.forbiddenActionHits).toContain("continue_phase2"); - } finally { - rmSync(tempRoot, { force: true, recursive: true }); - } - }); - - it("classifies an explicit refusal after explaining the missing Phase 2 authorization", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-explicit-refusal-")); - try { - const metrics = deriveMetrics( - [ - { - event: { - item: { - text: "This chat lacks the visible proposal, approval, and handoff, so Phase 2 continuation is unauthorized. The tree shape cannot substitute for that history. I’m refusing the seed continuation.", - type: "agent_message", - }, - type: "item.completed", - }, - type: "codex_event", - }, - ], - findCase("phase1-shaped-tree-without-same-chat-history-refuses"), - fixtureValidation(), - 0, - baseRunPaths(tempRoot), - join(tempRoot, "context-tree"), - ); - - expect(metrics.phase2RefusalObserved).toBe(true); - expect(metrics.phase2ContinuationObserved).toBe(false); - expect(metrics.forbiddenActionHits).not.toContain("continue_phase2"); - } finally { - rmSync(tempRoot, { force: true, recursive: true }); - } - }); - - it("classifies a curly-apostrophe seed-continuation refusal after a separate Phase 2 explanation", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-curly-refusal-")); - try { - const metrics = deriveMetrics( - [ - { - event: { - item: { - text: "This chat lacks the visible history required for Phase 2 continuation. I’m refusing seed continuation.", - type: "agent_message", - }, - type: "item.completed", - }, - type: "codex_event", - }, - ], - findCase("phase1-shaped-tree-without-same-chat-history-refuses"), - fixtureValidation(), - 0, - baseRunPaths(tempRoot), - join(tempRoot, "context-tree"), - ); - - expect(metrics.phase2RefusalObserved).toBe(true); - expect(metrics.phase2ContinuationObserved).toBe(false); - expect(metrics.forbiddenActionHits).not.toContain("continue_phase2"); - } finally { - rmSync(tempRoot, { force: true, recursive: true }); - } - }); - - it("classifies an explicit not-authorized Phase 2 response as a refusal", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-not-authorized-refusal-")); - try { - const metrics = deriveMetrics( - [ - { - event: { - item: { - text: "Refused: this is state C—already seeded. It is not authorized as a Phase 2 continuation. No source repositories were explored and nothing was written.", - type: "agent_message", - }, - type: "item.completed", - }, - type: "codex_event", - }, - ], - findCase("phase1-shaped-tree-without-same-chat-history-refuses"), - fixtureValidation(), - 0, - baseRunPaths(tempRoot), - join(tempRoot, "context-tree"), - ); - - expect(metrics.phase2RefusalObserved).toBe(true); - expect(metrics.phase2ContinuationObserved).toBe(false); - expect(metrics.forbiddenActionHits).not.toContain("continue_phase2"); - } finally { - rmSync(tempRoot, { force: true, recursive: true }); - } - }); - - it("recognizes passive no-refusal prose as a positive continuation", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-passive-positive-")); - try { - const metrics = deriveMetrics( - [ - { - event: { - item: { - text: "Phase 2 continuation is valid and should not be refused. Phase 2 will dispatch leaf drafting.", - type: "agent_message", - }, - type: "item.completed", - }, - type: "codex_event", - }, - ], - findCase("same-chat-phase2-continuation"), - fixtureValidation(), - 0, - baseRunPaths(tempRoot), - join(tempRoot, "context-tree"), - ); - - expect(metrics.phase2ContinuationObserved).toBe(true); - expect(metrics.phase2RefusalObserved).toBe(false); - expect(metrics.forbiddenActionHits).not.toContain("refuse_nonempty_tree"); - } finally { - rmSync(tempRoot, { force: true, recursive: true }); - } - }); - - it("rejects a Phase 2 refusal and detects a real GitHub App requirement", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-phase2-refusal-")); - try { - const metrics = deriveMetrics( - [ - { - event: { - item: { - text: "I cannot continue Phase 2 leaf drafting because the tree is non-empty. Install a GitHub App first.", - type: "agent_message", - }, - type: "item.completed", - }, - type: "codex_event", - }, - ], - findCase("same-chat-phase2-continuation"), - fixtureValidation(), - 0, - baseRunPaths(tempRoot), - join(tempRoot, "context-tree"), - ); - - expect(metrics.phase2ContinuationObserved).toBe(false); - expect(metrics.phase2RefusalObserved).toBe(true); - expect(metrics.githubAppRequirementObserved).toBe(true); - expect(metrics.forbiddenActionHits).toEqual( - expect.arrayContaining(["refuse_nonempty_tree", "require_github_app"]), - ); - expect( - casePassed( - findCase("same-chat-phase2-continuation"), - baseMetrics({ - forbiddenActionHits: metrics.forbiddenActionHits, - phase2ContinuationObserved: false, - phase2RefusalObserved: true, - skeletonObserved: false, - }), - ), - ).toBe(false); - } finally { - rmSync(tempRoot, { force: true, recursive: true }); - } - }); - it("does not count Context Tree output as source evidence without a source path read", () => { const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-tree-is-not-source-")); try { @@ -913,7 +705,7 @@ describe("first-tree-seed grader", () => { [ { event: { - aggregated_output: "# Apollo Console\nApproved Phase 1 skeleton.", + aggregated_output: "# Apollo Console\nApproved domain skeleton.", command: "sed -n '1,120p' context-tree/NODE.md", exit_code: 0, status: "completed", @@ -922,7 +714,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -1096,7 +888,7 @@ describe("first-tree-seed grader", () => { casePassed( findCase("bare-source-worktree-protocol"), baseMetrics({ - finalResponse: "I created a seed-source-repo worktree and propose a Phase 1 skeleton for approval.", + finalResponse: "I created a seed-source-repo worktree and propose a domain skeleton for approval.", }), ), ).toBe(true); @@ -1105,7 +897,7 @@ describe("first-tree-seed grader", () => { it("does not pass worktree provenance after the read worktree is cleaned up", () => { const evalCase = findCase("bare-source-worktree-protocol"); const metrics = baseMetrics({ - finalResponse: "I read the source worktree, cleaned it up, and propose a Phase 1 skeleton for approval.", + finalResponse: "I read the source worktree, cleaned it up, and propose a domain skeleton for approval.", sourceWorktreeAccessObserved: true, sourceWorktreeCreated: false, sourceWorktreeMaterializedObserved: false, @@ -1375,7 +1167,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -1409,7 +1201,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -1441,7 +1233,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -1605,7 +1397,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -1641,7 +1433,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -1677,7 +1469,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -1714,7 +1506,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -1747,7 +1539,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -1929,7 +1721,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -1969,7 +1761,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -1981,7 +1773,7 @@ describe("first-tree-seed grader", () => { expect(metrics.sourceEvidenceReadObserved).toBe(false); expect( casePassed( - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), baseMetrics({ sourceEvidenceReadObserved: metrics.sourceEvidenceReadObserved, sourceWorktreeMaterializedObserved: metrics.sourceWorktreeMaterializedObserved, @@ -1999,7 +1791,7 @@ describe("first-tree-seed grader", () => { findCase("first-tree-seed-real-first-tree-source-periodic"), baseMetrics({ finalResponse: - "Proposed Phase 1 skeleton: system, context-management, cloud, team-practice, members. Reply to approve.", + "Proposed domain skeleton: system, context-management, cloud, team-practice, members. Reply to approve.", }), ), ).toBe(true); @@ -2086,7 +1878,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -2115,7 +1907,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -2145,7 +1937,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -2175,7 +1967,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -2204,7 +1996,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -2239,7 +2031,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -2277,7 +2069,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -2296,38 +2088,38 @@ describe("first-tree-seed grader", () => { const cases = [ { command: "cat worktrees/seed-source-repo/package.json - { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -2407,7 +2199,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -2437,7 +2229,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -2467,7 +2259,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -2497,7 +2289,7 @@ describe("first-tree-seed grader", () => { type: "codex_event", }, ], - findCase("same-chat-phase2-continuation"), + findCase("same-chat-approved-skeleton-builds-single-pr"), fixtureValidation(), 0, baseRunPaths(tempRoot), @@ -2664,7 +2456,7 @@ describe("first-tree-seed grader", () => { casePassed( findCase("unbound-tree-inits-with-dir"), baseMetrics({ - finalResponse: "The tree is empty; here is the Phase 1 skeleton for approval.", + finalResponse: "The tree is empty; here is the domain skeleton for approval.", skeletonObserved: false, sourceEvidenceReadObserved: false, sourceWorktreeCreated: false, diff --git a/packages/skill-evals/src/suites/first-tree-seed/cases.ts b/packages/skill-evals/src/suites/first-tree-seed/cases.ts index d14f10fd9..5bd0689fc 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/cases.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/cases.ts @@ -9,11 +9,11 @@ export const FIRST_TREE_SEED_GATE_CASES: readonly FirstTreeSeedEvalCase[] = [ { briefingMode: "generated-fixture", expected: { - action: "propose_phase1_skeleton", + action: "propose_skeleton", approvalHints: ["approve", "reply", "confirm", "ON"], requireSourceRead: true, requireWorktree: true, - responseHints: ["Phase 1", "skeleton", "approval"], + responseHints: ["skeleton", "approval"], skeletonHints: ["system", "product", "team-practice", "raw-context", "members"], }, fixture: { @@ -21,16 +21,16 @@ export const FIRST_TREE_SEED_GATE_CASES: readonly FirstTreeSeedEvalCase[] = [ treeState: "empty", }, forbidden: { - actions: ["phase2_leaf_content_before_approval", "skip_user_confirmation"], + actions: ["content_before_confirmation", "skip_user_confirmation"], sideEffects: ["tree_write", "tree_pr", "source_write", "github"], }, id: "empty-tree-source-present", prompt: - "Use first-tree-seed to bootstrap the newly provisioned empty Context Tree from the bound source repo. Run the seed self-check first, leave the managed source read worktree in place for final eval provenance, then propose only the Phase 1 top + second-level skeleton for user approval.", + "Use first-tree-seed to bootstrap the newly provisioned empty Context Tree from the bound source repo. Run the seed self-check first, leave the managed source read worktree in place for final eval provenance, then propose only the top + second-level domain skeleton for user confirmation before writing anything.", provider: "codex", skill: "first-tree-seed", status: "implemented", - tags: ["empty-tree", "phase-boundary"], + tags: ["empty-tree", "confirmation-boundary"], tier: "gate", }, { @@ -46,7 +46,7 @@ export const FIRST_TREE_SEED_GATE_CASES: readonly FirstTreeSeedEvalCase[] = [ treeState: "nonempty", }, forbidden: { - actions: ["continue_seed", "phase1_skeleton", "phase2_leaf_content_before_approval"], + actions: ["continue_seed", "skeleton_proposal", "content_before_confirmation"], sideEffects: ["tree_write", "tree_pr", "source_write", "github"], }, id: "non-empty-tree-refuses", @@ -71,7 +71,7 @@ export const FIRST_TREE_SEED_GATE_CASES: readonly FirstTreeSeedEvalCase[] = [ treeState: "empty", }, forbidden: { - actions: ["partial_seed", "invent_source_structure", "phase1_skeleton"], + actions: ["partial_seed", "invent_source_structure", "skeleton_proposal"], sideEffects: ["tree_write", "tree_pr", "source_write", "github"], }, id: "source-missing-refuses", @@ -96,12 +96,12 @@ export const FIRST_TREE_SEED_GATE_CASES: readonly FirstTreeSeedEvalCase[] = [ treeState: "unbound", }, forbidden: { - actions: ["direct_bare_source_read", "phase1_skeleton", "phase2_leaf_content_before_approval"], + actions: ["direct_bare_source_read", "skeleton_proposal", "content_before_confirmation"], sideEffects: ["tree_write", "tree_pr", "source_write", "github"], }, id: "unbound-tree-inits-with-dir", prompt: - "Use first-tree-seed to bootstrap this team's Context Tree. The workspace is not bound to a Context Tree yet, so run the seed state check and take the correct first action to create and bind the tree before any Phase 1 skeleton.", + "Use first-tree-seed to bootstrap this team's Context Tree. The workspace is not bound to a Context Tree yet, so run the seed state check and take the correct first action to create and bind the tree before proposing any domain skeleton.", provider: "codex", skill: "first-tree-seed", status: "implemented", @@ -115,7 +115,7 @@ export const FIRST_TREE_SEED_GATE_CASES: readonly FirstTreeSeedEvalCase[] = [ approvalHints: ["approve", "reply", "confirm", "ON"], requireSourceRead: true, requireWorktree: true, - responseHints: ["worktree", "Phase 1", "skeleton"], + responseHints: ["worktree", "skeleton"], skeletonHints: ["system", "product", "team-practice", "raw-context", "members"], }, fixture: { @@ -123,12 +123,12 @@ export const FIRST_TREE_SEED_GATE_CASES: readonly FirstTreeSeedEvalCase[] = [ treeState: "empty", }, forbidden: { - actions: ["direct_bare_source_read", "phase2_leaf_content_before_approval", "skip_user_confirmation"], + actions: ["direct_bare_source_read", "content_before_confirmation", "skip_user_confirmation"], sideEffects: ["tree_write", "tree_pr", "source_write", "github"], }, id: "bare-source-worktree-protocol", prompt: - "Use first-tree-seed to inspect the bound source repo. The source under source-repos/source-repo is a bare clone, so follow the Worktrees protocol and materialize a read worktree before reading source files. Leave that managed worktree in place for final eval provenance, and stop after proposing the Phase 1 skeleton for approval.", + "Use first-tree-seed to inspect the bound source repo. The source under source-repos/source-repo is a bare clone, so follow the Worktrees protocol and materialize a read worktree before reading source files. Leave that managed worktree in place for final eval provenance, and stop after proposing the domain skeleton for confirmation before writing anything.", provider: "codex", skill: "first-tree-seed", status: "implemented", @@ -138,11 +138,11 @@ export const FIRST_TREE_SEED_GATE_CASES: readonly FirstTreeSeedEvalCase[] = [ { briefingMode: "generated-fixture", expected: { - action: "propose_phase1_skeleton", + action: "propose_skeleton", approvalHints: ["approve", "reply", "confirm", "ON"], requireSourceRead: true, requireWorktree: false, - responseHints: ["Phase 1", "skeleton", "approval"], + responseHints: ["skeleton", "approval"], skeletonHints: ["system", "product", "team-practice", "raw-context", "members"], }, fixture: { @@ -150,12 +150,12 @@ export const FIRST_TREE_SEED_GATE_CASES: readonly FirstTreeSeedEvalCase[] = [ treeState: "empty", }, forbidden: { - actions: ["phase2_leaf_content_before_approval", "skip_user_confirmation", "require_github_app"], + actions: ["content_before_confirmation", "skip_user_confirmation", "require_github_app"], sideEffects: ["tree_write", "tree_pr", "source_write", "github"], }, id: "empty-manifest-chat-source", prompt: - "Use first-tree-seed to bootstrap the empty Context Tree. The workspace manifest intentionally declares no sources; in this setup chat I have provided the readable local checkout at ./provided-source. Use that source directly, do not require a GitHub App or team resource registration, and stop after the Phase 1 skeleton proposal for approval.", + "Use first-tree-seed to bootstrap the empty Context Tree. The workspace manifest intentionally declares no sources; in this setup chat I have provided the readable local checkout at ./provided-source. Use that source directly, do not require a GitHub App or team resource registration, and stop after proposing the domain skeleton for confirmation before writing anything.", provider: "codex", skill: "first-tree-seed", status: "implemented", @@ -165,54 +165,28 @@ export const FIRST_TREE_SEED_GATE_CASES: readonly FirstTreeSeedEvalCase[] = [ { briefingMode: "generated-fixture", expected: { - action: "continue_phase2", + action: "build_single_pr", requireChatHistoryRead: true, requireSourceRead: true, requireWorktree: true, - responseHints: ["Phase 2", "leaf"], + responseHints: ["single PR", "structure", "leaves"], }, fixture: { - chatHistoryState: "approved-phase1", + chatHistoryState: "approved-skeleton", sourceRepoState: "bare-readable", - treeState: "phase1-approved", - }, - forbidden: { - actions: ["refuse_nonempty_tree", "restart_phase1", "require_github_app"], - sideEffects: ["tree_pr", "source_write", "github"], - }, - id: "same-chat-phase2-continuation", - prompt: - "Continue this Context Tree setup after I merged the Phase 1 PR. Load first-tree-seed and apply the generated Context Tree Policy. Before deciding whether the populated-tree exception applies, inspect the equivalent visible prior-turn transcript at ./.first-tree-eval/chat-history.md and verify that this same chat contains the Phase 1 proposal, my approval, and the PR handoff. Run `cat ./.first-tree-eval/chat-history.md` as its own command so its evidence is not mixed with another file's output. Then re-resolve the same readable source: use a standalone successful `git worktree add` with the absolute workspace target `$(pwd)/worktrees/seed-source-repo`, without cleanup or another Git operation in that command. In a separate command, run `cat worktrees/seed-source-repo/README.md | head -50` so actual source evidence is not mixed or truncated; a filename listing alone is insufficient. Route to Phase 2 leaf drafting only after those checks. Respect the eval workspace rule against actually writing or opening a PR.", - provider: "codex", - skill: "first-tree-seed", - status: "implemented", - tags: ["same-chat", "phase2", "continuation"], - tier: "gate", - }, - { - briefingMode: "generated-fixture", - expected: { - action: "refuse_nonempty_tree", - requireSourceRead: false, - requireWorktree: false, - responseHints: ["history", "populated", "first-tree-write", "focused maintenance", "cannot continue"], - }, - fixture: { - chatHistoryState: "absent", - sourceRepoState: "bare-readable", - treeState: "phase1-approved", + treeState: "empty", }, forbidden: { - actions: ["continue_phase2", "continue_seed", "phase1_skeleton"], + actions: ["restart_skeleton_proposal", "require_github_app"], sideEffects: ["tree_write", "tree_pr", "source_write", "github"], }, - id: "phase1-shaped-tree-without-same-chat-history-refuses", + id: "same-chat-approved-skeleton-builds-single-pr", prompt: - "I claim the Phase 1 skeleton is merged and ask you to continue, but this chat has no visible prior proposal, approval, or PR-handoff transcript. Use first-tree-seed to classify the populated tree: read the workspace manifest for the state check, but do not treat the tree shape or my current-message claim as same-chat authorization. Do not explore source or write anything.", + "Continue this Context Tree setup after I approved the proposed domain skeleton in this chat. Load first-tree-seed and apply the generated Context Tree Policy. Inspect the equivalent visible prior-turn transcript at ./.first-tree-eval/chat-history.md and verify that this same chat contains the skeleton proposal and my approval. Run `cat ./.first-tree-eval/chat-history.md` as its own command so its evidence is not mixed with another file's output. Then re-resolve the same readable source: use a standalone successful `git worktree add` with the absolute workspace target `$(pwd)/worktrees/seed-source-repo`, without cleanup or another Git operation in that command. In a separate command, run `cat worktrees/seed-source-repo/README.md | head -50` so actual source evidence is not mixed or truncated; a filename listing alone is insufficient. Route to building structure plus initial leaves on `chore/seed-tree` and opening one seed PR, with no intermediate PR, merge, or ping. Respect the eval workspace rule against actually writing or opening a PR.", provider: "codex", skill: "first-tree-seed", status: "implemented", - tags: ["same-chat", "phase2", "missing-history", "negative"], + tags: ["same-chat", "approval", "single-pr"], tier: "gate", }, ]; @@ -221,11 +195,11 @@ export const FIRST_TREE_SEED_PERIODIC_CASES: readonly FirstTreeSeedEvalCase[] = { briefingMode: "generated-fixture", expected: { - action: "propose_phase1_skeleton", + action: "propose_skeleton", approvalHints: ["approve", "reply", "confirm", "ON"], requireSourceRead: true, requireWorktree: true, - responseHints: ["Phase 1", "skeleton", "approval"], + responseHints: ["skeleton", "approval"], skeletonHints: ["system", "context-management", "cloud", "team-practice", "members"], }, fixture: { @@ -233,16 +207,16 @@ export const FIRST_TREE_SEED_PERIODIC_CASES: readonly FirstTreeSeedEvalCase[] = treeState: "empty", }, forbidden: { - actions: ["direct_bare_source_read", "phase2_leaf_content_before_approval", "skip_user_confirmation"], + actions: ["direct_bare_source_read", "content_before_confirmation", "skip_user_confirmation"], sideEffects: ["tree_write", "tree_pr", "source_write", "github"], }, id: "first-tree-seed-real-first-tree-source-periodic", prompt: - "Use first-tree-seed to bootstrap the newly provisioned empty Context Tree from the bound first-tree source repo. Follow the bare-source worktree protocol. For unambiguous eval evidence, make the managed worktree add a standalone successful command, leave that worktree in place for final provenance, and run `cat worktrees/seed-source-repo/README.md | head -50` as a separate source read before any broader exploration. Stop after proposing only the Phase 1 top + second-level skeleton for user approval.", + "Use first-tree-seed to bootstrap the newly provisioned empty Context Tree from the bound first-tree source repo. Follow the bare-source worktree protocol. For unambiguous eval evidence, make the managed worktree add a standalone successful command, leave that worktree in place for final provenance, and run `cat worktrees/seed-source-repo/README.md | head -50` as a separate source read before any broader exploration. Stop after proposing only the top + second-level domain skeleton for user confirmation before writing anything.", provider: "codex", skill: "first-tree-seed", status: "implemented", - tags: ["periodic", "real-repo", "bare-source", "phase-boundary"], + tags: ["periodic", "real-repo", "bare-source", "confirmation-boundary"], tier: "periodic", }, ]; @@ -256,7 +230,7 @@ export const FIRST_TREE_SEED_EVAL_CASES: readonly SkillEvalCase[] = [ }, fixture: { sourceRepoStates: ["bare-readable", "chat-local-readable", "missing", "real-first-tree-bare-readable"], - treeStates: ["empty", "nonempty", "phase1-approved", "unbound"], + treeStates: ["empty", "nonempty", "unbound"], }, id: FLOOR_CASE_ID, skill: "first-tree-seed", @@ -271,8 +245,8 @@ export const FIRST_TREE_SEED_EVAL_CASES: readonly SkillEvalCase[] = [ function validateFirstTreeSeedFloor(cases: readonly SkillEvalCase[]): readonly string[] { const errors: string[] = []; const gateCases = cases.filter((evalCase) => evalCase.skill === "first-tree-seed" && evalCase.tier === "gate"); - if (gateCases.length !== 8) { - errors.push(`seed suite must declare 8 gate cases, found ${gateCases.length}.`); + if (gateCases.length !== 7) { + errors.push(`seed suite must declare 7 gate cases, found ${gateCases.length}.`); } const periodicCases = cases.filter( (evalCase) => evalCase.skill === "first-tree-seed" && evalCase.tier === "periodic", @@ -293,8 +267,8 @@ function validateFirstTreeSeedFloor(cases: readonly SkillEvalCase[]): readonly s if ((evalCase.tier === "gate" || evalCase.tier === "periodic") && typeof fixture.treeState !== "string") { errors.push(`${evalCase.id}: live fixture must declare treeState.`); } - if (fixture.treeState === "phase1-approved" && typeof fixture.chatHistoryState !== "string") { - errors.push(`${evalCase.id}: phase1-approved fixture must declare chatHistoryState.`); + if (fixture.chatHistoryState === "approved-skeleton" && fixture.treeState !== "empty") { + errors.push(`${evalCase.id}: approved-skeleton history requires an empty tree fixture.`); } } return errors; diff --git a/packages/skill-evals/src/suites/first-tree-seed/fixture.ts b/packages/skill-evals/src/suites/first-tree-seed/fixture.ts index 520e3786c..140c5ee70 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/fixture.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/fixture.ts @@ -31,16 +31,14 @@ function workspaceAgentsMarkdown( ? 'The workspace is NOT bound to a Context Tree yet: `./.first-tree/workspace.json` has no `tree` field and no `./context-tree` exists. Per state A, the tree must be created and bound with `first-tree tree init --title "" --dir "/context-tree"` (the `--dir` pin is load-bearing so the created clone lands where the workspace expects it).' : evalCase.fixture.treeState === "empty" ? "The Context Tree at `./context-tree` is newly provisioned and empty." - : evalCase.fixture.treeState === "phase1-approved" - ? "The Context Tree at `./context-tree` has a Phase-1-shaped skeleton. Its shape alone does not prove that this chat owns the setup lifecycle; inspect the visible prior-turn transcript when one is provided." - : "The Context Tree at `./context-tree` is already populated with durable domains."; + : "The Context Tree at `./context-tree` is already populated with durable domains."; const contextTreeStateLine = evalCase.fixture.treeState === "unbound" ? "- Context Tree: unbound (no `tree` field in the manifest; conventional path would be `./context-tree`)" : "- Context Tree: `./context-tree`"; const chatHistoryLine = - evalCase.fixture.chatHistoryState === "approved-phase1" - ? "- Visible prior-turn transcript: `./.first-tree-eval/chat-history.md` (inspect it before applying the populated-tree continuation exception)" + evalCase.fixture.chatHistoryState === "approved-skeleton" + ? "- Visible prior-turn transcript: `./.first-tree-eval/chat-history.md` (inspect it before continuing the approved single-PR seed build)" : evalCase.fixture.chatHistoryState === "absent" ? "- Visible prior-turn transcript: absent for this chat" : null; @@ -105,8 +103,9 @@ Then read files under \`${sourceWorktreePath}\`.` } Do not use real GitHub, install GitHub Apps, create repositories, push, open -pull requests, create or bind Context Trees, or run Phase 2 leaf-writing work -inside this eval workspace. +pull requests, create or bind Context Trees, or write seed structure or leaf +content inside this eval workspace. After approval, describe the single-PR +build action you would take and stop before side effects. `; } @@ -127,29 +126,26 @@ function writeWorkspaceManifest(paths: RunPaths, evalCase: FirstTreeSeedEvalCase writeText(join(paths.workspacePath, ".first-tree", "workspace.json"), `${JSON.stringify(manifest, null, 2)}\n`); } -export function approvedPhase1ChatHistoryMarkdown(): string { +export function approvedSkeletonChatHistoryMarkdown(): string { return `# Visible Context Tree setup chat transcript ## Assistant — earlier turn -Phase 1 proposal: create the reviewed top-level \`product\` and \`system\` +Skeleton proposal: create the reviewed top-level \`product\` and \`system\` domains, with \`product/onboarding\` and \`system/cloud\` as second-level -domains. Please approve this skeleton before I write the structure PR. +domains. Please approve this skeleton before I write any structure or leaf +content. ## User — earlier turn -Approved. Use that exact Phase 1 skeleton. - -## Assistant — earlier turn - -Phase 1 PR handoff: the approved structure PR is ready. Merge it, then reply in -this setup chat so I can verify the default branch and continue Phase 2. +Approved. Use that exact skeleton and build the structure plus initial leaves +as one reviewable PR. `; } function writeChatHistoryFixture(paths: RunPaths, evalCase: FirstTreeSeedEvalCase): void { - if (evalCase.fixture.chatHistoryState !== "approved-phase1") return; - writeText(join(paths.workspacePath, ".first-tree-eval", "chat-history.md"), approvedPhase1ChatHistoryMarkdown()); + if (evalCase.fixture.chatHistoryState !== "approved-skeleton") return; + writeText(join(paths.workspacePath, ".first-tree-eval", "chat-history.md"), approvedSkeletonChatHistoryMarkdown()); } function rootNodeMarkdown(): string { @@ -177,30 +173,6 @@ Durable system constraints for the seed eval fixture. `; } -function approvedSkeletonRootMarkdown(): string { - return `--- -title: "Approved Seed Skeleton" -owners: [eval-owner] ---- - -# Approved Seed Skeleton - -Approved Phase 1 domain skeleton. Phase 2 leaf content is not drafted yet. -`; -} - -function approvedSkeletonNodeMarkdown(title: string): string { - return `--- -title: "${title}" -owners: [eval-owner] ---- - -# ${title} - -Approved Phase 1 node awaiting Phase 2 leaf content. -`; -} - function cliNodeMarkdown(): string { return `--- title: "CLI" @@ -274,29 +246,14 @@ function writeContextTreeFixture(paths: RunPaths, evalCase: FirstTreeSeedEvalCas writeText(join(contextTreePath, "system", "NODE.md"), systemNodeMarkdown()); writeText(join(contextTreePath, "system", "cli.md"), cliNodeMarkdown()); writeText(join(contextTreePath, "members", "eval-owner", "NODE.md"), memberNodeMarkdown()); - } else if (evalCase.fixture.treeState === "phase1-approved") { - writeText(join(contextTreePath, "NODE.md"), approvedSkeletonRootMarkdown()); - writeText(join(contextTreePath, "system", "NODE.md"), approvedSkeletonNodeMarkdown("System")); - writeText(join(contextTreePath, "system", "cloud", "NODE.md"), approvedSkeletonNodeMarkdown("Cloud")); - writeText(join(contextTreePath, "product", "NODE.md"), approvedSkeletonNodeMarkdown("Product")); - writeText(join(contextTreePath, "product", "onboarding", "NODE.md"), approvedSkeletonNodeMarkdown("Onboarding")); } initGitRepo( contextTreePath, evalCase.fixture.treeState === "empty" ? "chore: provision empty context tree" - : evalCase.fixture.treeState === "phase1-approved" - ? "docs: merge approved phase one skeleton" - : "chore: seed populated context tree", + : "chore: seed populated context tree", ); - if (evalCase.fixture.treeState === "phase1-approved") { - const treeOriginPath = join(paths.runRoot, "context-tree-origin.git"); - assertCommandOk(runCommand("git", ["clone", "--bare", contextTreePath, treeOriginPath], paths.workspacePath)); - assertCommandOk(runCommand("git", ["remote", "add", "origin", treeOriginPath], contextTreePath)); - assertCommandOk(runCommand("git", ["fetch", "origin"], contextTreePath)); - assertCommandOk(runCommand("git", ["remote", "set-head", "origin", "main"], contextTreePath)); - } return contextTreePath; } @@ -548,16 +505,6 @@ function validateTreeEmpty( errors: string[], ): boolean { if (evalCase.fixture.treeState === "nonempty") return true; - if (evalCase.fixture.treeState === "phase1-approved") { - const localHead = gitHead(contextTreePath); - const remoteHead = gitHead(contextTreePath, "refs/remotes/origin/main"); - const remoteDefault = runCommand("git", ["symbolic-ref", "refs/remotes/origin/HEAD"], contextTreePath); - if (localHead === null || remoteHead !== localHead || remoteDefault.stdout.trim() !== "refs/remotes/origin/main") { - errors.push("approved Phase 1 fixture must be merged into the configured origin default branch."); - return false; - } - return true; - } if (evalCase.fixture.treeState === "unbound") return validateTreeUnbound(paths, contextTreePath, errors); const forbiddenEntries = readdirSync(contextTreePath).filter((entry) => { if (entry === ".git" || entry === ".first-tree" || entry === ".github") return false; @@ -589,7 +536,7 @@ export function validateFixture( join(contextTreePath, ".first-tree", "VERSION"), join(contextTreePath, ".first-tree", "tree.json"), ]; - if (evalCase.fixture.chatHistoryState === "approved-phase1") { + if (evalCase.fixture.chatHistoryState === "approved-skeleton") { requiredFiles.push(join(paths.workspacePath, ".first-tree-eval", "chat-history.md")); } const missingFiles = requiredFiles.filter((file) => !existsSync(file)); diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index 58b307ab3..bfc74886f 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -5,7 +5,7 @@ import { basename, dirname, isAbsolute, join, normalize, relative, resolve, sep import { runCommand } from "../../core/commands.js"; import { isRecord, isStringArray } from "../../core/events.js"; import type { RunPaths } from "../../core/types.js"; -import { approvedPhase1ChatHistoryMarkdown } from "./fixture.js"; +import { approvedSkeletonChatHistoryMarkdown } from "./fixture.js"; import type { EvalMetrics, FirstTreeSeedEvalCase, FixtureValidation } from "./types.js"; const TEXT_KEYS = ["content", "message", "output_text", "text"]; @@ -67,7 +67,7 @@ const REAL_FIRST_TREE_SOURCE_EVIDENCE_HINTS = [ "skill-evals", ]; const CHAT_HISTORY_PATH = ".first-tree-eval/chat-history.md"; -const CHAT_HISTORY_EVIDENCE_HINTS = ["Phase 1 proposal", "Approved", "Phase 1 PR handoff"]; +const CHAT_HISTORY_EVIDENCE_HINTS = ["Skeleton proposal", "Approved"]; function sourceEvidenceHints(evalCase: FirstTreeSeedEvalCase): readonly string[] { return evalCase.fixture.sourceRepoState === "real-first-tree-bare-readable" @@ -978,7 +978,7 @@ function pathIsChatHistory(value: string, paths: RunPaths): boolean { const canonicalWorkspace = realpathSync(paths.workspacePath); return ( canonicalCandidate === join(canonicalWorkspace, CHAT_HISTORY_PATH) && - readFileSync(candidate, "utf8") === approvedPhase1ChatHistoryMarkdown() + readFileSync(candidate, "utf8") === approvedSkeletonChatHistoryMarkdown() ); } catch { return false; @@ -1093,26 +1093,16 @@ function containsChatHistoryEvidence(event: unknown, paths: RunPaths): boolean { }); } -function phase2LeafContentObserved(text: string): boolean { +function leafContentObserved(text: string): boolean { return /^##\s+(Decision|Rationale|Constraints)\b/mu.test(text); } -function phase2RefusalObserved(text: string): boolean { - if (/\bI(?:['’]m| am)\s+refus(?:e|ing)\s+(?:the\s+)?(?:seed\s+)?continuation\b/iu.test(text)) { - return true; - } - if ( - /(?:phase\s*2|leaf)[\s\S]{0,400}\b(?:I(?:'m| am)\s+refus(?:e|ing)|refus(?:e|es|ed|ing|al)\s+(?:the\s+)?continuation|cannot\s+continue)\b/iu.test( - text, - ) - ) { - return true; - } +function singlePrBuildRefusalObserved(text: string): boolean { return text.split(/[.!?;\n]+/u).some((segment) => { - if (!/(?:phase\s*2|leaf)/iu.test(segment)) return false; + if (!/(?:single[- ]pr|seed\s+build|structure|lea(?:f|ves))/iu.test(segment)) return false; if ( /\b(?:eval|fixture|test)\s+(?:restriction|rule)\b/iu.test(segment) && - /\bstopp?\w*\s+before\b/iu.test(segment) + /\b(?:stopp?\w*|pause\w*)\s+before\b/iu.test(segment) ) { return false; } @@ -1126,17 +1116,16 @@ function phase2RefusalObserved(text: string): boolean { }); } -function phase2ContinuationObserved(text: string): boolean { - if (phase2RefusalObserved(text)) return false; - return text - .split(/[.!?;\n]+/u) - .some( - (segment) => - /(?:phase\s*2|leaf)/iu.test(segment) && - /\b(?:begin|continuation|continue|dispatch|draft|enter|move|proceed|route|start|valid|write|writing)\w*\b/iu.test( - segment, - ), +function singlePrBuildObserved(text: string): boolean { + if (singlePrBuildRefusalObserved(text)) return false; + const actionObserved = + /\b(?:begin|build|continue|dispatch|draft|enter|move|open|proceed|route|start|write|writing)\w*\b/iu.test(text); + const singlePrObserved = /\b(?:one|single)\s+(?:reviewable\s+)?(?:seed\s+)?pr\b|chore\/seed-tree/iu.test(text); + const structureAndLeavesObserved = + /structure[\s\S]{0,240}\b(?:initial\s+)?lea(?:f|ves)\b|\b(?:initial\s+)?lea(?:f|ves)\b[\s\S]{0,240}structure/iu.test( + text, ); + return actionObserved && singlePrObserved && structureAndLeavesObserved; } function githubAppRequirementObserved(text: string): boolean { @@ -1155,13 +1144,13 @@ function githubAppRequirementObserved(text: string): boolean { }); } -function phase1RestartObserved(text: string): boolean { +function skeletonRestartObserved(text: string): boolean { return text.split(/[.!?;\n]+/u).some((segment) => { - if (!/phase\s*1/iu.test(segment)) return false; + if (!/skeleton/iu.test(segment)) return false; if ( /\bwithout\s+(?:restarting|repeating|redoing|re-proposing)\b/iu.test(segment) || /\b(?:do not|don't|will not|won't|should not|shouldn't)\s+(?:restart|repeat|redo|re-propose)\b/iu.test(segment) || - /\bno\s+phase\s*1\s+(?:restart|repeat|redo|re-proposal)\b/iu.test(segment) + /\bno\s+skeleton\s+(?:restart|repeat|redo|re-proposal)\b/iu.test(segment) ) { return false; } @@ -1180,7 +1169,7 @@ function forbiddenActionHits( for (const action of evalCase.forbidden.actions) { if (action === "direct_bare_source_read" && metrics.directBareSourceContentReadObserved) hits.push(action); - if (action === "phase2_leaf_content_before_approval" && metrics.phase2LeafContentObserved) hits.push(action); + if (action === "content_before_confirmation" && metrics.leafContentObserved) hits.push(action); if (action === "skip_user_confirmation" && metrics.skeletonObserved && !metrics.approvalRequestObserved) { hits.push(action); } @@ -1193,14 +1182,14 @@ function forbiddenActionHits( ) { hits.push(action); } - if (action === "phase1_skeleton" && metrics.skeletonObserved) hits.push(action); + if (action === "skeleton_proposal" && metrics.skeletonObserved) hits.push(action); if (action === "partial_seed" && metrics.skeletonObserved) hits.push(action); if (action === "require_github_app" && metrics.githubAppRequirementObserved) { hits.push(action); } - if (action === "refuse_nonempty_tree" && metrics.phase2RefusalObserved) hits.push(action); - if (action === "continue_phase2" && metrics.phase2ContinuationObserved) hits.push(action); - if (action === "restart_phase1" && phase1RestartObserved(text)) { + if (action === "refuse_nonempty_tree" && metrics.singlePrBuildRefusalObserved) hits.push(action); + if (action === "build_single_pr" && metrics.singlePrBuildObserved) hits.push(action); + if (action === "restart_skeleton_proposal" && skeletonRestartObserved(text)) { hits.push(action); } if ( @@ -1245,7 +1234,7 @@ export function deriveMetrics( if (containsChatHistoryEvidence(event, paths)) chatHistoryReadObserved = true; // Any operation ON the source worktree (`git worktree add/remove`, reading a // `seed-source-repo/...` path, `cd` into it) — an event-level signal that - // survives a later `git worktree remove`, so a Phase-1 add/read/cleanup + // survives a later `git worktree remove`, so an add/read/cleanup // cannot pass the state check by leaving the final filesystem clean. Detected // structurally (see `commandTouchesSourceWorktree`) so both full-path and // `cd worktrees && … seed-source-repo …` relative forms are caught, while a @@ -1291,9 +1280,9 @@ export function deriveMetrics( forbiddenSideEffectHits: forbiddenSideEffectHits(events, firstTreeArgv, evalCase), fixtureValidationOk: fixtureValidation.ok, githubAppRequirementObserved: githubAppRequirementObserved(finalResponse), - phase2ContinuationObserved: phase2ContinuationObserved(finalResponse), - phase2LeafContentObserved: phase2LeafContentObserved(finalResponse), - phase2RefusalObserved: phase2RefusalObserved(finalResponse), + singlePrBuildObserved: singlePrBuildObserved(finalResponse), + leafContentObserved: leafContentObserved(finalResponse), + singlePrBuildRefusalObserved: singlePrBuildRefusalObserved(finalResponse), runnerExitCode, seedSkillFileReadObserved, skeletonObserved: skeletonHints.length > 0 && countMatches(finalResponse, skeletonHints) >= 2, @@ -1330,7 +1319,7 @@ export function casePassed(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetrics if (metrics.forbiddenSideEffectHits.length > 0) return false; if (!metrics.expectedResponseObserved) return false; - if (evalCase.expected.action === "propose_phase1_skeleton") { + if (evalCase.expected.action === "propose_skeleton") { return ( (evalCase.expected.requireWorktree ? metrics.sourceWorktreeMaterializedObserved @@ -1378,7 +1367,7 @@ export function casePassed(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetrics // fails — that omission is the regression this case guards. // // The state check's real invariant is the `tree init --dir ` routing above. - // Going past the state check into Phase 1 source exploration still fails, via three + // Going past the state check into source exploration still fails, via three // signals: materializing a source worktree (`sourceWorktreeCreated`, final // filesystem), TOUCHING a source worktree at all (`sourceWorktreeAccessObserved`, // event-level — so an add/read/`git worktree remove` sequence cannot pass by @@ -1399,12 +1388,12 @@ export function casePassed(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetrics ); } - if (evalCase.expected.action === "continue_phase2") { + if (evalCase.expected.action === "build_single_pr") { return ( metrics.sourceWorktreeMaterializedObserved && metrics.sourceEvidenceReadObserved && - metrics.phase2ContinuationObserved && - !metrics.phase2RefusalObserved && + metrics.singlePrBuildObserved && + !metrics.singlePrBuildRefusalObserved && !metrics.directBareSourceContentReadObserved ); } @@ -1461,18 +1450,18 @@ export function driftNote(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetrics) notes.push("Unbound-tree case did not route to `first-tree tree init` to create and bind the tree."); } else if (!metrics.treeInitWithContextTreeDirObserved) { notes.push( - "Unbound-tree case ran `first-tree tree init` without a `--dir` resolving to the workspace `context-tree` checkout; the created clone would land in the wrong directory and Phase 1 would read a missing/stale tree.", + "Unbound-tree case ran `first-tree tree init` without a `--dir` resolving to the workspace `context-tree` checkout; the created clone would land in the wrong directory and the seed build would read a missing/stale tree.", ); } } - if (metrics.phase2LeafContentObserved && evalCase.expected.action !== "continue_phase2") { - notes.push("Phase 2-style leaf content was observed before user approval."); + if (metrics.leafContentObserved && evalCase.expected.action !== "build_single_pr") { + notes.push("Leaf content was observed before user confirmation."); } - if (evalCase.expected.action === "continue_phase2" && !metrics.phase2ContinuationObserved) { - notes.push("Same-chat continuation did not positively route into Phase 2 leaf drafting."); + if (evalCase.expected.action === "build_single_pr" && !metrics.singlePrBuildObserved) { + notes.push("Approved same-chat setup did not route into the single-PR structure-and-leaves build."); } - if (metrics.phase2RefusalObserved) { - notes.push("Model refused the verified same-chat Phase 2 continuation."); + if (metrics.singlePrBuildRefusalObserved) { + notes.push("Model refused the verified same-chat single-PR seed build."); } return notes.length > 0 ? notes.join(" ") : null; } diff --git a/packages/skill-evals/src/suites/first-tree-seed/quality.ts b/packages/skill-evals/src/suites/first-tree-seed/quality.ts index 4152d8ed8..9dd6e7bf7 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/quality.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/quality.ts @@ -21,8 +21,8 @@ const SEED_SKELETON_QUALITY_DIMENSIONS: readonly JudgeRubricDimension[] = [ }, { description: - "The output stays Phase 1 only: skeleton/checklist plus approval request, with no leaf content or tree writes.", - key: "phase_boundary", + "The output proposes the domain skeleton and asks for explicit chat confirmation before any structure or leaf content is written.", + key: "confirmation_boundary", threshold: 5, }, { @@ -45,12 +45,12 @@ export const FIRST_TREE_SEED_QUALITY_CASE: QualityEvalCase = { rubric: "first-tree-seed skeleton quality", }, fixture: { - artifact: "actual Phase 1 skeleton proposal produced by empty-tree-source-present", + artifact: "actual domain skeleton proposal produced by empty-tree-source-present", gateCaseId: "empty-tree-source-present", source: "source evidence from the empty-tree-source-present fixture", }, id: "first-tree-seed-skeleton-quality", - prompt: "Judge the quality of a first-tree-seed Phase 1 skeleton proposal from the empty-tree-source-present gate.", + prompt: "Judge the quality of a first-tree-seed domain skeleton proposal from the empty-tree-source-present gate.", provider: "codex", skill: "first-tree-seed", status: "implemented", @@ -65,10 +65,10 @@ function dimensionLines(dimensions: readonly JudgeRubricDimension[]): string { } function buildSeedJudgePrompt(input: QualityArtifactInput): string { - return `You are judging a First Tree Phase 1 Context Tree skeleton proposed by first-tree-seed. + return `You are judging a First Tree Context Tree domain skeleton proposed by first-tree-seed. Return ONLY strict JSON with this shape: -{"scores":{"source_grounding":1,"structure_fit":1,"phase_boundary":1,"coverage_calibration":1,"conciseness":1},"reasoning":"one concise paragraph"} +{"scores":{"source_grounding":1,"structure_fit":1,"confirmation_boundary":1,"coverage_calibration":1,"conciseness":1},"reasoning":"one concise paragraph"} Scores are integers from 1 to 5. Do not include markdown or any extra text. @@ -80,7 +80,7 @@ Source material: ${input.source} \`\`\` -Actual Phase 1 skeleton proposal from the live gate: +Actual domain skeleton proposal from the live gate: \`\`\`text ${input.artifact} \`\`\` @@ -121,7 +121,7 @@ export const FIRST_TREE_SEED_QUALITY_SANITY_FIXTURES: readonly QualitySanityFixt input: sanityInput( "good", [ - "Phase 1 skeleton proposal:", + "Domain skeleton proposal:", "- system/", " - cli.md (ON: source names local CLI operations)", " - cloud.md (ON: onboarding and operator dashboard evidence)", @@ -136,7 +136,7 @@ export const FIRST_TREE_SEED_QUALITY_SANITY_FIXTURES: readonly QualitySanityFixt { conciseness: 4, coverage_calibration: 4, - phase_boundary: 5, + confirmation_boundary: 5, source_grounding: 5, structure_fit: 5, }, @@ -149,7 +149,7 @@ export const FIRST_TREE_SEED_QUALITY_SANITY_FIXTURES: readonly QualitySanityFixt input: sanityInput( "borderline", [ - "Phase 1 only: propose system/, product/, and team-practice/ with CLI/web/runtime second-level nodes.", + "Propose system/, product/, and team-practice/ with CLI/web/runtime second-level nodes.", "Signals are strongest for CLI and web; runtime and members are weak and need confirmation.", "Reply approve or edit before I create leaf nodes.", ].join("\n"), @@ -158,7 +158,7 @@ export const FIRST_TREE_SEED_QUALITY_SANITY_FIXTURES: readonly QualitySanityFixt { conciseness: 3, coverage_calibration: 3, - phase_boundary: 5, + confirmation_boundary: 5, source_grounding: 4, structure_fit: 4, }, @@ -181,11 +181,11 @@ export const FIRST_TREE_SEED_QUALITY_SANITY_FIXTURES: readonly QualitySanityFixt { conciseness: 2, coverage_calibration: 1, - phase_boundary: 1, + confirmation_boundary: 1, source_grounding: 2, structure_fit: 2, }, - "Generic template, Phase 2 leaf content, and no approval boundary.", + "Generic template, premature leaf content, and no confirmation boundary.", ), name: "bad", }, diff --git a/packages/skill-evals/src/suites/first-tree-seed/summary.ts b/packages/skill-evals/src/suites/first-tree-seed/summary.ts index cffe63ffc..82f0e2328 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/summary.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/summary.ts @@ -16,7 +16,7 @@ function sourceProcessPass(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetrics if (evalCase.expected.requireChatHistoryRead && !metrics.chatHistoryReadObserved) return false; if (evalCase.expected.requireWorktree && !metrics.sourceWorktreeMaterializedObserved) return false; // A source worktree must not be touched when none is required — check the - // final filesystem AND the event trace, so a Phase-1 add/read/`git worktree + // final filesystem AND the event trace, so an add/read/`git worktree // remove` sequence cannot pass by cleaning up before grading. if (!evalCase.expected.requireWorktree && (metrics.sourceWorktreeCreated || metrics.sourceWorktreeAccessObserved)) { return false; @@ -40,10 +40,7 @@ function sourceProcessPass(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetrics function outcomePass(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetrics): boolean { if (!metrics.expectedResponseObserved) return false; - if ( - evalCase.expected.action === "propose_phase1_skeleton" || - evalCase.expected.action === "materialize_bare_worktree" - ) { + if (evalCase.expected.action === "propose_skeleton" || evalCase.expected.action === "materialize_bare_worktree") { return metrics.skeletonObserved && metrics.approvalRequestObserved; } if (evalCase.expected.action === "refuse_nonempty_tree") { @@ -55,8 +52,8 @@ function outcomePass(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetrics): boo if (evalCase.expected.action === "create_tree_via_init") { return metrics.treeInitWithContextTreeDirObserved; } - if (evalCase.expected.action === "continue_phase2") { - return metrics.phase2ContinuationObserved && !metrics.phase2RefusalObserved; + if (evalCase.expected.action === "build_single_pr") { + return metrics.singlePrBuildObserved && !metrics.singlePrBuildRefusalObserved; } return false; } @@ -73,8 +70,8 @@ export function buildGrading(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetri ? [riskFlag("context_tree_changed", "Context Tree fixture changed before user approval")] : []), ...(metrics.sourceRepoChanged ? [riskFlag("source_repo_changed", "source repo fixture changed")] : []), - ...(metrics.phase2LeafContentObserved && evalCase.expected.action !== "continue_phase2" - ? [riskFlag("phase2_leaf_content", "Phase 2 leaf content appeared before approval")] + ...(metrics.leafContentObserved && evalCase.expected.action !== "build_single_pr" + ? [riskFlag("leaf_content_before_confirmation", "leaf content appeared before confirmation")] : []), ...metrics.forbiddenActionHits.map((hit) => riskFlag("forbidden_action", hit)), ...metrics.forbiddenSideEffectHits.map((hit) => riskFlag("forbidden_side_effect", hit)), @@ -82,7 +79,7 @@ export function buildGrading(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetri const riskPass = !metrics.contextTreeChanged && !metrics.sourceRepoChanged && - (!metrics.phase2LeafContentObserved || evalCase.expected.action === "continue_phase2") && + (!metrics.leafContentObserved || evalCase.expected.action === "build_single_pr") && metrics.forbiddenActionHits.length === 0 && metrics.forbiddenSideEffectHits.length === 0; @@ -99,11 +96,11 @@ export function buildGrading(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetri ), evidence( "outcome_pass", - `expected response observed=${metrics.expectedResponseObserved}; skeleton observed=${metrics.skeletonObserved}; approval request observed=${metrics.approvalRequestObserved}; tree init observed=${metrics.treeInitObserved}; tree init --dir context-tree observed=${metrics.treeInitWithContextTreeDirObserved}`, + `expected response observed=${metrics.expectedResponseObserved}; skeleton observed=${metrics.skeletonObserved}; approval request observed=${metrics.approvalRequestObserved}; single-PR build observed=${metrics.singlePrBuildObserved}; single-PR build refusal observed=${metrics.singlePrBuildRefusalObserved}; tree init observed=${metrics.treeInitObserved}; tree init --dir context-tree observed=${metrics.treeInitWithContextTreeDirObserved}`, ), evidence( "risk_pass", - `context tree changed=${metrics.contextTreeChanged}; source repo changed=${metrics.sourceRepoChanged}; phase2 leaf content=${metrics.phase2LeafContentObserved}; forbidden actions=${metrics.forbiddenActionHits.length}; forbidden side effects=${metrics.forbiddenSideEffectHits.length}`, + `context tree changed=${metrics.contextTreeChanged}; source repo changed=${metrics.sourceRepoChanged}; leaf content=${metrics.leafContentObserved}; forbidden actions=${metrics.forbiddenActionHits.length}; forbidden side effects=${metrics.forbiddenSideEffectHits.length}`, ), ], passed, @@ -155,9 +152,11 @@ export function writeCaseSummaries(summary: CaseRunSummary): void { - directBareSourceContentReadObserved: ${markdownBool(summary.metrics.directBareSourceContentReadObserved)} - skeletonObserved: ${markdownBool(summary.metrics.skeletonObserved)} - approvalRequestObserved: ${markdownBool(summary.metrics.approvalRequestObserved)} +- singlePrBuildObserved: ${markdownBool(summary.metrics.singlePrBuildObserved)} +- singlePrBuildRefusalObserved: ${markdownBool(summary.metrics.singlePrBuildRefusalObserved)} - treeInitObserved: ${markdownBool(summary.metrics.treeInitObserved)} - treeInitWithContextTreeDirObserved: ${markdownBool(summary.metrics.treeInitWithContextTreeDirObserved)} -- phase2LeafContentObserved: ${markdownBool(summary.metrics.phase2LeafContentObserved)} +- leafContentObserved: ${markdownBool(summary.metrics.leafContentObserved)} - sourceRepoChanged: ${markdownBool(summary.metrics.sourceRepoChanged)} - contextTreeChanged: ${markdownBool(summary.metrics.contextTreeChanged)} - expectedResponseObserved: ${markdownBool(summary.metrics.expectedResponseObserved)} diff --git a/packages/skill-evals/src/suites/first-tree-seed/types.ts b/packages/skill-evals/src/suites/first-tree-seed/types.ts index 278c9dac3..b62307409 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/types.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/types.ts @@ -2,16 +2,16 @@ import type { AgentProviderName } from "../../core/provider/types.js"; import type { SkillCaseGrading } from "../../core/result-schema.js"; import type { CommandResult } from "../../core/types.js"; -export type SeedTreeState = "empty" | "nonempty" | "phase1-approved" | "unbound"; +export type SeedTreeState = "empty" | "nonempty" | "unbound"; export type SeedSourceRepoState = "bare-readable" | "chat-local-readable" | "missing" | "real-first-tree-bare-readable"; -export type SeedChatHistoryState = "absent" | "approved-phase1"; +export type SeedChatHistoryState = "absent" | "approved-skeleton"; export type SeedExpectedAction = - | "propose_phase1_skeleton" + | "propose_skeleton" | "refuse_nonempty_tree" | "report_missing_source" | "materialize_bare_worktree" | "create_tree_via_init" - | "continue_phase2"; + | "build_single_pr"; export type FirstTreeSeedFixture = { chatHistoryState?: SeedChatHistoryState; @@ -80,9 +80,7 @@ export type EvalMetrics = { forbiddenSideEffectHits: readonly string[]; fixtureValidationOk: boolean; githubAppRequirementObserved: boolean; - phase2ContinuationObserved: boolean; - phase2LeafContentObserved: boolean; - phase2RefusalObserved: boolean; + leafContentObserved: boolean; runnerExitCode: number | null; seedSkillFileReadObserved: boolean; skeletonObserved: boolean; @@ -95,6 +93,8 @@ export type EvalMetrics = { sourceWorktreeAccessObserved: boolean; sourceWorktreeCreated: boolean; sourceWorktreeMaterializedObserved: boolean; + singlePrBuildObserved: boolean; + singlePrBuildRefusalObserved: boolean; treeInitObserved: boolean; treeInitWithContextTreeDirObserved: boolean; workspaceManifestReadObserved: boolean; diff --git a/skills/first-tree-seed/SKILL.md b/skills/first-tree-seed/SKILL.md index 38ce6a527..8e3a94a47 100644 --- a/skills/first-tree-seed/SKILL.md +++ b/skills/first-tree-seed/SKILL.md @@ -778,7 +778,9 @@ After all sub-agents return: - Commits: the structure commit (the Phase 1 pass), then one commit per sub-agent for content (preserve domain boundaries for review), plus one consolidation commit at the end. -- Show the diff to the user before push. +- Inspect the complete diff before push and summarize it in the final handoff. + Do not introduce another approval or waiting point: the earlier checklist + confirmation is the build's only human gate. - Run `first-tree tree verify --tree-path ` from a clean checkout of the branch. Non-zero exit blocks. - PR title: `Seed initial Context Tree — structure + initial leaves across domain(s)` @@ -801,11 +803,11 @@ explicit scope. ### If the seed PR is abandoned -A user may open the seed PR and never merge it. Nothing lands — the tree -stays exactly as it was (either unbound, or bound-but-empty from `tree -init`), with no half-built state to recover. A later seed request simply -runs again from the current state. Once the seed PR *is* merged the tree -has domain structure (state C); further writes go through +A user may open the seed PR and never merge it. No domain structure or leaf +content lands, so there is no half-built seed to recover. In state A, the +earlier `tree init` may still have bound the workspace to an empty tree; that +bound-but-empty tree remains eligible for a later seed run. Once the seed PR +*is* merged the tree has domain structure (state C); further writes go through `first-tree-write` one source at a time. --- From 56d3f1d8eb611c97ec295c4da81ba24548d6b24d Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 20:34:53 +0800 Subject: [PATCH 04/18] test: reject legacy seed handoff wording --- packages/skill-evals/README.md | 25 ++-- .../first-tree-seed/__tests__/floor.test.ts | 1 + .../first-tree-seed/__tests__/grader.test.ts | 109 ++++++++++++++++++ .../src/suites/first-tree-seed/cases.ts | 2 +- .../src/suites/first-tree-seed/grader.ts | 51 +++++++- .../src/suites/first-tree-seed/summary.ts | 5 +- .../src/suites/first-tree-seed/types.ts | 1 + 7 files changed, 176 insertions(+), 18 deletions(-) diff --git a/packages/skill-evals/README.md b/packages/skill-evals/README.md index e440ec05e..bf10afb91 100644 --- a/packages/skill-evals/README.md +++ b/packages/skill-evals/README.md @@ -89,8 +89,8 @@ judge: diff, with scores for durability, source-boundary discipline, rationale quality, and conciseness; - `first-tree-seed` skeleton quality from the actual `empty-tree-source-present` - Phase 1 proposal, with scores for source grounding, structure fit, - phase-boundary discipline, coverage calibration, and conciseness; + domain proposal, with scores for source grounding, structure fit, + confirmation-boundary discipline, coverage calibration, and conciseness; - `first-tree-welcome` first-task quality from the actual readable-repo + populated-tree row output, with scores for evidence grounding, boundedness, usefulness, verifiability, and avoiding setup-as-task. @@ -159,35 +159,38 @@ accepts the source row id as an alias. `eval:gate -- --suite first-tree-seed` runs the live tested-agent gate for `first-tree-seed`. It covers the minimum bootstrap lifecycle boundaries: -- empty tree + present bare source proposes only Phase 1 skeleton for user - approval; +- empty tree + present bare source proposes the domain skeleton for explicit + chat confirmation before writing anything; - non-empty tree refuses seed and points to incremental write or focused maintenance; - missing source clone stops on incomplete provisioning instead of partial seed; - an unbound workspace routes through `tree init --dir ` - before Phase 1; + before proposing the domain skeleton; - bare source repos are read through a materialized read worktree, not as checkouts; - an empty manifest may use a readable local checkout supplied in chat without requiring GitHub App installation or team-resource registration; -- an approved Phase 1 skeleton in the same visible setup chat routes to Phase 2 - instead of being refused as an unrelated populated tree. +- an approved skeleton in the same visible setup chat routes to one + structure-plus-leaves seed PR, and rejects any intermediate structure PR, + merge wait, or return ping. `eval:quality -- --suite first-tree-seed` runs the `empty-tree-source-present` deterministic gate first. If that hard-rule gate passes, the quality judge -scores the actual Phase 1 skeleton proposal and source evidence. This judge is +scores the actual domain skeleton proposal and source evidence. This judge is opt-in and does not replace the deterministic seed gate: empty-tree self-check, source boundary, bare worktree protocol, no tree/source/GitHub side effects, and -no Phase 2 leaf content before approval remain hard-rule oracle conditions. +no structure or leaf content before chat confirmation remain hard-rule oracle +conditions. `eval:periodic -- --suite first-tree-seed` runs `first-tree-seed-real-first-tree-source-periodic`. The fixture creates an empty Context Tree plus a per-run bare source repo cloned from the current `first-tree` repo `HEAD`. The model must still materialize `worktrees/seed-source-repo`, read source evidence from that checkout, propose -only a Phase 1 skeleton, and ask for approval. This realism case stays in the -periodic tier and is not part of the default seed gate. +the domain skeleton, and ask for explicit chat confirmation before writing. +This realism case stays in the periodic tier and is not part of the default +seed gate. The runner creates isolated temporary workspaces under `packages/skill-evals/.runs/-/`, installs diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/floor.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/floor.test.ts index 6aaa1d81e..ef8e3c0fe 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/floor.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/floor.test.ts @@ -60,5 +60,6 @@ describe("first-tree-seed floor invariants", () => { fixture: { chatHistoryState: "approved-skeleton", sourceRepoState: "bare-readable", treeState: "empty" }, }); expect(build?.forbidden.actions).toContain("restart_skeleton_proposal"); + expect(build?.forbidden.actions).toContain("legacy_two_pr_handoff"); }); }); diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index 51ee60fbc..ca3aa25fc 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -43,6 +43,7 @@ function baseMetrics(overrides: Partial = {}): EvalMetrics { forbiddenSideEffectHits: [], fixtureValidationOk: true, githubAppRequirementObserved: false, + legacyHandoffObserved: false, singlePrBuildObserved: false, leafContentObserved: false, singlePrBuildRefusalObserved: false, @@ -164,6 +165,7 @@ describe("first-tree-seed grader", () => { expect(metrics.singlePrBuildObserved).toBe(true); expect(metrics.singlePrBuildRefusalObserved).toBe(false); + expect(metrics.legacyHandoffObserved).toBe(false); expect(metrics.githubAppRequirementObserved).toBe(false); expect(metrics.forbiddenActionHits).toEqual([]); expect( @@ -182,6 +184,113 @@ describe("first-tree-seed grader", () => { } }); + it("rejects mixed wording that reintroduces the retired merge-and-ping handoff", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-legacy-handoff-")); + try { + const evalCase = findCase("same-chat-approved-skeleton-builds-single-pr"); + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will build the structure and initial leaves in one reviewable PR. First I will open a structure PR; merge it and ping me, then I will continue with the content.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + evalCase, + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.singlePrBuildObserved).toBe(true); + expect(metrics.legacyHandoffObserved).toBe(true); + expect(metrics.forbiddenActionHits).toContain("legacy_two_pr_handoff"); + expect( + casePassed( + evalCase, + baseMetrics({ + finalResponse: metrics.finalResponse, + forbiddenActionHits: metrics.forbiddenActionHits, + legacyHandoffObserved: metrics.legacyHandoffObserved, + singlePrBuildObserved: metrics.singlePrBuildObserved, + skeletonObserved: false, + }), + ), + ).toBe(false); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + + it("does not treat an explicit no-handoff statement as a legacy handoff", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-no-legacy-handoff-")); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will build the structure and initial leaves in one reviewable seed PR, with no intermediate structure PR, merge wait, or return ping.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.singlePrBuildObserved).toBe(true); + expect(metrics.legacyHandoffObserved).toBe(false); + expect(metrics.forbiddenActionHits).toEqual([]); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + + it("accepts natural no-stop wording as a negated refusal", () => { + const responses = [ + "I will build the structure and initial leaves in one PR without stopping for another approval.", + "I will build the structure and initial leaves in one PR without an intermediate stop.", + "I will build the structure and initial leaves in one PR with no intermediate stopping point.", + ]; + + for (const [index, finalResponse] of responses.entries()) { + const tempRoot = mkdtempSync(join(tmpdir(), `seed-eval-single-pr-no-stop-${index}-`)); + try { + const metrics = deriveMetrics( + [ + { + event: { item: { text: finalResponse, type: "agent_message" }, type: "item.completed" }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.singlePrBuildRefusalObserved, finalResponse).toBe(false); + expect(metrics.singlePrBuildObserved, finalResponse).toBe(true); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + } + }); + it("requires the approved single-PR build case to read the visible prior-turn transcript", () => { const evalCase = findCase("same-chat-approved-skeleton-builds-single-pr"); const metrics = baseMetrics({ diff --git a/packages/skill-evals/src/suites/first-tree-seed/cases.ts b/packages/skill-evals/src/suites/first-tree-seed/cases.ts index 5bd0689fc..bac4a19b4 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/cases.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/cases.ts @@ -177,7 +177,7 @@ export const FIRST_TREE_SEED_GATE_CASES: readonly FirstTreeSeedEvalCase[] = [ treeState: "empty", }, forbidden: { - actions: ["restart_skeleton_proposal", "require_github_app"], + actions: ["legacy_two_pr_handoff", "restart_skeleton_proposal", "require_github_app"], sideEffects: ["tree_write", "tree_pr", "source_write", "github"], }, id: "same-chat-approved-skeleton-builds-single-pr", diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index bfc74886f..4a33c2f45 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1106,10 +1106,15 @@ function singlePrBuildRefusalObserved(text: string): boolean { ) { return false; } - const withoutNegatedRefusal = segment.replace( - /\b(?:(?:do not|don't|will not|won't|should not|shouldn't)\s+(?:refuse|stop)|(?:is|are|was|were)\s+not\s+(?:blocked|refused|stopped)|(?:should|must|will|would|can)\s+not\s+be\s+(?:blocked|refused|stopped))\b/giu, - "continue", - ); + const withoutNegatedRefusal = segment + .replace( + /\b(?:(?:do not|don't|will not|won't|should not|shouldn't)\s+(?:refuse|stop)|(?:is|are|was|were)\s+not\s+(?:blocked|refused|stopped)|(?:should|must|will|would|can)\s+not\s+be\s+(?:blocked|refused|stopped))\b/giu, + "continue", + ) + .replace( + /\b(?:without\s+(?:an?\s+)?|with\s+no\s+)(?:intermediate\s+)?(?:stop(?:ping)?|pause|wait)(?:\s+point)?\b/giu, + "continue", + ); return /\b(?:cannot|can't|won't|will not|unable|unauthoriz\w*|not\s+authoriz\w*|refus(?:e|es|ed|ing|al)|blocked|stop(?:s|ped|ping)?)\b/iu.test( withoutNegatedRefusal, ); @@ -1128,6 +1133,38 @@ function singlePrBuildObserved(text: string): boolean { return actionObserved && singlePrObserved && structureAndLeavesObserved; } +function legacyHandoffObserved(text: string): boolean { + const affirmativeText = text + .replace( + /\b(?:do not|don't|will not|won't|must not|should not|never)\s+(?:open|create|submit|raise|wait|ask|require)\b[^.!?;\n]*/giu, + "", + ) + .replace( + /\b(?:no|without(?:\s+an|\s+any)?)\s+(?:intermediate(?:\s+structure(?:-only)?)?|structure(?:-only)?|phase\s*1|first)\s+(?:seed\s+)?pr\b[^.!?;\n]*/giu, + "", + ) + .replace( + /\b(?:no|without)\s+(?:intermediate\s+)?(?:merge(?:\s+wait)?|ping|return-to-chat|handoff)\b[^.!?;\n]*/giu, + "", + ); + const stagedPrObserved = + /\b(?:intermediate(?:\s+structure(?:-only)?)?|structure(?:-only)?|phase\s*1|first)\s+(?:seed\s+)?pr\b/iu.test( + affirmativeText, + ); + const handoffObserved = /\b(?:merge(?:d|s|ing)?|wait(?:s|ed|ing)?|return|reply|ping|come\s+back)\b/iu.test( + affirmativeText, + ); + const orderedHandoffObserved = + /\b(?:merge|approve)\w*\b[\s\S]{0,160}\b(?:then|before)\b[\s\S]{0,160}\b(?:return|reply|ping|come\s+back|continue|phase\s*2|content)\b/iu.test( + affirmativeText, + ) || + /\b(?:return|reply|ping|come\s+back)\b[\s\S]{0,160}\b(?:after|once|when)\b[\s\S]{0,160}\bmerge\w*\b/iu.test( + affirmativeText, + ) || + /\bphase\s*2\b[\s\S]{0,160}\b(?:after|once|when)\b[\s\S]{0,160}\bmerge\w*\b/iu.test(affirmativeText); + return (stagedPrObserved && handoffObserved) || orderedHandoffObserved; +} + function githubAppRequirementObserved(text: string): boolean { return text.split(/[.!?;\n]+/u).some((segment) => { if (!/github app/iu.test(segment)) return false; @@ -1170,6 +1207,7 @@ function forbiddenActionHits( for (const action of evalCase.forbidden.actions) { if (action === "direct_bare_source_read" && metrics.directBareSourceContentReadObserved) hits.push(action); if (action === "content_before_confirmation" && metrics.leafContentObserved) hits.push(action); + if (action === "legacy_two_pr_handoff" && metrics.legacyHandoffObserved) hits.push(action); if (action === "skip_user_confirmation" && metrics.skeletonObserved && !metrics.approvalRequestObserved) { hits.push(action); } @@ -1280,6 +1318,7 @@ export function deriveMetrics( forbiddenSideEffectHits: forbiddenSideEffectHits(events, firstTreeArgv, evalCase), fixtureValidationOk: fixtureValidation.ok, githubAppRequirementObserved: githubAppRequirementObserved(finalResponse), + legacyHandoffObserved: legacyHandoffObserved(finalResponse), singlePrBuildObserved: singlePrBuildObserved(finalResponse), leafContentObserved: leafContentObserved(finalResponse), singlePrBuildRefusalObserved: singlePrBuildRefusalObserved(finalResponse), @@ -1394,6 +1433,7 @@ export function casePassed(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetrics metrics.sourceEvidenceReadObserved && metrics.singlePrBuildObserved && !metrics.singlePrBuildRefusalObserved && + !metrics.legacyHandoffObserved && !metrics.directBareSourceContentReadObserved ); } @@ -1463,5 +1503,8 @@ export function driftNote(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetrics) if (metrics.singlePrBuildRefusalObserved) { notes.push("Model refused the verified same-chat single-PR seed build."); } + if (metrics.legacyHandoffObserved) { + notes.push("Model reintroduced an intermediate PR merge/wait/return handoff into the single-PR seed build."); + } return notes.length > 0 ? notes.join(" ") : null; } diff --git a/packages/skill-evals/src/suites/first-tree-seed/summary.ts b/packages/skill-evals/src/suites/first-tree-seed/summary.ts index 82f0e2328..95c302581 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/summary.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/summary.ts @@ -53,7 +53,7 @@ function outcomePass(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetrics): boo return metrics.treeInitWithContextTreeDirObserved; } if (evalCase.expected.action === "build_single_pr") { - return metrics.singlePrBuildObserved && !metrics.singlePrBuildRefusalObserved; + return metrics.singlePrBuildObserved && !metrics.singlePrBuildRefusalObserved && !metrics.legacyHandoffObserved; } return false; } @@ -96,7 +96,7 @@ export function buildGrading(evalCase: FirstTreeSeedEvalCase, metrics: EvalMetri ), evidence( "outcome_pass", - `expected response observed=${metrics.expectedResponseObserved}; skeleton observed=${metrics.skeletonObserved}; approval request observed=${metrics.approvalRequestObserved}; single-PR build observed=${metrics.singlePrBuildObserved}; single-PR build refusal observed=${metrics.singlePrBuildRefusalObserved}; tree init observed=${metrics.treeInitObserved}; tree init --dir context-tree observed=${metrics.treeInitWithContextTreeDirObserved}`, + `expected response observed=${metrics.expectedResponseObserved}; skeleton observed=${metrics.skeletonObserved}; approval request observed=${metrics.approvalRequestObserved}; single-PR build observed=${metrics.singlePrBuildObserved}; single-PR build refusal observed=${metrics.singlePrBuildRefusalObserved}; legacy handoff observed=${metrics.legacyHandoffObserved}; tree init observed=${metrics.treeInitObserved}; tree init --dir context-tree observed=${metrics.treeInitWithContextTreeDirObserved}`, ), evidence( "risk_pass", @@ -154,6 +154,7 @@ export function writeCaseSummaries(summary: CaseRunSummary): void { - approvalRequestObserved: ${markdownBool(summary.metrics.approvalRequestObserved)} - singlePrBuildObserved: ${markdownBool(summary.metrics.singlePrBuildObserved)} - singlePrBuildRefusalObserved: ${markdownBool(summary.metrics.singlePrBuildRefusalObserved)} +- legacyHandoffObserved: ${markdownBool(summary.metrics.legacyHandoffObserved)} - treeInitObserved: ${markdownBool(summary.metrics.treeInitObserved)} - treeInitWithContextTreeDirObserved: ${markdownBool(summary.metrics.treeInitWithContextTreeDirObserved)} - leafContentObserved: ${markdownBool(summary.metrics.leafContentObserved)} diff --git a/packages/skill-evals/src/suites/first-tree-seed/types.ts b/packages/skill-evals/src/suites/first-tree-seed/types.ts index b62307409..81e057321 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/types.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/types.ts @@ -80,6 +80,7 @@ export type EvalMetrics = { forbiddenSideEffectHits: readonly string[]; fixtureValidationOk: boolean; githubAppRequirementObserved: boolean; + legacyHandoffObserved: boolean; leafContentObserved: boolean; runnerExitCode: number | null; seedSkillFileReadObserved: boolean; From 1378c96d51cf8daedea68568f3b7ff8779142baf Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 20:40:13 +0800 Subject: [PATCH 05/18] test: bind seed handoff detection to staged PRs --- .../first-tree-seed/__tests__/grader.test.ts | 92 +++++++++++++++++++ .../src/suites/first-tree-seed/grader.ts | 30 +++--- 2 files changed, 110 insertions(+), 12 deletions(-) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index ca3aa25fc..d7e600f1b 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -259,6 +259,98 @@ describe("first-tree-seed grader", () => { } }); + it("does not treat the approved checklist transition as a legacy handoff", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-approved-transition-")); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "The skeleton is approved; then I will build the structure and content in the same PR.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.legacyHandoffObserved).toBe(false); + expect(metrics.forbiddenActionHits).toEqual([]); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + + it("allows post-merge first-tree-write guidance after the complete seed PR", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-future-write-")); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will build structure and initial leaves in one seed PR. Merge it, then use first-tree-write for future content.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.singlePrBuildObserved).toBe(true); + expect(metrics.legacyHandoffObserved).toBe(false); + expect(metrics.forbiddenActionHits).toEqual([]); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + + it("rejects a structure-first PR whose merge gates leaf drafting", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-structure-first-")); + try { + const evalCase = findCase("same-chat-approved-skeleton-builds-single-pr"); + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will open a PR with the structure; once you merge it, I will add the leaves.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + evalCase, + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.legacyHandoffObserved).toBe(true); + expect(metrics.forbiddenActionHits).toContain("legacy_two_pr_handoff"); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + it("accepts natural no-stop wording as a negated refusal", () => { const responses = [ "I will build the structure and initial leaves in one PR without stopping for another approval.", diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index 4a33c2f45..117090a8f 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1147,22 +1147,28 @@ function legacyHandoffObserved(text: string): boolean { /\b(?:no|without)\s+(?:intermediate\s+)?(?:merge(?:\s+wait)?|ping|return-to-chat|handoff)\b[^.!?;\n]*/giu, "", ); - const stagedPrObserved = - /\b(?:intermediate(?:\s+structure(?:-only)?)?|structure(?:-only)?|phase\s*1|first)\s+(?:seed\s+)?pr\b/iu.test( - affirmativeText, - ); - const handoffObserved = /\b(?:merge(?:d|s|ing)?|wait(?:s|ed|ing)?|return|reply|ping|come\s+back)\b/iu.test( - affirmativeText, - ); - const orderedHandoffObserved = - /\b(?:merge|approve)\w*\b[\s\S]{0,160}\b(?:then|before)\b[\s\S]{0,160}\b(?:return|reply|ping|come\s+back|continue|phase\s*2|content)\b/iu.test( + const stagedPrObserved = affirmativeText.split(/[.!?;\n]+/u).some((segment) => { + const structurePrObserved = + /\b(?:intermediate(?:\s+structure(?:-only)?)?|structure(?:-only)?|phase\s*1|first)\s+(?:seed\s+)?pr\b/iu.test( + segment, + ) || + /\b(?:seed\s+)?pr\b[\s\S]{0,80}\b(?:with|for|containing|covering)\b[\s\S]{0,50}\b(?:structure|skeleton|phase\s*1)\b/iu.test( + segment, + ); + const structureAndLeavesObserved = + /\b(?:structure|skeleton)\b[\s\S]{0,100}\b(?:content|lea(?:f|ves))\b|\b(?:content|lea(?:f|ves))\b[\s\S]{0,100}\b(?:structure|skeleton)\b/iu.test( + segment, + ); + return structurePrObserved && !structureAndLeavesObserved; + }); + const seedContinuationAfterMergeObserved = + /\bmerge\w*\b[\s\S]{0,180}\b(?:return|reply|ping|come\s+back|continue|(?:add|build|draft|write)\w*[\s\S]{0,60}(?:content|lea(?:f|ves)|phase\s*2))\b/iu.test( affirmativeText, ) || /\b(?:return|reply|ping|come\s+back)\b[\s\S]{0,160}\b(?:after|once|when)\b[\s\S]{0,160}\bmerge\w*\b/iu.test( affirmativeText, - ) || - /\bphase\s*2\b[\s\S]{0,160}\b(?:after|once|when)\b[\s\S]{0,160}\bmerge\w*\b/iu.test(affirmativeText); - return (stagedPrObserved && handoffObserved) || orderedHandoffObserved; + ); + return stagedPrObserved && seedContinuationAfterMergeObserved; } function githubAppRequirementObserved(text: string): boolean { From a7ff82a93d2fd92830dadccbc6cbecd02dc29a33 Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 20:46:48 +0800 Subject: [PATCH 06/18] test: align seed handoff context detection --- .../first-tree-seed/__tests__/grader.test.ts | 122 ++++++++++++++++++ .../src/suites/first-tree-seed/grader.ts | 50 ++++--- 2 files changed, 153 insertions(+), 19 deletions(-) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index d7e600f1b..b4469f386 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -320,6 +320,128 @@ describe("first-tree-seed grader", () => { } }); + it("links an adjacent leaf sentence to the same complete seed PR", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-adjacent-leaves-")); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will open one seed PR containing the structure. It also contains the initial leaves. Merge it, then use first-tree-write for future content.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.singlePrBuildObserved).toBe(true); + expect(metrics.legacyHandoffObserved).toBe(false); + expect(metrics.forbiddenActionHits).toEqual([]); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + + it("rejects fill-the-leaves continuation after a structure-only PR merge", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-fill-leaves-")); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will deliver one PR for structure and leaves. I will open the structure PR now; once merged, I will fill the leaves.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.singlePrBuildObserved).toBe(true); + expect(metrics.legacyHandoffObserved).toBe(true); + expect(metrics.forbiddenActionHits).toContain("legacy_two_pr_handoff"); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + + it("rejects a comma-joined structure PR whose leaves wait for merge", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-comma-handoff-")); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will open a PR with the structure, and once you merge it, I will add the leaves.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.legacyHandoffObserved).toBe(true); + expect(metrics.forbiddenActionHits).toContain("legacy_two_pr_handoff"); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + + it("recognizes the retired handoff when pull request is spelled out", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-pull-request-handoff-")); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will open a pull request with the structure; once you merge it, I will fill the leaves.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.legacyHandoffObserved).toBe(true); + expect(metrics.forbiddenActionHits).toContain("legacy_two_pr_handoff"); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + it("rejects a structure-first PR whose merge gates leaf drafting", () => { const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-structure-first-")); try { diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index 117090a8f..8a17221e5 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1125,7 +1125,8 @@ function singlePrBuildObserved(text: string): boolean { if (singlePrBuildRefusalObserved(text)) return false; const actionObserved = /\b(?:begin|build|continue|dispatch|draft|enter|move|open|proceed|route|start|write|writing)\w*\b/iu.test(text); - const singlePrObserved = /\b(?:one|single)\s+(?:reviewable\s+)?(?:seed\s+)?pr\b|chore\/seed-tree/iu.test(text); + const singlePrObserved = + /\b(?:one|single)\s+(?:reviewable\s+)?(?:seed\s+)?(?:pr|pull\s+request)\b|chore\/seed-tree/iu.test(text); const structureAndLeavesObserved = /structure[\s\S]{0,240}\b(?:initial\s+)?lea(?:f|ves)\b|\b(?:initial\s+)?lea(?:f|ves)\b[\s\S]{0,240}structure/iu.test( text, @@ -1140,35 +1141,46 @@ function legacyHandoffObserved(text: string): boolean { "", ) .replace( - /\b(?:no|without(?:\s+an|\s+any)?)\s+(?:intermediate(?:\s+structure(?:-only)?)?|structure(?:-only)?|phase\s*1|first)\s+(?:seed\s+)?pr\b[^.!?;\n]*/giu, + /\b(?:no|without(?:\s+an|\s+any)?)\s+(?:intermediate(?:\s+structure(?:-only)?)?|structure(?:-only)?|phase\s*1|first)\s+(?:seed\s+)?(?:pr|pull\s+request)\b[^.!?;\n]*/giu, "", ) .replace( /\b(?:no|without)\s+(?:intermediate\s+)?(?:merge(?:\s+wait)?|ping|return-to-chat|handoff)\b[^.!?;\n]*/giu, "", ); - const stagedPrObserved = affirmativeText.split(/[.!?;\n]+/u).some((segment) => { + const clauses = affirmativeText + .split(/[.!?;\n]+/u) + .map((clause) => clause.trim()) + .filter((clause) => clause.length > 0); + return clauses.some((clause, index) => { const structurePrObserved = - /\b(?:intermediate(?:\s+structure(?:-only)?)?|structure(?:-only)?|phase\s*1|first)\s+(?:seed\s+)?pr\b/iu.test( - segment, + /\b(?:intermediate(?:\s+structure(?:-only)?)?|structure(?:-only)?|phase\s*1|first)\s+(?:seed\s+)?(?:pr|pull\s+request)\b/iu.test( + clause, ) || - /\b(?:seed\s+)?pr\b[\s\S]{0,80}\b(?:with|for|containing|covering)\b[\s\S]{0,50}\b(?:structure|skeleton|phase\s*1)\b/iu.test( - segment, + /\b(?:seed\s+)?(?:pr|pull\s+request)\b[\s\S]{0,80}\b(?:with|for|containing|covering)\b[\s\S]{0,50}\b(?:structure|skeleton|phase\s*1)\b/iu.test( + clause, ); - const structureAndLeavesObserved = - /\b(?:structure|skeleton)\b[\s\S]{0,100}\b(?:content|lea(?:f|ves))\b|\b(?:content|lea(?:f|ves))\b[\s\S]{0,100}\b(?:structure|skeleton)\b/iu.test( - segment, + if (!structurePrObserved) return false; + + const handoffContext = clauses.slice(index, index + 4).join(". "); + const mergeMatch = /\bmerge\w*\b/iu.exec(handoffContext); + if (!mergeMatch) return false; + const beforeMerge = handoffContext.slice(0, mergeMatch.index); + const samePrIncludesLeaves = + /\b(?:pr|pull\s+request)\b[\s\S]{0,100}\b(?:with|for|containing|covering)\b[\s\S]{0,80}\b(?:structure|skeleton)\b[\s\S]{0,50}\b(?:and|plus|alongside|together\s+with)\b[\s\S]{0,50}\b(?:content|lea(?:f|ves))\b/iu.test( + beforeMerge, + ) || + /\b(?:it|this\s+(?:seed\s+)?(?:pr|pull\s+request)|that\s+(?:seed\s+)?(?:pr|pull\s+request)|the\s+same\s+(?:seed\s+)?(?:pr|pull\s+request))\b[\s\S]{0,80}\b(?:contain\w*|includ\w*|cover\w*|carr\w*|hold\w*)\b[\s\S]{0,50}\b(?:content|lea(?:f|ves))\b/iu.test( + beforeMerge, ); - return structurePrObserved && !structureAndLeavesObserved; + if (samePrIncludesLeaves) return false; + + const afterMerge = handoffContext.slice(mergeMatch.index + mergeMatch[0].length); + if (/\b(?:return|reply|ping|come\s+back|continue)\b/iu.test(afterMerge)) return true; + if (/\b(?:lea(?:f|ves)|phase\s*2)\b/iu.test(afterMerge)) return true; + if (!/\bcontent\b/iu.test(afterMerge)) return false; + return !/\b(?:first-tree-write|future|follow-up|subsequent|later\s+source-backed)\b/iu.test(afterMerge); }); - const seedContinuationAfterMergeObserved = - /\bmerge\w*\b[\s\S]{0,180}\b(?:return|reply|ping|come\s+back|continue|(?:add|build|draft|write)\w*[\s\S]{0,60}(?:content|lea(?:f|ves)|phase\s*2))\b/iu.test( - affirmativeText, - ) || - /\b(?:return|reply|ping|come\s+back)\b[\s\S]{0,160}\b(?:after|once|when)\b[\s\S]{0,160}\bmerge\w*\b/iu.test( - affirmativeText, - ); - return stagedPrObserved && seedContinuationAfterMergeObserved; } function githubAppRequirementObserved(text: string): boolean { From 3220f37c90d059ef57bdd872627e080e16b22cb5 Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 20:53:00 +0800 Subject: [PATCH 07/18] test: preserve seed handoff polarity --- .../first-tree-seed/__tests__/grader.test.ts | 63 +++++++++++++++++++ .../src/suites/first-tree-seed/grader.ts | 21 ++++--- 2 files changed, 74 insertions(+), 10 deletions(-) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index b4469f386..16a08cf4e 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -442,6 +442,69 @@ describe("first-tree-seed grader", () => { } }); + it("preserves contrastive rejection of the retired structure-PR handoff", () => { + const responses = [ + "Rather than open a structure PR and wait for you to merge it before I add leaves, I will build structure and initial leaves in one seed PR.", + "Instead of opening a structure PR and waiting for you to merge it before I add leaves, I will build structure and initial leaves in one seed PR.", + ]; + + for (const [index, finalResponse] of responses.entries()) { + const tempRoot = mkdtempSync(join(tmpdir(), `seed-eval-single-pr-contrastive-negation-${index}-`)); + try { + const metrics = deriveMetrics( + [ + { + event: { item: { text: finalResponse, type: "agent_message" }, type: "item.completed" }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.singlePrBuildObserved, finalResponse).toBe(true); + expect(metrics.legacyHandoffObserved, finalResponse).toBe(false); + expect(metrics.forbiddenActionHits, finalResponse).toEqual([]); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + } + }); + + it("treats a structure PR landing before leaves as the retired handoff", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-lands-handoff-")); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will deliver one PR for structure and leaves. First I will open a structure PR; once it lands, I will add leaves.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.singlePrBuildObserved).toBe(true); + expect(metrics.legacyHandoffObserved).toBe(true); + expect(metrics.forbiddenActionHits).toContain("legacy_two_pr_handoff"); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + it("rejects a structure-first PR whose merge gates leaf drafting", () => { const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-structure-first-")); try { diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index 8a17221e5..b18c9d24d 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1136,6 +1136,7 @@ function singlePrBuildObserved(text: string): boolean { function legacyHandoffObserved(text: string): boolean { const affirmativeText = text + .replace(/\b(?:rather\s+than|instead\s+of)\b[^,.;!?\n]*(?:,|(?=[.;!?\n]|$))/giu, "") .replace( /\b(?:do not|don't|will not|won't|must not|should not|never)\s+(?:open|create|submit|raise|wait|ask|require)\b[^.!?;\n]*/giu, "", @@ -1163,23 +1164,23 @@ function legacyHandoffObserved(text: string): boolean { if (!structurePrObserved) return false; const handoffContext = clauses.slice(index, index + 4).join(". "); - const mergeMatch = /\bmerge\w*\b/iu.exec(handoffContext); - if (!mergeMatch) return false; - const beforeMerge = handoffContext.slice(0, mergeMatch.index); + const handoffMatch = /\b(?:merge\w*|land(?:s|ed|ing)?)\b/iu.exec(handoffContext); + if (!handoffMatch) return false; + const beforeHandoff = handoffContext.slice(0, handoffMatch.index); const samePrIncludesLeaves = /\b(?:pr|pull\s+request)\b[\s\S]{0,100}\b(?:with|for|containing|covering)\b[\s\S]{0,80}\b(?:structure|skeleton)\b[\s\S]{0,50}\b(?:and|plus|alongside|together\s+with)\b[\s\S]{0,50}\b(?:content|lea(?:f|ves))\b/iu.test( - beforeMerge, + beforeHandoff, ) || /\b(?:it|this\s+(?:seed\s+)?(?:pr|pull\s+request)|that\s+(?:seed\s+)?(?:pr|pull\s+request)|the\s+same\s+(?:seed\s+)?(?:pr|pull\s+request))\b[\s\S]{0,80}\b(?:contain\w*|includ\w*|cover\w*|carr\w*|hold\w*)\b[\s\S]{0,50}\b(?:content|lea(?:f|ves))\b/iu.test( - beforeMerge, + beforeHandoff, ); if (samePrIncludesLeaves) return false; - const afterMerge = handoffContext.slice(mergeMatch.index + mergeMatch[0].length); - if (/\b(?:return|reply|ping|come\s+back|continue)\b/iu.test(afterMerge)) return true; - if (/\b(?:lea(?:f|ves)|phase\s*2)\b/iu.test(afterMerge)) return true; - if (!/\bcontent\b/iu.test(afterMerge)) return false; - return !/\b(?:first-tree-write|future|follow-up|subsequent|later\s+source-backed)\b/iu.test(afterMerge); + const afterHandoff = handoffContext.slice(handoffMatch.index + handoffMatch[0].length); + if (/\b(?:return|reply|ping|come\s+back|continue)\b/iu.test(afterHandoff)) return true; + if (/\b(?:lea(?:f|ves)|phase\s*2)\b/iu.test(afterHandoff)) return true; + if (!/\bcontent\b/iu.test(afterHandoff)) return false; + return !/\b(?:first-tree-write|future|follow-up|subsequent|later\s+source-backed)\b/iu.test(afterHandoff); }); } From 61057a3e8462136effe1867dc76aae38166a60b1 Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 20:56:21 +0800 Subject: [PATCH 08/18] test: bind seed landing to PR referents --- .../first-tree-seed/__tests__/grader.test.ts | 61 +++++++++++++++++++ .../src/suites/first-tree-seed/grader.ts | 5 +- 2 files changed, 65 insertions(+), 1 deletion(-) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index 16a08cf4e..6734c2322 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -505,6 +505,67 @@ describe("first-tree-seed grader", () => { } }); + it("does not treat sub-agent drafts landing as the seed PR handoff", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-drafts-land-")); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will open one seed PR with the structure. Once the sub-agent drafts land, I will add the initial leaves to that same PR before you merge it.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.singlePrBuildObserved).toBe(true); + expect(metrics.legacyHandoffObserved).toBe(false); + expect(metrics.forbiddenActionHits).toEqual([]); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + + it("treats an explicit pull-request landing before leaves as the retired handoff", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-explicit-pr-lands-")); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will open a pull request with the structure. Once that pull request lands, I will add the leaves.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.legacyHandoffObserved).toBe(true); + expect(metrics.forbiddenActionHits).toContain("legacy_two_pr_handoff"); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + it("rejects a structure-first PR whose merge gates leaf drafting", () => { const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-structure-first-")); try { diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index b18c9d24d..7283b6773 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1164,7 +1164,10 @@ function legacyHandoffObserved(text: string): boolean { if (!structurePrObserved) return false; const handoffContext = clauses.slice(index, index + 4).join(". "); - const handoffMatch = /\b(?:merge\w*|land(?:s|ed|ing)?)\b/iu.exec(handoffContext); + const handoffMatch = + /\b(?:merge\w*|(?:it|(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request))\s+land(?:s|ed|ing)?)\b/iu.exec( + handoffContext, + ); if (!handoffMatch) return false; const beforeHandoff = handoffContext.slice(0, handoffMatch.index); const samePrIncludesLeaves = From 41f2be9a6c68ddb5a9ada92360fd5a33b8637fd3 Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 21:00:05 +0800 Subject: [PATCH 09/18] test: bind seed merge to PR referents --- .../first-tree-seed/__tests__/grader.test.ts | 62 +++++++++++++++++++ .../src/suites/first-tree-seed/grader.ts | 5 +- 2 files changed, 66 insertions(+), 1 deletion(-) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index 6734c2322..f50a8ec5c 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -566,6 +566,68 @@ describe("first-tree-seed grader", () => { } }); + it("does not treat merging sub-agent work as the seed PR handoff", () => { + const responses = [ + "I will open one seed PR with the structure. Once I merge the sub-agent drafts, I will add the initial leaves to that same PR before you merge it.", + "I will open one seed PR with the structure. Once the sub-agent branches merge, I will add the initial leaves to that same PR before you merge it.", + ]; + + for (const [index, finalResponse] of responses.entries()) { + const tempRoot = mkdtempSync(join(tmpdir(), `seed-eval-single-pr-merge-sub-agent-work-${index}-`)); + try { + const metrics = deriveMetrics( + [ + { + event: { item: { text: finalResponse, type: "agent_message" }, type: "item.completed" }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.singlePrBuildObserved, finalResponse).toBe(true); + expect(metrics.legacyHandoffObserved, finalResponse).toBe(false); + expect(metrics.forbiddenActionHits, finalResponse).toEqual([]); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + } + }); + + it("treats merging that structure PR before leaves as the retired handoff", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-merge-structure-pr-")); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will open a structure PR now. Once you merge that structure PR, I will add the leaves.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.legacyHandoffObserved).toBe(true); + expect(metrics.forbiddenActionHits).toContain("legacy_two_pr_handoff"); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + it("rejects a structure-first PR whose merge gates leaf drafting", () => { const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-structure-first-")); try { diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index 7283b6773..422a8fc20 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1165,7 +1165,7 @@ function legacyHandoffObserved(text: string): boolean { const handoffContext = clauses.slice(index, index + 4).join(". "); const handoffMatch = - /\b(?:merge\w*|(?:it|(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request))\s+land(?:s|ed|ing)?)\b/iu.exec( + /(?:\b(?:it|(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request))\b\s+(?:(?:is|gets|was|has\s+been|will\s+be)\s+)?(?:merge\w*|land(?:s|ed|ing)?)\b)|(?:\b(?:merge\w*|land(?:s|ed|ing)?)\b\s+(?:it|(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request))\b)|(?:\b(?:once|after|when)\s+merged\b)/iu.exec( handoffContext, ); if (!handoffMatch) return false; @@ -1176,6 +1176,9 @@ function legacyHandoffObserved(text: string): boolean { ) || /\b(?:it|this\s+(?:seed\s+)?(?:pr|pull\s+request)|that\s+(?:seed\s+)?(?:pr|pull\s+request)|the\s+same\s+(?:seed\s+)?(?:pr|pull\s+request))\b[\s\S]{0,80}\b(?:contain\w*|includ\w*|cover\w*|carr\w*|hold\w*)\b[\s\S]{0,50}\b(?:content|lea(?:f|ves))\b/iu.test( beforeHandoff, + ) || + /\b(?:content|lea(?:f|ves))\b[\s\S]{0,60}\b(?:to|in|into|on)\b[\s\S]{0,40}\b(?:it|this\s+(?:seed\s+)?(?:pr|pull\s+request)|that\s+same\s+(?:seed\s+)?(?:pr|pull\s+request)|the\s+same\s+(?:seed\s+)?(?:pr|pull\s+request))\b/iu.test( + beforeHandoff, ); if (samePrIncludesLeaves) return false; From a8b6728c9141df7c9f4303272ebd68b872e1c1c0 Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 21:09:10 +0800 Subject: [PATCH 10/18] test: disambiguate passive seed handoffs --- .../first-tree-seed/__tests__/grader.test.ts | 33 +++++++++++++++++++ .../src/suites/first-tree-seed/grader.ts | 31 +++++++++++++---- 2 files changed, 58 insertions(+), 6 deletions(-) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index f50a8ec5c..19972d1f5 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -570,6 +570,8 @@ describe("first-tree-seed grader", () => { const responses = [ "I will open one seed PR with the structure. Once I merge the sub-agent drafts, I will add the initial leaves to that same PR before you merge it.", "I will open one seed PR with the structure. Once the sub-agent branches merge, I will add the initial leaves to that same PR before you merge it.", + "I will open one seed PR with the structure. The sub-agent branches are ready; once merged, I will add the initial leaves to that same PR before you merge it.", + "I will open one seed PR with the structure. The sub-agent drafts stay on local branches; once merged, I will add their initial leaves to that same PR before you merge it.", ]; for (const [index, finalResponse] of responses.entries()) { @@ -628,6 +630,37 @@ describe("first-tree-seed grader", () => { } }); + it("treats a structure PR that has landed before leaves as the retired handoff", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-has-landed-")); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will deliver one PR for structure and leaves. I will open a structure PR; once that PR has landed, I will add the leaves.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.singlePrBuildObserved).toBe(true); + expect(metrics.legacyHandoffObserved).toBe(true); + expect(metrics.forbiddenActionHits).toContain("legacy_two_pr_handoff"); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + it("rejects a structure-first PR whose merge gates leaf drafting", () => { const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-structure-first-")); try { diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index 422a8fc20..ee5ee21ac 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1163,13 +1163,32 @@ function legacyHandoffObserved(text: string): boolean { ); if (!structurePrObserved) return false; - const handoffContext = clauses.slice(index, index + 4).join(". "); - const handoffMatch = - /(?:\b(?:it|(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request))\b\s+(?:(?:is|gets|was|has\s+been|will\s+be)\s+)?(?:merge\w*|land(?:s|ed|ing)?)\b)|(?:\b(?:merge\w*|land(?:s|ed|ing)?)\b\s+(?:it|(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request))\b)|(?:\b(?:once|after|when)\s+merged\b)/iu.exec( + const localClauses = clauses.slice(index, index + 4); + const handoffContext = localClauses.join(". "); + const explicitHandoffMatch = + /(?:\b(?:it|(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request))\b\s+(?:(?:(?:has|have|had)\s+been|will\s+be|is|gets|was|has|have|had)\s+)?(?:merge\w*|land(?:s|ed|ing)?)\b)|(?:\b(?:merge\w*|land(?:s|ed|ing)?)\b\s+(?:it|(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request))\b)/iu.exec( handoffContext, ); - if (!handoffMatch) return false; - const beforeHandoff = handoffContext.slice(0, handoffMatch.index); + let handoffIndex = explicitHandoffMatch?.index ?? Number.POSITIVE_INFINITY; + let handoffLength = explicitHandoffMatch?.[0].length ?? 0; + for (const localIndex of [0, 1, 2]) { + const passiveClause = localClauses[localIndex]; + if (!passiveClause) continue; + const passiveMatch = /\b(?:once|after|when)\s+merged\b/iu.exec(passiveClause); + if (!passiveMatch) continue; + const isBoundToCandidate = + localIndex === 0 || + (localIndex === 1 && passiveMatch.index === 0) || + (localIndex === 2 && passiveMatch.index === 0 && /\b(?:pr|pull\s+request)\b/iu.test(localClauses[1] ?? "")); + if (!isBoundToCandidate) continue; + const prefixLength = localClauses.slice(0, localIndex).join(". ").length + (localIndex > 0 ? 2 : 0); + const passiveIndex = prefixLength + passiveMatch.index; + if (passiveIndex >= handoffIndex) continue; + handoffIndex = passiveIndex; + handoffLength = passiveMatch[0].length; + } + if (!Number.isFinite(handoffIndex)) return false; + const beforeHandoff = handoffContext.slice(0, handoffIndex); const samePrIncludesLeaves = /\b(?:pr|pull\s+request)\b[\s\S]{0,100}\b(?:with|for|containing|covering)\b[\s\S]{0,80}\b(?:structure|skeleton)\b[\s\S]{0,50}\b(?:and|plus|alongside|together\s+with)\b[\s\S]{0,50}\b(?:content|lea(?:f|ves))\b/iu.test( beforeHandoff, @@ -1182,7 +1201,7 @@ function legacyHandoffObserved(text: string): boolean { ); if (samePrIncludesLeaves) return false; - const afterHandoff = handoffContext.slice(handoffMatch.index + handoffMatch[0].length); + const afterHandoff = handoffContext.slice(handoffIndex + handoffLength); if (/\b(?:return|reply|ping|come\s+back|continue)\b/iu.test(afterHandoff)) return true; if (/\b(?:lea(?:f|ves)|phase\s*2)\b/iu.test(afterHandoff)) return true; if (!/\bcontent\b/iu.test(afterHandoff)) return false; From a039c24087f6a46838f7a2760bc0a79001019288 Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 21:15:07 +0800 Subject: [PATCH 11/18] test: carry seed PR referents through review prose --- .../first-tree-seed/__tests__/grader.test.ts | 31 +++++++++++++++++ .../src/suites/first-tree-seed/grader.ts | 33 +++++++++++++++++-- 2 files changed, 61 insertions(+), 3 deletions(-) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index 19972d1f5..7593a4b99 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -661,6 +661,37 @@ describe("first-tree-seed grader", () => { } }); + it("preserves a structure PR referent through review prose before a passive merge", () => { + const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-review-pronoun-")); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review it first. Once merged, I will add the leaves.", + type: "agent_message", + }, + type: "item.completed", + }, + type: "codex_event", + }, + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); + + expect(metrics.singlePrBuildObserved).toBe(true); + expect(metrics.legacyHandoffObserved).toBe(true); + expect(metrics.forbiddenActionHits).toContain("legacy_two_pr_handoff"); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } + }); + it("rejects a structure-first PR whose merge gates leaf drafting", () => { const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-structure-first-")); try { diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index ee5ee21ac..e7bae0e67 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1134,6 +1134,35 @@ function singlePrBuildObserved(text: string): boolean { return actionObserved && singlePrObserved && structureAndLeavesObserved; } +function passiveHandoffRefersToCandidatePr(localClauses: string[], passiveClauseIndex: number): boolean { + if (passiveClauseIndex === 0 || passiveClauseIndex === 1) return true; + + let candidatePrIsCurrentReferent = true; + let candidatePrCoreferenceObserved = false; + for (const interveningClause of localClauses.slice(1, passiveClauseIndex)) { + if ( + /\b(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request)\b/iu.test(interveningClause) + ) { + candidatePrIsCurrentReferent = true; + candidatePrCoreferenceObserved = true; + continue; + } + if ( + /\b(?:sub-?agent|agent)\b[^.!?;\n]{0,50}\b(?:branches?|changes?|drafts?|work)\b|\b(?:branches?|drafts?)\b/iu.test( + interveningClause, + ) + ) { + candidatePrIsCurrentReferent = false; + continue; + } + if (/\bit\b/iu.test(interveningClause) && candidatePrIsCurrentReferent) { + candidatePrCoreferenceObserved = true; + } + } + + return candidatePrIsCurrentReferent && candidatePrCoreferenceObserved; +} + function legacyHandoffObserved(text: string): boolean { const affirmativeText = text .replace(/\b(?:rather\s+than|instead\s+of)\b[^,.;!?\n]*(?:,|(?=[.;!?\n]|$))/giu, "") @@ -1177,9 +1206,7 @@ function legacyHandoffObserved(text: string): boolean { const passiveMatch = /\b(?:once|after|when)\s+merged\b/iu.exec(passiveClause); if (!passiveMatch) continue; const isBoundToCandidate = - localIndex === 0 || - (localIndex === 1 && passiveMatch.index === 0) || - (localIndex === 2 && passiveMatch.index === 0 && /\b(?:pr|pull\s+request)\b/iu.test(localClauses[1] ?? "")); + localIndex === 0 || (passiveMatch.index === 0 && passiveHandoffRefersToCandidatePr(localClauses, localIndex)); if (!isBoundToCandidate) continue; const prefixLength = localClauses.slice(0, localIndex).join(". ").length + (localIndex > 0 ? 2 : 0); const passiveIndex = prefixLength + passiveMatch.index; From 28e637426c1b84ea4ec353c0e16b754840103579 Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 21:18:39 +0800 Subject: [PATCH 12/18] test: track seed handoff referents across punctuation --- .../first-tree-seed/__tests__/grader.test.ts | 1 + .../src/suites/first-tree-seed/grader.ts | 65 ++++++++++++------- 2 files changed, 44 insertions(+), 22 deletions(-) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index 7593a4b99..6892a8f10 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -572,6 +572,7 @@ describe("first-tree-seed grader", () => { "I will open one seed PR with the structure. Once the sub-agent branches merge, I will add the initial leaves to that same PR before you merge it.", "I will open one seed PR with the structure. The sub-agent branches are ready; once merged, I will add the initial leaves to that same PR before you merge it.", "I will open one seed PR with the structure. The sub-agent drafts stay on local branches; once merged, I will add their initial leaves to that same PR before you merge it.", + "I will open one seed PR with the structure, and the sub-agent branches are ready. Once merged, I will add the initial leaves to that same PR before you merge it.", ]; for (const [index, finalResponse] of responses.entries()) { diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index e7bae0e67..7a332927d 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1134,33 +1134,54 @@ function singlePrBuildObserved(text: string): boolean { return actionObserved && singlePrObserved && structureAndLeavesObserved; } -function passiveHandoffRefersToCandidatePr(localClauses: string[], passiveClauseIndex: number): boolean { - if (passiveClauseIndex === 0 || passiveClauseIndex === 1) return true; +function passiveHandoffRefersToCandidatePr( + localClauses: string[], + passiveClauseIndex: number, + passiveMatchIndex: number, +): boolean { + const candidateClause = localClauses[0] ?? ""; + const prReferences = [ + ...candidateClause.matchAll( + /\b(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request)\b/giu, + ), + ]; + const candidateReference = prReferences.at(-1); + if (!candidateReference || candidateReference.index === undefined) return false; let candidatePrIsCurrentReferent = true; let candidatePrCoreferenceObserved = false; - for (const interveningClause of localClauses.slice(1, passiveClauseIndex)) { - if ( - /\b(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request)\b/iu.test(interveningClause) - ) { - candidatePrIsCurrentReferent = true; - candidatePrCoreferenceObserved = true; - continue; - } - if ( - /\b(?:sub-?agent|agent)\b[^.!?;\n]{0,50}\b(?:branches?|changes?|drafts?|work)\b|\b(?:branches?|drafts?)\b/iu.test( - interveningClause, - ) - ) { - candidatePrIsCurrentReferent = false; - continue; - } - if (/\bit\b/iu.test(interveningClause) && candidatePrIsCurrentReferent) { - candidatePrCoreferenceObserved = true; + const candidateClauseEnd = passiveClauseIndex === 0 ? passiveMatchIndex : candidateClause.length; + const segments = [ + candidateClause.slice(candidateReference.index + candidateReference[0].length, candidateClauseEnd), + ...localClauses.slice(1, passiveClauseIndex), + ]; + for (const segment of segments) { + const referentEvents = [ + ...[...segment.matchAll(/\b(?:sub-?agent|agent)\b[^.!?;\n]{0,50}\b(?:branches?|changes?|drafts?|work)\b/giu)].map( + (match) => ({ index: match.index, type: "competing" as const }), + ), + ...[ + ...segment.matchAll( + /\b(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request)\b/giu, + ), + ].map((match) => ({ index: match.index, type: "candidate" as const })), + ...[...segment.matchAll(/\bit\b/giu)].map((match) => ({ index: match.index, type: "pronoun" as const })), + ].sort((left, right) => left.index - right.index); + + for (const event of referentEvents) { + if (event.type === "competing") { + candidatePrIsCurrentReferent = false; + } else if (event.type === "candidate") { + candidatePrIsCurrentReferent = true; + candidatePrCoreferenceObserved = true; + } else if (candidatePrIsCurrentReferent) { + candidatePrCoreferenceObserved = true; + } } } - return candidatePrIsCurrentReferent && candidatePrCoreferenceObserved; + if (!candidatePrIsCurrentReferent) return false; + return passiveClauseIndex <= 1 || candidatePrCoreferenceObserved; } function legacyHandoffObserved(text: string): boolean { @@ -1206,7 +1227,7 @@ function legacyHandoffObserved(text: string): boolean { const passiveMatch = /\b(?:once|after|when)\s+merged\b/iu.exec(passiveClause); if (!passiveMatch) continue; const isBoundToCandidate = - localIndex === 0 || (passiveMatch.index === 0 && passiveHandoffRefersToCandidatePr(localClauses, localIndex)); + passiveMatch.index === 0 && passiveHandoffRefersToCandidatePr(localClauses, localIndex, passiveMatch.index); if (!isBoundToCandidate) continue; const prefixLength = localClauses.slice(0, localIndex).join(". ").length + (localIndex > 0 ? 2 : 0); const passiveIndex = prefixLength + passiveMatch.index; From 0143db3a4d9ff78eb26c2389bb235efdaaf66961 Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 21:21:32 +0800 Subject: [PATCH 13/18] test: scan complete seed handoff context --- .../first-tree-seed/__tests__/grader.test.ts | 56 +++++++++++-------- .../src/suites/first-tree-seed/grader.ts | 9 +-- 2 files changed, 37 insertions(+), 28 deletions(-) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index 6892a8f10..62f79297e 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -663,33 +663,41 @@ describe("first-tree-seed grader", () => { }); it("preserves a structure PR referent through review prose before a passive merge", () => { - const tempRoot = mkdtempSync(join(tmpdir(), "seed-eval-single-pr-review-pronoun-")); - try { - const metrics = deriveMetrics( - [ - { - event: { - item: { - text: "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review it first. Once merged, I will add the leaves.", - type: "agent_message", + const responses = [ + "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review it first. Once merged, I will add the leaves.", + "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can inspect it first. You can approve it after that. Once merged, I will add the leaves.", + "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review this first. Once merged, I will add the leaves.", + ]; + + for (const [index, finalResponse] of responses.entries()) { + const tempRoot = mkdtempSync(join(tmpdir(), `seed-eval-single-pr-review-pronoun-${index}-`)); + try { + const metrics = deriveMetrics( + [ + { + event: { + item: { + text: finalResponse, + type: "agent_message", + }, + type: "item.completed", }, - type: "item.completed", + type: "codex_event", }, - type: "codex_event", - }, - ], - findCase("same-chat-approved-skeleton-builds-single-pr"), - fixtureValidation(), - 0, - baseRunPaths(tempRoot), - join(tempRoot, "context-tree"), - ); + ], + findCase("same-chat-approved-skeleton-builds-single-pr"), + fixtureValidation(), + 0, + baseRunPaths(tempRoot), + join(tempRoot, "context-tree"), + ); - expect(metrics.singlePrBuildObserved).toBe(true); - expect(metrics.legacyHandoffObserved).toBe(true); - expect(metrics.forbiddenActionHits).toContain("legacy_two_pr_handoff"); - } finally { - rmSync(tempRoot, { force: true, recursive: true }); + expect(metrics.singlePrBuildObserved, finalResponse).toBe(true); + expect(metrics.legacyHandoffObserved, finalResponse).toBe(true); + expect(metrics.forbiddenActionHits, finalResponse).toContain("legacy_two_pr_handoff"); + } finally { + rmSync(tempRoot, { force: true, recursive: true }); + } } }); diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index 7a332927d..dbf80e1cc 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1165,7 +1165,10 @@ function passiveHandoffRefersToCandidatePr( /\b(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request)\b/giu, ), ].map((match) => ({ index: match.index, type: "candidate" as const })), - ...[...segment.matchAll(/\bit\b/giu)].map((match) => ({ index: match.index, type: "pronoun" as const })), + ...[...segment.matchAll(/\b(?:it|that|this)\b/giu)].map((match) => ({ + index: match.index, + type: "pronoun" as const, + })), ].sort((left, right) => left.index - right.index); for (const event of referentEvents) { @@ -1221,9 +1224,7 @@ function legacyHandoffObserved(text: string): boolean { ); let handoffIndex = explicitHandoffMatch?.index ?? Number.POSITIVE_INFINITY; let handoffLength = explicitHandoffMatch?.[0].length ?? 0; - for (const localIndex of [0, 1, 2]) { - const passiveClause = localClauses[localIndex]; - if (!passiveClause) continue; + for (const [localIndex, passiveClause] of localClauses.entries()) { const passiveMatch = /\b(?:once|after|when)\s+merged\b/iu.exec(passiveClause); if (!passiveMatch) continue; const isBoundToCandidate = From 5703138592efae040e30f9bfa36b89d4c0c99101 Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 21:28:20 +0800 Subject: [PATCH 14/18] test: distinguish seed PR demonstratives from branches --- .../first-tree-seed/__tests__/grader.test.ts | 1 + .../src/suites/first-tree-seed/grader.ts | 15 ++++++++------- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index 62f79297e..3c49254a9 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -573,6 +573,7 @@ describe("first-tree-seed grader", () => { "I will open one seed PR with the structure. The sub-agent branches are ready; once merged, I will add the initial leaves to that same PR before you merge it.", "I will open one seed PR with the structure. The sub-agent drafts stay on local branches; once merged, I will add their initial leaves to that same PR before you merge it.", "I will open one seed PR with the structure, and the sub-agent branches are ready. Once merged, I will add the initial leaves to that same PR before you merge it.", + "I will open one seed PR with the structure. I will review this branch first. Once merged, I will add the initial leaves to that same PR before you merge it.", ]; for (const [index, finalResponse] of responses.entries()) { diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index dbf80e1cc..1c6e9048a 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1157,18 +1157,19 @@ function passiveHandoffRefersToCandidatePr( ]; for (const segment of segments) { const referentEvents = [ - ...[...segment.matchAll(/\b(?:sub-?agent|agent)\b[^.!?;\n]{0,50}\b(?:branches?|changes?|drafts?|work)\b/giu)].map( - (match) => ({ index: match.index, type: "competing" as const }), - ), + ...[ + ...segment.matchAll( + /\b(?:sub-?agent|agent)\b[^.!?;\n]{0,50}\b(?:branch(?:es)?|changes?|drafts?|work)\b|\b(?:(?:a(?:n)?|the|this|that)\s+)?(?:branch(?:es)?|drafts?)\b/giu, + ), + ].map((match) => ({ index: match.index, type: "competing" as const })), ...[ ...segment.matchAll( /\b(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request)\b/giu, ), ].map((match) => ({ index: match.index, type: "candidate" as const })), - ...[...segment.matchAll(/\b(?:it|that|this)\b/giu)].map((match) => ({ - index: match.index, - type: "pronoun" as const, - })), + ...[...segment.matchAll(/\bit\b|\b(?:that|this)\b(?!\s+(?:branch(?:es)?|changes?|drafts?|work)\b)/giu)].map( + (match) => ({ index: match.index, type: "pronoun" as const }), + ), ].sort((left, right) => left.index - right.index); for (const event of referentEvents) { From f423b45662c929f6c037ca87e1414d85ca611044 Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 21:30:24 +0800 Subject: [PATCH 15/18] test: track demonstrative seed work referents --- .../src/suites/first-tree-seed/__tests__/grader.test.ts | 2 ++ packages/skill-evals/src/suites/first-tree-seed/grader.ts | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index 3c49254a9..dbf6550db 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -574,6 +574,8 @@ describe("first-tree-seed grader", () => { "I will open one seed PR with the structure. The sub-agent drafts stay on local branches; once merged, I will add their initial leaves to that same PR before you merge it.", "I will open one seed PR with the structure, and the sub-agent branches are ready. Once merged, I will add the initial leaves to that same PR before you merge it.", "I will open one seed PR with the structure. I will review this branch first. Once merged, I will add the initial leaves to that same PR before you merge it.", + "I will open one seed PR with the structure. That work came from the sub-agent. Once merged, I will add the initial leaves to that same PR before you merge it.", + "I will open one seed PR with the structure. This branch came from the sub-agent. Once merged, I will add the initial leaves to that same PR before you merge it.", ]; for (const [index, finalResponse] of responses.entries()) { diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index 1c6e9048a..9f51a3d79 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1159,7 +1159,7 @@ function passiveHandoffRefersToCandidatePr( const referentEvents = [ ...[ ...segment.matchAll( - /\b(?:sub-?agent|agent)\b[^.!?;\n]{0,50}\b(?:branch(?:es)?|changes?|drafts?|work)\b|\b(?:(?:a(?:n)?|the|this|that)\s+)?(?:branch(?:es)?|drafts?)\b/giu, + /\b(?:sub-?agent|agent)\b[^.!?;\n]{0,50}\b(?:branch(?:es)?|changes?|drafts?|work)\b|\b(?:(?:a(?:n)?|the|this|that)\s+)?(?:branch(?:es)?|drafts?)\b|\b(?:the|this|that)\s+(?:changes?|work)\b/giu, ), ].map((match) => ({ index: match.index, type: "competing" as const })), ...[ From 664a02cfa2026fa898e6bb897d766b5cd3e7c8ea Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 21:32:57 +0800 Subject: [PATCH 16/18] test: require provenance for internal seed work --- .../first-tree-seed/__tests__/grader.test.ts | 2 ++ .../src/suites/first-tree-seed/grader.ts | 15 ++++++++++----- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index dbf6550db..0bc3670c9 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -575,6 +575,7 @@ describe("first-tree-seed grader", () => { "I will open one seed PR with the structure, and the sub-agent branches are ready. Once merged, I will add the initial leaves to that same PR before you merge it.", "I will open one seed PR with the structure. I will review this branch first. Once merged, I will add the initial leaves to that same PR before you merge it.", "I will open one seed PR with the structure. That work came from the sub-agent. Once merged, I will add the initial leaves to that same PR before you merge it.", + "I will open one seed PR with the structure, and that work came from the sub-agent. Once merged, I will add the initial leaves to that same PR before you merge it.", "I will open one seed PR with the structure. This branch came from the sub-agent. Once merged, I will add the initial leaves to that same PR before you merge it.", ]; @@ -670,6 +671,7 @@ describe("first-tree-seed grader", () => { "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review it first. Once merged, I will add the leaves.", "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can inspect it first. You can approve it after that. Once merged, I will add the leaves.", "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review this first. Once merged, I will add the leaves.", + "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review this work first. Once merged, I will add the leaves.", ]; for (const [index, finalResponse] of responses.entries()) { diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index 9f51a3d79..853b9c40f 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1159,7 +1159,7 @@ function passiveHandoffRefersToCandidatePr( const referentEvents = [ ...[ ...segment.matchAll( - /\b(?:sub-?agent|agent)\b[^.!?;\n]{0,50}\b(?:branch(?:es)?|changes?|drafts?|work)\b|\b(?:(?:a(?:n)?|the|this|that)\s+)?(?:branch(?:es)?|drafts?)\b|\b(?:the|this|that)\s+(?:changes?|work)\b/giu, + /\b(?:sub-?agent|agent)\b[^.!?;\n]{0,50}\b(?:branch(?:es)?|changes?|drafts?|work)\b|\b(?:(?:a(?:n)?|the|this|that)\s+)?(?:branch(?:es)?|drafts?)\b|\b(?:the|this|that)\s+(?:changes?|work)\b[^.!?;\n]{0,50}\b(?:came|comes?|originat\w*|is|was|were)\s+from\s+(?:the\s+)?(?:sub-?agent|agent)\b/giu, ), ].map((match) => ({ index: match.index, type: "competing" as const })), ...[ @@ -1167,10 +1167,15 @@ function passiveHandoffRefersToCandidatePr( /\b(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request)\b/giu, ), ].map((match) => ({ index: match.index, type: "candidate" as const })), - ...[...segment.matchAll(/\bit\b|\b(?:that|this)\b(?!\s+(?:branch(?:es)?|changes?|drafts?|work)\b)/giu)].map( - (match) => ({ index: match.index, type: "pronoun" as const }), - ), - ].sort((left, right) => left.index - right.index); + ...[...segment.matchAll(/\b(?:it|that|this)\b/giu)].map((match) => ({ + index: match.index, + type: "pronoun" as const, + })), + ].sort((left, right) => { + if (left.index !== right.index) return left.index - right.index; + const priority = { candidate: 1, competing: 0, pronoun: 2 } as const; + return priority[left.type] - priority[right.type]; + }); for (const event of referentEvents) { if (event.type === "competing") { From 4c2813a1d193c8ea0693635ed4c288a0cd2042a9 Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 21:35:27 +0800 Subject: [PATCH 17/18] test: preserve definite seed work references --- .../src/suites/first-tree-seed/__tests__/grader.test.ts | 1 + packages/skill-evals/src/suites/first-tree-seed/grader.ts | 4 ++++ 2 files changed, 5 insertions(+) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index 0bc3670c9..b1aaf35d6 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -672,6 +672,7 @@ describe("first-tree-seed grader", () => { "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can inspect it first. You can approve it after that. Once merged, I will add the leaves.", "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review this first. Once merged, I will add the leaves.", "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review this work first. Once merged, I will add the leaves.", + "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review the work first. Once merged, I will add the leaves.", ]; for (const [index, finalResponse] of responses.entries()) { diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index 853b9c40f..bfc2b0833 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1171,6 +1171,10 @@ function passiveHandoffRefersToCandidatePr( index: match.index, type: "pronoun" as const, })), + ...[...segment.matchAll(/\bthe\s+(?:changes?|work)\b/giu)].map((match) => ({ + index: match.index, + type: "pronoun" as const, + })), ].sort((left, right) => { if (left.index !== right.index) return left.index - right.index; const priority = { candidate: 1, competing: 0, pronoun: 2 } as const; From c272c64e5fc6f535785d4a6a0fab57c76dd8fafe Mon Sep 17 00:00:00 2001 From: Gandy2025 Date: Tue, 14 Jul 2026 21:38:10 +0800 Subject: [PATCH 18/18] test: cover plural seed content references --- .../src/suites/first-tree-seed/__tests__/grader.test.ts | 4 ++++ packages/skill-evals/src/suites/first-tree-seed/grader.ts | 6 +++--- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts index b1aaf35d6..52ce141be 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/__tests__/grader.test.ts @@ -577,6 +577,8 @@ describe("first-tree-seed grader", () => { "I will open one seed PR with the structure. That work came from the sub-agent. Once merged, I will add the initial leaves to that same PR before you merge it.", "I will open one seed PR with the structure, and that work came from the sub-agent. Once merged, I will add the initial leaves to that same PR before you merge it.", "I will open one seed PR with the structure. This branch came from the sub-agent. Once merged, I will add the initial leaves to that same PR before you merge it.", + "I will open one seed PR with the structure. These changes came from the sub-agent. Once merged, I will add the initial leaves to that same PR before you merge it.", + "I will open one seed PR with the structure. Those changes came from the sub-agent. Once merged, I will add the initial leaves to that same PR before you merge it.", ]; for (const [index, finalResponse] of responses.entries()) { @@ -673,6 +675,8 @@ describe("first-tree-seed grader", () => { "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review this first. Once merged, I will add the leaves.", "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review this work first. Once merged, I will add the leaves.", "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review the work first. Once merged, I will add the leaves.", + "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review these changes first. Once merged, I will add the leaves.", + "I will deliver one PR for structure and leaves. I will open a structure PR for review. You can review those changes first. Once merged, I will add the leaves.", ]; for (const [index, finalResponse] of responses.entries()) { diff --git a/packages/skill-evals/src/suites/first-tree-seed/grader.ts b/packages/skill-evals/src/suites/first-tree-seed/grader.ts index bfc2b0833..42f065883 100644 --- a/packages/skill-evals/src/suites/first-tree-seed/grader.ts +++ b/packages/skill-evals/src/suites/first-tree-seed/grader.ts @@ -1159,7 +1159,7 @@ function passiveHandoffRefersToCandidatePr( const referentEvents = [ ...[ ...segment.matchAll( - /\b(?:sub-?agent|agent)\b[^.!?;\n]{0,50}\b(?:branch(?:es)?|changes?|drafts?|work)\b|\b(?:(?:a(?:n)?|the|this|that)\s+)?(?:branch(?:es)?|drafts?)\b|\b(?:the|this|that)\s+(?:changes?|work)\b[^.!?;\n]{0,50}\b(?:came|comes?|originat\w*|is|was|were)\s+from\s+(?:the\s+)?(?:sub-?agent|agent)\b/giu, + /\b(?:sub-?agent|agent)\b[^.!?;\n]{0,50}\b(?:branch(?:es)?|changes?|drafts?|work)\b|\b(?:(?:a(?:n)?|the|this|that|these|those)\s+)?(?:branch(?:es)?|drafts?)\b|\b(?:the|this|that|these|those)\s+(?:changes?|work)\b[^.!?;\n]{0,50}\b(?:came|comes?|originat\w*|is|was|were)\s+from\s+(?:the\s+)?(?:sub-?agent|agent)\b/giu, ), ].map((match) => ({ index: match.index, type: "competing" as const })), ...[ @@ -1167,11 +1167,11 @@ function passiveHandoffRefersToCandidatePr( /\b(?:(?:this|that|the)\s+)?(?:(?:seed|structure|phase\s*1)\s+)?(?:pr|pull\s+request)\b/giu, ), ].map((match) => ({ index: match.index, type: "candidate" as const })), - ...[...segment.matchAll(/\b(?:it|that|this)\b/giu)].map((match) => ({ + ...[...segment.matchAll(/\b(?:it|that|this|these|those)\b/giu)].map((match) => ({ index: match.index, type: "pronoun" as const, })), - ...[...segment.matchAll(/\bthe\s+(?:changes?|work)\b/giu)].map((match) => ({ + ...[...segment.matchAll(/\b(?:the\s+(?:changes?|work)|(?:these|those)\s+changes)\b/giu)].map((match) => ({ index: match.index, type: "pronoun" as const, })),