Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions packages/client/src/runtime/agent-briefing.ts
Original file line number Diff line number Diff line change
Expand Up @@ -912,7 +912,7 @@ guidance above to surface/setup/seed first.
| \`first-tree-welcome\` | the onboarding first chat — a natural welcome / "help me get started" message from the user; value-first intro, not a repo scan or tree setup chat |
| \`first-tree-read\` | read relevant Context Tree files before acting from task / path / feature signals |
| \`first-tree-write\` | reflect a concrete source artifact (PR, design doc, meeting note, review thread, or pasted source) into the Context Tree |
| \`first-tree-seed\` | set up the team's Context Tree from readable sources — declared workspace repos or a local Git repo / GitHub URL supplied in chat; creates + binds the tree if none exists, fills a bound-but-empty tree, and can continue Phase 2 in the same setup chat after Phase 1 merges |
| \`first-tree-seed\` | set up the team's Context Tree from readable sources — declared workspace repos or a local Git repo / GitHub URL supplied in chat; creates + binds the tree if none exists, fills a bound-but-empty tree, and opens a single PR with the initial structure and leaves |
| \`first-tree-file-bug\` | the user hit a bug in First Tree itself (CLI, runtime, chat, web, GitHub, or tree tooling) and wants it reported — gathers repro + version + chat/user IDs and opens an issue on First Tree's own repo (not the user's own/bound repo) via the user's \`gh\` CLI |`;
}
return `## First Tree Family
Expand All @@ -928,7 +928,7 @@ harness skills (\`tdoc\`, \`review\`, \`simplify\`, \`update-config\`,
| \`first-tree-welcome\` | the onboarding first chat — a natural welcome / "help me get started" message from the user; value-first intro, not a repo scan or tree setup chat |
| \`first-tree-read\` | read relevant Context Tree files before acting from task / path / feature signals |
| \`first-tree-write\` | reflect a concrete source artifact (PR, design doc, meeting note, review thread, or pasted source) into the Context Tree |
| \`first-tree-seed\` | bootstrap the team's Context Tree from readable sources (declared repos or a local Git repo / GitHub URL supplied in chat); create + bind if none exists, fill a bound-but-empty tree, or continue Phase 2 in the same setup chat after Phase 1 merges |
| \`first-tree-seed\` | bootstrap the team's Context Tree from readable sources (declared repos or a local Git repo / GitHub URL supplied in chat); create + bind if none exists, fill a bound-but-empty tree, delivered as a single reviewable PR |
| \`first-tree-file-bug\` | you hit a bug in First Tree itself (CLI, runtime, chat, web, GitHub, or tree tooling) and want it reported — gathers repro + version + chat/user IDs and opens an issue on First Tree's own repo (not the user's own/bound repo) via the user's \`gh\` CLI |`;
}

Expand Down
25 changes: 14 additions & 11 deletions packages/skill-evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,8 @@ judge:
diff, with scores for durability, source-boundary discipline, rationale
quality, and conciseness;
- `first-tree-seed` skeleton quality from the actual `empty-tree-source-present`
Phase 1 proposal, with scores for source grounding, structure fit,
phase-boundary discipline, coverage calibration, and conciseness;
domain proposal, with scores for source grounding, structure fit,
confirmation-boundary discipline, coverage calibration, and conciseness;
- `first-tree-welcome` first-task quality from the actual readable-repo +
populated-tree row output, with scores for evidence grounding, boundedness,
usefulness, verifiability, and avoiding setup-as-task.
Expand Down Expand Up @@ -159,35 +159,38 @@ accepts the source row id as an alias.
`eval:gate -- --suite first-tree-seed` runs the live tested-agent gate for
`first-tree-seed`. It covers the minimum bootstrap lifecycle boundaries:

- empty tree + present bare source proposes only Phase 1 skeleton for user
approval;
- empty tree + present bare source proposes the domain skeleton for explicit
chat confirmation before writing anything;
- non-empty tree refuses seed and points to incremental write or focused
maintenance;
- missing source clone stops on incomplete provisioning instead of partial
seed;
- an unbound workspace routes through `tree init --dir <managed tree path>`
before Phase 1;
before proposing the domain skeleton;
- bare source repos are read through a materialized read worktree, not as
checkouts;
- an empty manifest may use a readable local checkout supplied in chat without
requiring GitHub App installation or team-resource registration;
- an approved Phase 1 skeleton in the same visible setup chat routes to Phase 2
instead of being refused as an unrelated populated tree.
- an approved skeleton in the same visible setup chat routes to one
structure-plus-leaves seed PR, and rejects any intermediate structure PR,
merge wait, or return ping.

`eval:quality -- --suite first-tree-seed` runs the `empty-tree-source-present`
deterministic gate first. If that hard-rule gate passes, the quality judge
scores the actual Phase 1 skeleton proposal and source evidence. This judge is
scores the actual domain skeleton proposal and source evidence. This judge is
opt-in and does not replace the deterministic seed gate: empty-tree self-check,
source boundary, bare worktree protocol, no tree/source/GitHub side effects, and
no Phase 2 leaf content before approval remain hard-rule oracle conditions.
no structure or leaf content before chat confirmation remain hard-rule oracle
conditions.

`eval:periodic -- --suite first-tree-seed` runs
`first-tree-seed-real-first-tree-source-periodic`. The fixture creates an empty
Context Tree plus a per-run bare source repo cloned from the current
`first-tree` repo `HEAD`. The model must still materialize
`worktrees/seed-source-repo`, read source evidence from that checkout, propose
only a Phase 1 skeleton, and ask for approval. This realism case stays in the
periodic tier and is not part of the default seed gate.
the domain skeleton, and ask for explicit chat confirmation before writing.
This realism case stays in the periodic tier and is not part of the default
seed gate.

The runner creates isolated temporary workspaces under
`packages/skill-evals/.runs/<timestamp>-<case-id>/`, installs
Expand Down
6 changes: 3 additions & 3 deletions packages/skill-evals/src/core/__tests__/judge.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -324,11 +324,11 @@ describe("quality runner with fake judge", () => {
[
"first-tree-seed-skeleton-quality",
JSON.stringify({
reasoning: "The skeleton is source-grounded and stays in Phase 1.",
reasoning: "The skeleton is source-grounded and preserves the chat confirmation boundary.",
scores: {
conciseness: 4,
coverage_calibration: 4,
phase_boundary: 5,
confirmation_boundary: 5,
source_grounding: 5,
structure_fit: 4,
},
Expand Down Expand Up @@ -358,7 +358,7 @@ describe("quality runner with fake judge", () => {
expect(batch.failed).toBe(0);
expect(batch.passed).toBe(1);
expect(batch.cases[0]?.judge_scores).toMatchObject({
phase_boundary: 5,
confirmation_boundary: 5,
source_grounding: 5,
});
} finally {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,52 +42,23 @@ describe("first-tree-seed chat-first fixtures", () => {
}
}, 20_000);

it("materializes an approved Phase 1 skeleton without Phase 2 leaves", () => {
const evalCase = gateCase("same-chat-phase2-continuation");
it("keeps the tree empty while carrying same-chat skeleton approval", () => {
const evalCase = gateCase("same-chat-approved-skeleton-builds-single-pr");
const paths = createRunPaths({ caseId: evalCase.id, packageRoot, startedAt: new Date().toISOString() });

try {
const reporter = createEvalReporter(evalCase.id, false);
const contextTreePath = setupFixture(evalCase, paths, reporter);
const localHead = execFileSync("git", ["rev-parse", "HEAD"], {
cwd: contextTreePath,
encoding: "utf8",
}).trim();
const remoteHead = execFileSync("git", ["rev-parse", "origin/main"], {
cwd: contextTreePath,
encoding: "utf8",
}).trim();
const remoteDefault = execFileSync("git", ["symbolic-ref", "refs/remotes/origin/HEAD"], {
cwd: contextTreePath,
encoding: "utf8",
}).trim();

expect(existsSync(join(contextTreePath, "product", "onboarding", "NODE.md"))).toBe(true);
expect(existsSync(join(contextTreePath, "product", "onboarding", "flow.md"))).toBe(false);
expect(existsSync(join(contextTreePath, "product"))).toBe(false);
expect(existsSync(join(contextTreePath, "system"))).toBe(false);
const history = readFileSync(join(paths.workspacePath, ".first-tree-eval", "chat-history.md"), "utf8");
expect(history).toContain("## Assistant — earlier turn");
expect(history).toContain("## User — earlier turn");
expect(history).toContain("Phase 1 proposal");
expect(history).toContain("Skeleton proposal");
expect(history).toContain("Approved");
expect(history).toContain("Phase 1 PR handoff");
expect(remoteHead).toBe(localHead);
expect(remoteDefault).toBe("refs/remotes/origin/main");
expect(validateFixture(paths, contextTreePath, evalCase, false, reporter).ok).toBe(true);
} finally {
rmSync(paths.runRoot, { force: true, recursive: true });
}
}, 20_000);

it("keeps the Phase-1-shaped negative sibling free of same-chat history", () => {
const evalCase = gateCase("phase1-shaped-tree-without-same-chat-history-refuses");
const paths = createRunPaths({ caseId: evalCase.id, packageRoot, startedAt: new Date().toISOString() });

try {
const reporter = createEvalReporter(evalCase.id, false);
const contextTreePath = setupFixture(evalCase, paths, reporter);

expect(existsSync(join(contextTreePath, "product", "onboarding", "NODE.md"))).toBe(true);
expect(existsSync(join(paths.workspacePath, ".first-tree-eval", "chat-history.md"))).toBe(false);
expect(history).toContain("one reviewable PR");
expect(history).not.toContain("merge it");
expect(history).not.toContain("reply in this setup chat");
expect(validateFixture(paths, contextTreePath, evalCase, false, reporter).ok).toBe(true);
} finally {
rmSync(paths.runRoot, { force: true, recursive: true });
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,38 +22,44 @@ describe("first-tree-seed floor invariants", () => {
expect(skillMarkdown).toMatch(/rather than asking for the First\s+Tree GitHub App/);
});

it("checks same-chat Phase 2 continuation before refusing state C", () => {
const continuation = skillMarkdown.indexOf("Check for a Phase 2 continuation before classifying state C");
const stateC = skillMarkdown.indexOf("**C — Already seeded.**");

expect(continuation).toBeGreaterThan(-1);
expect(stateC).toBeGreaterThan(continuation);
expect(skillMarkdown).toContain("this setup chat's visible history");
expect(skillMarkdown).toContain("re-resolve the same readable sources and enter Phase 2");
expect(skillMarkdown).toContain("applies only to the Context Tree checkout");
it("keeps the confirmation gate while collapsing delivery to one PR", () => {
expect(skillMarkdown).toContain("same run");
expect(skillMarkdown).toContain("single reviewable PR");
expect(skillMarkdown).toContain("There is no\nintermediate PR and no merge between the passes");
expect(skillMarkdown).toContain("The one human gate is the **in-chat checklist approval**, not a PR merge.");
expect(skillMarkdown).toContain("Do not write anything — structure or content — before the user has signed");
expect(skillMarkdown).toContain("there is no ping, no merge, and no waiting");
expect(skillMarkdown).toContain("chore/seed-tree");
expect(skillMarkdown).toMatch(/Do not\s+use `git ls-tree`, `git show`, `git grep`/);
expect(skillMarkdown).toContain("not a leftover checkout");
});

it("delays App coverage guidance until a reviewable milestone", () => {
expect(skillMarkdown).toContain("After the Phase 1 PR is open");
expect(skillMarkdown).toContain("After the seed PR is open");
expect(skillMarkdown).toContain("do not interrupt source resolution, structure");
expect(skillMarkdown).toContain("relay only a recovery URL returned");
});

it("ships behavioral gates for chat-supplied sources and same-chat Phase 2 continuation", () => {
it("ships behavioral gates for chat-supplied sources and post-confirmation single-PR delivery", () => {
const chatSource = FIRST_TREE_SEED_GATE_CASES.find((evalCase) => evalCase.id === "empty-manifest-chat-source");
expect(chatSource).toMatchObject({
expected: { action: "propose_phase1_skeleton", requireSourceRead: true, requireWorktree: false },
expected: { action: "propose_skeleton", requireSourceRead: true, requireWorktree: false },
fixture: { sourceRepoState: "chat-local-readable", treeState: "empty" },
});
expect(chatSource?.forbidden.actions).toContain("require_github_app");

const continuation = FIRST_TREE_SEED_GATE_CASES.find((evalCase) => evalCase.id === "same-chat-phase2-continuation");
expect(continuation).toMatchObject({
expected: { action: "continue_phase2", requireSourceRead: true, requireWorktree: true },
fixture: { sourceRepoState: "bare-readable", treeState: "phase1-approved" },
const build = FIRST_TREE_SEED_GATE_CASES.find(
(evalCase) => evalCase.id === "same-chat-approved-skeleton-builds-single-pr",
);
expect(build).toMatchObject({
expected: {
action: "build_single_pr",
requireChatHistoryRead: true,
requireSourceRead: true,
requireWorktree: true,
},
fixture: { chatHistoryState: "approved-skeleton", sourceRepoState: "bare-readable", treeState: "empty" },
});
expect(continuation?.forbidden.actions).toContain("refuse_nonempty_tree");
expect(build?.forbidden.actions).toContain("restart_skeleton_proposal");
expect(build?.forbidden.actions).toContain("legacy_two_pr_handoff");
});
});
Loading
Loading