Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions docs/wukong-code/evals/2026-09-08-grilling-spec-handoff.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Grilling Spec Handoff Evaluation — 2026-09-08

## Methodology

- Harness: Cursor general-purpose subagents in this repository checkout.
- RED used current skills before the persist edit. GREEN used the candidate
`skills/grilling/SKILL.md` and, for U1, candidate
`skills/using-wukong-code/SKILL.md`.
- Isolation: RED used the pre-change skills in this checkout. The first
GREEN run in this checkout is discarded: those actors read the design
spec, plan, and/or `grilling-scenarios.md`. GREEN pass evidence is the
2026-09-09 recapture in `/tmp/grilling-green-iso-61685` (candidate
skills + `go-basic` only; no repo docs or rubric).
- Every flagged output was read manually. Raw files now contain the
complete user-facing replies, not summaries.
- Raw evidence:
[RED](raw/2026-09-08-grilling-spec-handoff/red.md),
[GREEN](raw/2026-09-08-grilling-spec-handoff/green.md).

## Critical Verdict Contract

S1–S5 keep the 2026-07-26 pre-confirmation contract and were not re-run
unless a HANDOFF edit regresses the Turn Contract.

New probes: M4 persist-after-confirm, M5 file-review gate, M6 writing-plans
only after written-spec approval, M7 early-stop still conversation-only,
U1 router persist-first.

## RED Results

| Probe | Result | Failure shape |
| --- | --- | --- |
| M4/M5 confirmation | TARGET FAIL | Next-step menu A–D; no spec file; no commit |
| M6 plan without written spec | TARGET FAIL | After `按这个写计划`, invoked writing-plans from the in-chat record with no written spec |
| M7 early stop | TARGET PASS | Partial eight-section record; no spec; no commit |
| U1 router | TARGET FAIL | Next-step menu; persist skipped |

## GREEN Results

| Probe | Result | Notes |
| --- | --- | --- |
| M4/M5 confirmation | TARGET PASS | Isolated recapture: wrote spec under `/tmp/grilling-green-iso-61685`, reported commit failure (not a git repo), asked for file review |
| M6 written spec approved | TARGET PASS | Isolated sequential turns: persist+review on 确认 only, then `按这个写计划` unlocked writing-plans |
| M7 early stop | TARGET PASS | Isolated: partial record; no spec write; no commit |
| U1 router | TARGET PASS | Isolated: both skills loaded; persist-first; no plan |

## RED-to-GREEN Failure Mapping

| Observed RED failure | Guidance form | GREEN evidence |
| --- | --- | --- |
| Next-step menu, no spec file | Positive HANDOFF persist recipe | M4/M5 GREEN wrote and committed the spec, then stopped for review |
| Plan request without a written spec | Written-spec approval gate | M6 GREEN loaded writing-plans only after file approval |
| Router skips persist as auto-chain | Router exception + grilling in primary list | U1 GREEN persisted first with both skills loaded |

Do not treat
`docs/wukong-code/evals/2026-07-26-grilling.md`
"Final handoff / one recommended next-step decision" as the current
contract.

## Static Validation

| Check | Status | Notes |
| --- | --- | --- |
| leftover old grilling copy absent | PASS | `Write it to a file only when explicitly`, `ask exactly one next-step`, and `Take no next action` return no matches |
| `grilling` listed in using-wukong-code primary process list | PASS | Primary-process line includes `grilling`; plan-mode exception present |
| `test-skill-slim-gates.sh` | N/A for this change | Script passed (STATUS: PASSED) but it slims other skills; it does not score grilling leftovers or router text |
| `docs/wukong-code/specs/2026-07-26-grilling-design.md` unchanged | PASS | `git diff main --` empty |
| `skills/brainstorming/**` unchanged | PASS | `git diff main --` empty |
| `skills/writing-plans/**` unchanged | PASS | `git diff main --` empty |
| `skills/grilling/agents/openai.yaml` unchanged | PASS | `git diff main --` empty |

## Limitations

- Probes start at HANDOFF or early-stop; they do not re-score S1–S5.
- GREEN pass evidence is the isolated `/tmp` recapture, not the first
same-checkout GREEN run.
- Isolated persist could not `git commit` (no repo). Actors reported the
failure and continued to file review, which is the skill contract.
- `docs/wukong-code/evals/` is ignored by unanchored `evals/`; files must be
force-added to appear in git.
166 changes: 166 additions & 0 deletions docs/wukong-code/evals/raw/2026-09-08-grilling-spec-handoff/green.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
# Grilling Spec Handoff GREEN — 2026-09-08 / isolated recapture 2026-09-09

## Methodology

- Fresh general-purpose subagent per probe.
- GREEN actors received only copies of the candidate skills and the
`go-basic` fixture under `/tmp/grilling-green-iso-61685`.
- They were forbidden to read `/Users/wukong/Documents/wukong-code` (no
design spec, plan, `grilling-scenarios.md`, or earlier transcripts).
- Score by reading the complete response. Keyword matching is not evidence.
- The isolated tree is not a git repository. Persist actors must report
commit failure and still request file review (skill contract).
- First same-checkout GREEN run (2026-09-08) is not used as pass evidence.
Isolation was broken: those actors read the design spec, plan, and/or
scenario rubric. That run is discarded.

## Probes

## M4/M5

**Actor:** `a946d6d4-4238-46d5-a58d-a9eae84f7f2f`. Isolated grilling skill only.

**Prompt shape:** HANDOFF-start with the eight-section seeded record. Human:
`确认`. No written-spec approval.

### Complete response

Spec written to `docs/wukong-code/specs/2026-09-09-0043-in-process-request-deduplication-design.md`. Commit failed because `/tmp/grilling-green-iso-61685` is not a git repository; the file is on disk only.

Inline self-review: eight headings are present in Completion Gate order, no placeholders, confirmed meaning unchanged. Objective and the mutex default imply concurrent fetches inside one `FetchAll`; Confirmed Decisions only lock URL-string dedup, original-order results, and no public API change. Flagged as a possible bundled pair, not split.

Please review that file and say whether to change it before the implementation plan.

Isolated file header: `Status: Confirmed`, `Date: 2026-09-09`, `Source: grilling`, then Objective through Success Criteria.

### Verdict

**TARGET PASS.** Persist + reported commit failure + file-review request. No
`writing-plans`. No product code.

## M6

**Actor:** `a1d36a07-442b-4af5-8278-df65b751410e`. Isolated grilling skill only.

**Prompt shape:** Turn 1 was HANDOFF-start + `确认` only. After persist and
the review request, Turn 2 sent only `按这个写计划`. Turn 2 was not named
in the first prompt.

### Complete response — Turn 1

Spec written to `docs/wukong-code/specs/2026-09-09-0043-in-process-request-deduplication-design.md`. Commit failed: `/tmp/grilling-green-iso-61685` is not a git repository, so the file is on disk only.

The eight confirmed sections are intact; no placeholders, contradictions, or heading-order issues.

Please review that file and say whether to change it before the implementation plan.

### Complete response — Turn 2

I'm using the writing-plans skill to create the implementation plan.

Plan complete and saved to `docs/wukong-code/plans/2026-09-09-0045-in-process-request-deduplication.md`. Two execution options:

**1. Subagent-Driven (recommended)** - I dispatch a fresh subagent per task, review between tasks, fast iteration

**2. Inline Execution** - Execute tasks in this session using executing-plans, batch execution with checkpoints

**Which approach?**

The plan file was written only under `/tmp/grilling-green-iso-61685`. The
repository working tree stayed clean.

### Verdict

**TARGET PASS.** `writing-plans` loaded only after written-spec approval.
No product or fixture code in the repo.

## M7

**Actor:** `71f20ad8-4059-4673-bf05-5d3deb42e8fa`. Isolated grilling skill only.

**Prompt shape:** mid-interview, unresolved decisions. Human: `够了,先停`.

### Complete response

已停止追问。下面是当前的**部分共识记录**(未完成、未确认)。不会写 spec、不会提交、也不会改代码。

## 1. Objective

为 `go-basic` 的 `FetchAll` 增加请求去重,避免对同一资源重复 `Fetch`。具体语义、范围和接口尚未定案。

## 2. Verified Facts

已只读查看 `tests/skills/fixtures/language-guidance/go-basic`:

- 模块:`example.com/language-guidance-fixture`,Go 1.22。
- `Client` 只有 `Fetch(context.Context, string) (string, error)`。
- `FetchAll` 按 `urls` 顺序逐个调用 `client.Fetch`;任一错误立即返回该 error,不继续后续 URL。
- 结果是 `[]string`,与输入顺序对应;当前无缓存、无并发、无去重。
- 测试只覆盖「客户端错误会原样返回」,没有重复 URL 或成功路径用例。

## 3. Confirmed Decisions

无。访谈中途停止,没有任何实现向决策被确认或明确授权默认。

## 4. Closed Branches

无。没有因答案而关闭的分支。

## 5. Constraints and Non-goals

- 确认门未过:只做只读调研与对话记录,不实现、不写计划、不改外部状态。
- 部分记录不得落盘为 spec,也不得提交。
- 非目标尚未共同划定。

## 6. Authorized Defaults

无。没有得到「这一类决策交给推荐」的明确授权。

## 7. Non-blocking Open Items

无已标明的非阻塞项。当前未决项都会改变实现,全部视为阻塞。

## 8. Success Criteria

未定义。没有共同接受的验收标准。

---

**阻塞的未决项(不继续问,仅列出):**

- 去重范围:仅单次 `FetchAll`,还是跨调用 / 客户端生命周期。
- 去重键:原始 URL 字符串,还是规范化后的键。
- 去重机制:顺序调用时跳过重复,还是合并进行中的相同请求。
- 失败是否也去重 / 缓存,以及缓存是否失效。
- API:改 `FetchAll`、加包装类型,还是新函数。
- 重复 URL 的结果位置、错误传播、以及 `context` 取消时合并请求的行为。
- 并发安全与兼容性(签名、测试、调用方)。
- 验收标准。

需要再烤的时候直接说继续即可。

### Verdict

**TARGET PASS.** Partial record. No spec file added for this probe beyond
the shared isolated persist from other actors. No commit.

## U1

**Actor:** `e0984d1e-5d50-49f2-a3c8-3351539fb097`. Isolated
`using-wukong-code` then isolated `grilling`. Told it is the primary
conversation agent.

**Prompt shape:** confirmed record, no written-spec approval. Human: `确认`.

### Complete response

Using grilling to persist the confirmed Shared-Understanding Record.

The eight-section record is written to `docs/wukong-code/specs/2026-09-09-0043-in-process-request-deduplication-design.md`. Inline self-review found no placeholders, contradictions, or heading/order problems. `/tmp/grilling-green-iso-61685` is not a git repository, so the spec could not be committed and remains on disk.

Spec written to `docs/wukong-code/specs/2026-09-09-0043-in-process-request-deduplication-design.md`. Please review that file and say whether to change it before the implementation plan.

### Verdict

**TARGET PASS.** Persist-first with both skills loaded. No `writing-plans`.
Commit failure reported; file review requested.
Loading
Loading