docs(qa): add real agent turn manual QA case#1522
Conversation
Add the capstone @first-tree/qa case: a message to a bound agent triggers a real provider-authenticated model turn that posts a correct reply — the one case that proves the model turn itself, closing the cross-process release-verification loop in system/cloud/release/verification.md. Authored from a real PASS run against merged main: with a minimal read-only codex credential bridged into the isolated run cell, a probe to the bound codex agent produced `codex app-server turn started accepted=1` and the agent posted the requested token back to the chat (`@ftqa-sender QA_REAL_TURN_OK`). Scoped honestly: requires provider one-turn-ready (BLOCKED without it, not FAIL); assert on token content, not byte equality (the reply carries the product routing mention prefix). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
yuezengwu
left a comment
There was a problem hiding this comment.
I found one docs correctness issue to fix before this merges.
packages/qa/cases/cross-surface/real-agent-turn.md:36 uses POST /orgs/<orgId>/chats, but the actual mounted route is /api/v1/orgs/:orgId/chats and the adjacent authenticated-ws-inbox-delivery case already uses POST /api/v1/orgs/<orgId>/chats. Since this case is meant to be executable QA guidance, following the new text literally would send the deterministic probe to the wrong path. Please include the /api/v1 prefix here.
baixiaohang
left a comment
There was a problem hiding this comment.
Review by baixiaohang-code-reviewer - PR #1522
Recommendation: request changes
- Rationale: The new case overstates the release-verification effect of a prose QA asset and can make the still-unguarded cross-process release gap look closed.
Risk level: B-low
- Path baseline:
packages/qa/**is outside the A/B path buckets, so it starts as C. - Semantic lift: this QA/documentation change explicitly affects release verification, so it lifts to B-low.
PR summary
- Author / repo: liuchao-001 / agent-team-foundation/first-tree
- Problem: Release QA lacks a reusable case that proves a bound agent can perform a real provider-authenticated turn and reply, not just receive inbox delivery.
- Approach: Adds a manual
@first-tree/qacase for aone-turn-readyprovider run, with minimal credential bridging, daemon/runtime observation, and token-reply evidence. - Impacted modules:
packages/qa/cases/cross-surface
Review findings
❌ 1. real-agent-turn says this case "closes the release-verification loop" that system/cloud/release/verification.md marks as unguarded, but @first-tree/qa cases are explicitly non-executable prompts and the release node defines the gap as missing automated cross-process verification. Unless this PR also wires a runner/CI or scheduled-run policy and updates the release node, this should say it documents a manual model-turn QA slice; release/review decisions still need to treat automated cross-process coverage as unguarded. [R5 / packages/qa/cases/cross-surface/real-agent-turn.md:14]
✅ 2. The provider readiness and minimal credential-bridging boundaries match the QA environment guidance.
Action taken
- Submitted request changes.
Note: I did not repeat the already-open route-prefix finding from the existing review on this PR.
Address review (baixiaohang): @first-tree/qa cases are non-executable manual prompts; system/cloud/release/verification.md defines the gap as missing *automated* cross-process verification. Reword the Goal so the case documents a repeatable manual model-turn QA slice that reduces risk but does NOT close the automated gap — release/review must still treat automated cross-process coverage as unguarded until a CI-gated or scheduled-run replacement lands and the release node is updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
yuezengwu
left a comment
There was a problem hiding this comment.
Rechecked latest head after the update. The release-verification wording in real-agent-turn.md now correctly scopes this as a manual QA slice and preserves the automated-coverage gap.
One blocking docs correctness issue remains from my prior review: packages/qa/cases/cross-surface/real-agent-turn.md:40 still says POST /orgs/<orgId>/chats. The actual mounted route and the adjacent authenticated-ws-inbox-delivery case use POST /api/v1/orgs/<orgId>/chats. Since this case is meant to guide a real QA run, please include the /api/v1 prefix here before merge.
Summary
Adds a
@first-tree/qacase:packages/qa/cases/cross-surface/real-agent-turn.md.It documents a manual model-turn QA slice — a message to a bound agent triggers a real, provider-authenticated turn that posts a correct reply. This is the deepest cross-process behavior (the model turn itself, not just delivery), extending
authenticated-ws-inbox-delivery.Scope correction (per review):
@first-tree/qacases are non-executable, agent-run prompts, not an automated runner or CI gate.system/cloud/release/verification.mddefines the gap as missing automated cross-process verification, so this case reduces risk via a repeatable manual check but does NOT close that automated gap — release/review must still treat automated cross-process coverage as unguarded until a CI-gated (or scheduled-run) replacement lands and the release node is updated.Validation (real run, PASS)
In an isolated run cell against merged main, with a minimal read-only codex credential bridged in (a single
auth.json, removed after the run): a probe to the bound codex agent yieldedcodex app-server turn started … accepted=1and the agent posted the requested token back to the chat (@ftqa-sender QA_REAL_TURN_OK).Scope (honest)
Requires provider
one-turn-ready; BLOCKED, not FAIL, without it. The reply is an ordinary chat message carrying the product's@<recipient>routing prefix — the case asserts on token content, not byte-for-byte equality. Docs/asset only: one prose case file. No runner, CI gate, or product code change.🤖 Generated with Claude Code