Skip to content

docs(qa): add real agent turn manual QA case#1522

Open
liuchao-001 wants to merge 2 commits into
mainfrom
docs/qa-real-agent-turn-case
Open

docs(qa): add real agent turn manual QA case#1522
liuchao-001 wants to merge 2 commits into
mainfrom
docs/qa-real-agent-turn-case

Conversation

@liuchao-001

@liuchao-001 liuchao-001 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a @first-tree/qa case: packages/qa/cases/cross-surface/real-agent-turn.md.

It documents a manual model-turn QA slice — a message to a bound agent triggers a real, provider-authenticated turn that posts a correct reply. This is the deepest cross-process behavior (the model turn itself, not just delivery), extending authenticated-ws-inbox-delivery.

Scope correction (per review): @first-tree/qa cases are non-executable, agent-run prompts, not an automated runner or CI gate. system/cloud/release/verification.md defines the gap as missing automated cross-process verification, so this case reduces risk via a repeatable manual check but does NOT close that automated gap — release/review must still treat automated cross-process coverage as unguarded until a CI-gated (or scheduled-run) replacement lands and the release node is updated.

Validation (real run, PASS)

In an isolated run cell against merged main, with a minimal read-only codex credential bridged in (a single auth.json, removed after the run): a probe to the bound codex agent yielded codex app-server turn started … accepted=1 and the agent posted the requested token back to the chat (@ftqa-sender QA_REAL_TURN_OK).

Scope (honest)

Requires provider one-turn-ready; BLOCKED, not FAIL, without it. The reply is an ordinary chat message carrying the product's @<recipient> routing prefix — the case asserts on token content, not byte-for-byte equality. Docs/asset only: one prose case file. No runner, CI gate, or product code change.

🤖 Generated with Claude Code

Add the capstone @first-tree/qa case: a message to a bound agent triggers
a real provider-authenticated model turn that posts a correct reply — the
one case that proves the model turn itself, closing the cross-process
release-verification loop in system/cloud/release/verification.md.

Authored from a real PASS run against merged main: with a minimal
read-only codex credential bridged into the isolated run cell, a probe to
the bound codex agent produced `codex app-server turn started accepted=1`
and the agent posted the requested token back to the chat
(`@ftqa-sender QA_REAL_TURN_OK`). Scoped honestly: requires provider
one-turn-ready (BLOCKED without it, not FAIL); assert on token content,
not byte equality (the reply carries the product routing mention prefix).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@yuezengwu yuezengwu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one docs correctness issue to fix before this merges.

packages/qa/cases/cross-surface/real-agent-turn.md:36 uses POST /orgs/<orgId>/chats, but the actual mounted route is /api/v1/orgs/:orgId/chats and the adjacent authenticated-ws-inbox-delivery case already uses POST /api/v1/orgs/<orgId>/chats. Since this case is meant to be executable QA guidance, following the new text literally would send the deterministic probe to the wrong path. Please include the /api/v1 prefix here.

@baixiaohang baixiaohang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review by baixiaohang-code-reviewer - PR #1522

Recommendation: request changes

  • Rationale: The new case overstates the release-verification effect of a prose QA asset and can make the still-unguarded cross-process release gap look closed.

Risk level: B-low

  • Path baseline: packages/qa/** is outside the A/B path buckets, so it starts as C.
  • Semantic lift: this QA/documentation change explicitly affects release verification, so it lifts to B-low.

PR summary

  • Author / repo: liuchao-001 / agent-team-foundation/first-tree
  • Problem: Release QA lacks a reusable case that proves a bound agent can perform a real provider-authenticated turn and reply, not just receive inbox delivery.
  • Approach: Adds a manual @first-tree/qa case for a one-turn-ready provider run, with minimal credential bridging, daemon/runtime observation, and token-reply evidence.
  • Impacted modules: packages/qa/cases/cross-surface

Review findings
❌ 1. real-agent-turn says this case "closes the release-verification loop" that system/cloud/release/verification.md marks as unguarded, but @first-tree/qa cases are explicitly non-executable prompts and the release node defines the gap as missing automated cross-process verification. Unless this PR also wires a runner/CI or scheduled-run policy and updates the release node, this should say it documents a manual model-turn QA slice; release/review decisions still need to treat automated cross-process coverage as unguarded. [R5 / packages/qa/cases/cross-surface/real-agent-turn.md:14]
✅ 2. The provider readiness and minimal credential-bridging boundaries match the QA environment guidance.

Action taken

  • Submitted request changes.

Note: I did not repeat the already-open route-prefix finding from the existing review on this PR.

Address review (baixiaohang): @first-tree/qa cases are non-executable
manual prompts; system/cloud/release/verification.md defines the gap as
missing *automated* cross-process verification. Reword the Goal so the
case documents a repeatable manual model-turn QA slice that reduces risk
but does NOT close the automated gap — release/review must still treat
automated cross-process coverage as unguarded until a CI-gated or
scheduled-run replacement lands and the release node is updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@liuchao-001 liuchao-001 changed the title docs(qa): add real agent turn case (closes product loop) docs(qa): add real agent turn manual QA case Jul 8, 2026

@yuezengwu yuezengwu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked latest head after the update. The release-verification wording in real-agent-turn.md now correctly scopes this as a manual QA slice and preserves the automated-coverage gap.

One blocking docs correctness issue remains from my prior review: packages/qa/cases/cross-surface/real-agent-turn.md:40 still says POST /orgs/<orgId>/chats. The actual mounted route and the adjacent authenticated-ws-inbox-delivery case use POST /api/v1/orgs/<orgId>/chats. Since this case is meant to guide a real QA run, please include the /api/v1 prefix here before merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants