Skip to content

feat(context): add Agent Review workflow#1795

Draft
Gandy2025 wants to merge 2 commits into
mainfrom
feat/context-review-module
Draft

feat(context): add Agent Review workflow#1795
Gandy2025 wants to merge 2 commits into
mainfrom
feat/context-review-module

Conversation

@Gandy2025

Copy link
Copy Markdown
Contributor

What changed

This implements the Phase 2 Agent Review module proposed in Context Lite V1 PR #749, independently of the unfinished Phase 1 Write workflow.

  • Extend the existing Context Reviewer setting with an explicit agent_review workflow, human / autonomous governance, and merge method. Old stored settings continue to default to the legacy App workflow.
  • Add the versioned context_tree_pr_review / reviewPacketV1 task contract and a 32 KiB decoded UTF-8 boundary.
  • Deliver validated task metadata to the assigned Agent as an explicitly untrusted runtime context block; malformed or oversized packets fail closed.
  • Extend the existing context-tree-review Skill so one assigned Reviewer owns inspection, scoped repair, full re-review, current-head outcome, Human handoff, and optional expected-head merge. There is no fixed Auditor Agent.
  • Expose the live assignment through the existing agent Context Tree info endpoint and the read-only first-tree org context-tree review-config command.
  • Keep the existing App-published read-only review as legacy_app; App ingress and publishing do not create a second verdict while agent_review owns the Team workflow.
  • Preserve hidden workflow fields in the current Settings UI until the dedicated Phase 3 product controls ship.
  • Add a cross-surface QA case for the assembled workflow and its recovery/concurrency boundaries.

Scope and architecture

This is deliberately a module on existing primitives: Agent, ordinary task Chat/Inbox, Context Tree binding, runtime proof, org settings, the existing review Skill, and Host gh.

It does not add a Job table, scheduler, watcher, Coordinator service, dedicated review endpoint, Cloud merge credential, or second PR-state store.

Phase 1 task creation is not implemented here. Until it ships, tests and dogfood create an ordinary task Chat and inject the exact opening body plus taskType / reviewPacketV1 metadata that Phase 1 will later emit. No private test-only endpoint or alternate packet shape is introduced.

User and developer impact

  • Teams can configure an existing Agent as the standalone Context Tree Reviewer without installing the GitHub App.
  • In Human governance, the Agent stops at a current-head result and human handoff.
  • In Autonomous governance, the Agent may merge only the same reviewed head using GitHub's expected-head compare after configured checks pass.
  • Existing Context Reviewer installations remain on the legacy App behavior unless explicitly migrated.

Validation

  • pnpm check
  • pnpm typecheck
  • pnpm validate:skill
  • Targeted shared, server, client, CLI, Web, and Skill-eval suites all pass.
  • The unrelated context-tree-audit fixture that timed out only under full parallel-suite load passes in isolation: 12/12.
  • git diff --check

The full pnpm test run was attempted twice. Each run reached unrelated pre-existing timing-sensitive QA/skill fixtures under parallel load; the affected fixtures passed when rerun independently. No Agent Review targeted test failed.

Follow-up boundaries

  • Phase 1 will produce the same task packet from Write-created PRs.
  • Phase 3 will add Setup/Web controls, compatibility routing, and broader productization.
  • This PR does not deploy or enable Agent Review for any existing organization.

@Gandy2025 Gandy2025 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent code review — Agent Review (Phase 2)

Reviewed at head ec47509a (the 8df2c7fe..ec47509a delta is test-only: one
mock fixture in api-small-routes-extra.test.ts, production code unchanged).
Scope: architecture intrusion, legacy_app compatibility / single verdict,
task-packet trust boundary, repair + exact-head merge contract, Phase-1
simulation fidelity, test coverage.

Note on review type: my gh identity is the PR author, so GitHub only allows a
COMMENT review here. Treat this as one should-fix (medium) + minor notes;
I did not find an architectural blocker.


Should-fix (medium) — malformed-metadata fail-closed is bypassed by a stack overflow

renderAgentTaskContextForLLM (packages/client/src/runtime/agent-io.ts:330)
is designed to fail closed: when contextReviewTaskMetadataSchema.safeParse(...)
returns success === false it emits the <first-tree-task-context-error> block
telling the Reviewer "Do not repair, publish, or merge … report the malformed
task input."

That guarantee does not hold for a deeply-nested reviewPacketV1. The size
guard decodedStringBytes (packages/shared/src/schemas/context-review.ts:110)
is unbounded recursion, and zod v4's own parse recurses too, so a deeply-nested
value throws RangeError: Maximum call stack size exceeded — which safeParse
does not convert to {success:false}. The !parsed.success branch never
runs; the call throws instead.

Empirical repro (standalone, zod 4.4.3 matching zod@^4.0.0):

depth=2000  jsonBytes=4058  -> returned success=false
depth=3000  jsonBytes=6058  -> THREW RangeError
depth=5000  jsonBytes=10058 -> THREW RangeError

Reachability:

  • The trigger is metadata.taskType === "context_tree_pr_review" — the only
    gate before the recursive walk. Any chat participant who can attach message
    metadata can set it; sendMessageSchema.metadata is
    z.record(z.string(), z.unknown()) with no depth bound, stored verbatim.
  • ~3000 nesting levels ≈ 6 KB of JSON — far under the 32 KiB packet guard
    (which counts only string-leaf bytes and does not bound structural depth) and
    under transport body limits.
  • renderAgentTaskContextForLLM also runs over precedingMessages, so a
    recipient with nothing to do with Context Review still walks the payload if a
    prior message in the chat carries that taskType.

Observed impact in the built paths: the batch-inject handlers (TUI / cursor /
codex) and the single-message toSDKUserMessage path all wrap
formatInboundContent in try/catch and log a format failure, so the crafted
message is silently dropped rather than surfaced as malformed. That is a
denial-of-delivery for that turn and a defeat of the advertised fail-closed
degradation — not a process crash or RCE, but exactly the untrusted-input
robustness class this trust boundary exists to prevent.

Suggested direction (not prescriptive): bound structural depth before / inside
the byte walk, and wrap the size-check + parse so any throw maps to the same
<first-tree-task-context-error> fail-closed block instead of propagating.

Minor — 32 KiB boundary is client-runtime-only, not server-side

contextReviewTaskMetadataSchema / CONTEXT_REVIEW_PACKET_MAX_BYTES are
referenced only in agent-io.ts; there is no server-side reference. That is
consistent with Phase 1 not being built yet, but any framing that says the
32 KiB limit is enforced "server-side pre-persistence" is inaccurate for this
PR — today the server stores the packet as generic untyped metadata and the
only validation locus is the recipient runtime. Worth stating precisely so
Phase 1 owns the ingestion-side check explicitly.

Minor — fail-closed test covers only shallow-malformed input

agent-io.test.ts "fails closed when Agent Review task metadata is malformed"
uses { schemaVersion: 1, pullRequest: 42 } (shallow), which correctly returns
success:false. It does not exercise the deep-nesting vector above, so the
regression that matters is untested. A depth-based case would lock the fix.


Confirmed sound (no action)

  • Single verdict owner. Under agent_review, the App webhook path returns
    {handled:false, reason:"workflow_not_legacy_app"}
    (context-reviewer-pr.ts:232) and App publication is rejected in
    assertCurrentAuthority (context-reviewer-publisher.ts:430); the Agent path
    never submits a GitHub APPROVE. Two projections (PR comment + commit
    status), one authority. Tests cover both guards.
  • Legacy compatibility. Stored two-field configs default to
    legacy_app / human / squash; applyInputDelta preserves workflow fields
    across two-field writes; disabling nulls only agentUuid. Existing App
    reviewer teams are unaffected. legacy_app + autonomous is rejected at both
    input and storage refinement layers.
  • Packet trust boundary. reviewPacketV1Schema is .strict();
    repositoryPathSchema rejects absolute, \, and ./.. segments (no
    traversal); expectedHead is normalized 40-hex; repository is owner/name.
    The runtime wrapper labels every value as untrusted and defers to live GitHub;
    SKILL §6.1 requires live author == requesterGithubLogin, managed marker,
    and scope agreement before any write. Discovery-only, GitHub-authoritative.
  • Repair + exact-head merge. SKILL §6.3/§6.5 re-read live PR + source head
    before edit/commit/push, scope-limit writes, forbid force-push/retarget,
    restart from each successor head, never carry READY across a push, and merge
    only via gh pr merge --match-head-commit <SHA> with no --auto/--admin.
    Progress-gated NEEDS_HUMAN; bounded check-wait with no watcher/job. Faithful
    to the accepted design.
  • Phase-1 simulation fidelity. Because the runtime renders any sender's
    taskType/reviewPacketV1 metadata identically, injecting the real packet
    through an ordinary task Chat exercises the true consumption boundary with no
    test-only endpoint and no second packet shape. No hidden dependency.
  • Minimalism. No new table, endpoint, job, queue, coordinator, or merge
    credential. contextReviewer extended in place; /agent/context-tree/info
    reused for the read.

Overall: architecture is minimal and faithful to the R18-accepted trust model.
Recommend landing after the fail-closed hardening + its regression test; the two
minor items are precision/coverage, not blockers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant