Skip to content

orchestrate: split the invalid envelope verdict — an off-schema envelope from a completed agent FAILs a slice whose work is correct #386

Description

@rodrigorjsf

Found during orchestrate run prd352-20260803-015333 (PRD #352, 12 slices, all passed).

Symptom

The implementer on slice #364 finished its work correctly — eight files changed, +6 tests, all capability verbs run — then emitted an envelope that validate_envelope rejected with SCHEMA_MISMATCH. Four fields were wrong in shape, not content:

Field Emitted Schema requires
role null the literal "implementer"
filesChanged null array of path strings
verification object keyed by capability array of {capability, result}
notes array of strings a single string

Notably the content was richer than the schema allows — the agent put the tool's exitCode and prose evidence inline where an enum belongs, and split notes into a list because it had several distinct points. These are the mistakes of an agent reasoning about what would be useful, not one that ran out of room.

Why the current rule is too blunt

The failure taxonomy collapses "truncated" and "off-schema" into one invalid verdict, and both FAIL the slice. But they are different failures with different risk profiles:

  • A truncated envelope means the agent's turn was cut off, so its self-report is genuinely untrustworthy and the FAIL is right.
  • An off-schema envelope from an agent that ran to completion means the report is mis-formatted while the underlying work — which lives in the worktree, and which verify_changeset, the reviewer, and the orchestrator's own pre-merge gate all inspect independently — is unaffected.

FAILing the second case discards a completed implementation over a JSON shape error.

Workaround used

A once-only re-emission request via SendMessage to the still-resident subagent, spelling out the four field corrections and the exact target shape.

No verification was bypassed: verify_changeset, the reviewer pass, and the orchestrator's independent run_build + run_tests gate all still ran afterwards. Precedent: the same recovery shape already worked for a turn-budget cutoff on an earlier run.

Suggested fixes

  • Split the invalid verdict into invalid-truncated (fence unclosed / JSON unparseable → FAIL, as today) and invalid-schema (parses cleanly but does not match → sanction one re-emission request to the same agent before FAILing). The discriminator is cheap and deterministic: does the fenced block parse as JSON at all?
  • Put the literal envelope schema in each subagent definition's own prompt rather than relying on the agent to recall it. Three of the four errors here (verification shape, notes cardinality, role literal) are exactly what an agent gets wrong when writing from memory. (See the companion issue on schema introspection — adopting this fix naively at the orchestrator level introduced its own failure.)
  • Have validate_envelope return the offending field paths in a machine-usable list, not only a prose errorMessage, so an orchestrator can compose the correction request automatically.

Relationship to existing issues (read before deduping)

Environment

orchestrate 1.6.0 (plugins/orchestrate/.claude-plugin/plugin.json:3 on development) plus the PRD #352 umbrella branch (slice-executor delegation layer). Run prd352-20260803-015333; slice #364, implementer role.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions