Skip to content

fix: reject wrong-typed reasoner envelope fields as malformed output (#4166) - #4266

Merged
atomantic merged 2 commits into
mainfrom
claim/issue-4166
Aug 15, 2026
Merged

fix: reject wrong-typed reasoner envelope fields as malformed output (#4166)#4266
atomantic merged 2 commits into
mainfrom
claim/issue-4166

Conversation

@atomantic

@atomantic atomantic commented Aug 15, 2026

Copy link
Copy Markdown
Owner

Summary

processTaskOutput classified a reasoner run as unparseable-response only when the deliverable field failed validation (payload.proposal supplied but unusable). A wrong-typed non-deliverable field — {"analysis": 7}, or a pause with no resolvable target — still passed the envelope check, produced nothing usable, and was recorded as a legitimate no-proposal success. Garbage output was indistinguishable from "the loop correctly had nothing to propose", exactly the hole #2727 closed for proposal.

validateReasonerResponse now reports which envelope keys were supplied but unusable in a new invalidFields array, and the hook keys unparseable-response off that list instead of re-deriving per-field validity itself:

  • analysis present and not a string → 'analysis'
  • proposal non-null and dropped (bad scope / no title / unnormalizable slug / not a plain object) → 'proposal'
  • pause non-null and dropped (no reason, no resolvable target, "this" with nothing to block on) → 'pause'

null/undefined stays absent, not malformed — an explicit proposal: null remains the documented "nothing to propose" answer, and a valid envelope with a valid deliverable is unaffected. Dropping is still lenient and the function still never throws, so the shared contract only gains a field; the previously-returned keys are unchanged.

Caller audit (the issue's explicit ask): the only production caller of validateReasonerResponse is layeredIntelligenceHooks.js#processTaskOutput (plus the layeredIntelligence barrel re-export and the unit tests). Nothing depended on the lenient behavior for a non-deliverable field. Within the hook, a valid proposal overwrites reason on every branch of the filing path, so flagging an invalid pause alongside a good proposal cannot change a filed run's outcome — the new classification only reaches the no-proposal path. The user-facing reason string is unchanged (unparseable-response already has copy in client/src/utils/layeredIntelligenceReasons.js), so no client change is needed.

Also replaces the hand-rolled proposalAttemptedButInvalid check in the hook with the validator's own report (one source of truth for per-field validity).

From the review round:

  • The hook logs which fields were unusable (⚠️ Layered Intelligence: <app> reasoner envelope has unusable fields: analysis, pause) — unparseable-response alone can't tell an operator whether the model returned no JSON at all or one wrong-typed key. That reason token feeds retry counting and the hard-exclusion gate, so the diagnostic shouldn't be discarded.
  • The client gloss for unparseable-response was inaccurate for the newly-covered cases ("returned no usable JSON" for a payload that is valid JSON) — reworded to cover both.
  • A well-formed blockOnIssue: "this" pause dropped only because its proposal failed validation is not double-reported: one root cause, one entry. This sharpens the report only — invalidFields is already non-empty in that case, so the malformed/clean verdict is unchanged.
  • The blockOnIssue-asymmetry (a surviving proposal re-derives reason, a surviving pause does not) predates this PR (Base programmatic-I/O task learning outcomes on the output-hook result, not the exit code #2727) and is now stated in the comment rather than silently changed.
  • beforeEach now restores the validator mock's default, matching the existing leak guards for gatherSources / computeHardExclusionGate.

Test plan

  • New validateReasonerResponse > invalidFields (#4166) unit block in server/services/layeredIntelligence.test.js: wrong-typed analysis; supplied-but-unusable proposal and pause (including "this" with no proposal); absent/explicitly-null fields never flagged; empty for a fully valid envelope; all three collected in envelope order.
  • New processTaskOutput cases in server/services/autonomousJobs/layeredIntelligenceHooks.test.js: unparseable-response for {"analysis": 7}, a targetless pause, and a multi-field-invalid envelope; the offending field names reaching the log; and an end-to-end case driving the hook through the real validateReasonerResponse (the suite otherwise stubs it) so the two halves are verified together, not only in isolation. The existing "well-formed envelope that proposes nothing → no-proposal" and "explicit proposal: nullno-proposal" cases stay green.
  • Verified every new test fails against the pre-fix source (8 failures) and passes after.
  • cd server && npx vitest run services/layeredIntelligence.test.js services/autonomousJobs/ services/layeredIntelligenceBarrel.test.js services/layeredIntelligenceOutcomes.test.js → 6 files, 559 tests passed.
  • cd client && npm test → 647 files, 7879 tests passed (covers the reworded reason gloss).
  • Full server suite run: the only failures are 55 pre-existing PostgreSQL-dependent suites that fail in this environment because the configured Postgres instance is not running (… require PostgreSQL) — the identical set fails before the change, and none touch Layered Intelligence.

Closes #4166

@atomantic
atomantic merged commit f709d80 into main Aug 15, 2026
7 checks passed
@atomantic
atomantic deleted the claim/issue-4166 branch August 15, 2026 05:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reject wrong-typed reasoner envelope fields as malformed output

1 participant