fix: reject wrong-typed reasoner envelope fields as malformed output (#4166) - #4266
Merged
Conversation
…ppress collateral pause reports, harden test isolation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
processTaskOutputclassified a reasoner run asunparseable-responseonly when the deliverable field failed validation (payload.proposalsupplied but unusable). A wrong-typed non-deliverable field —{"analysis": 7}, or apausewith no resolvable target — still passed the envelope check, produced nothing usable, and was recorded as a legitimateno-proposalsuccess. Garbage output was indistinguishable from "the loop correctly had nothing to propose", exactly the hole #2727 closed forproposal.validateReasonerResponsenow reports which envelope keys were supplied but unusable in a newinvalidFieldsarray, and the hook keysunparseable-responseoff that list instead of re-deriving per-field validity itself:analysispresent and not a string →'analysis'proposalnon-null and dropped (bad scope / no title / unnormalizable slug / not a plain object) →'proposal'pausenon-null and dropped (no reason, no resolvable target,"this"with nothing to block on) →'pause'null/undefinedstays absent, not malformed — an explicitproposal: nullremains the documented "nothing to propose" answer, and a valid envelope with a valid deliverable is unaffected. Dropping is still lenient and the function still never throws, so the shared contract only gains a field; the previously-returned keys are unchanged.Caller audit (the issue's explicit ask): the only production caller of
validateReasonerResponseislayeredIntelligenceHooks.js#processTaskOutput(plus thelayeredIntelligencebarrel re-export and the unit tests). Nothing depended on the lenient behavior for a non-deliverable field. Within the hook, a valid proposal overwritesreasonon every branch of the filing path, so flagging an invalidpausealongside a good proposal cannot change a filed run's outcome — the new classification only reaches the no-proposal path. The user-facing reason string is unchanged (unparseable-responsealready has copy inclient/src/utils/layeredIntelligenceReasons.js), so no client change is needed.Also replaces the hand-rolled
proposalAttemptedButInvalidcheck in the hook with the validator's own report (one source of truth for per-field validity).From the review round:
⚠️ Layered Intelligence: <app> reasoner envelope has unusable fields: analysis, pause) —unparseable-responsealone can't tell an operator whether the model returned no JSON at all or one wrong-typed key. That reason token feeds retry counting and the hard-exclusion gate, so the diagnostic shouldn't be discarded.unparseable-responsewas inaccurate for the newly-covered cases ("returned no usable JSON" for a payload that is valid JSON) — reworded to cover both.blockOnIssue: "this"pause dropped only because its proposal failed validation is not double-reported: one root cause, one entry. This sharpens the report only —invalidFieldsis already non-empty in that case, so the malformed/clean verdict is unchanged.blockOnIssue-asymmetry (a surviving proposal re-derivesreason, a surviving pause does not) predates this PR (Base programmatic-I/O task learning outcomes on the output-hook result, not the exit code #2727) and is now stated in the comment rather than silently changed.beforeEachnow restores the validator mock's default, matching the existing leak guards forgatherSources/computeHardExclusionGate.Test plan
validateReasonerResponse > invalidFields (#4166)unit block inserver/services/layeredIntelligence.test.js: wrong-typedanalysis; supplied-but-unusableproposalandpause(including"this"with no proposal); absent/explicitly-null fields never flagged; empty for a fully valid envelope; all three collected in envelope order.processTaskOutputcases inserver/services/autonomousJobs/layeredIntelligenceHooks.test.js:unparseable-responsefor{"analysis": 7}, a targetlesspause, and a multi-field-invalid envelope; the offending field names reaching the log; and an end-to-end case driving the hook through the realvalidateReasonerResponse(the suite otherwise stubs it) so the two halves are verified together, not only in isolation. The existing "well-formed envelope that proposes nothing →no-proposal" and "explicitproposal: null→no-proposal" cases stay green.cd server && npx vitest run services/layeredIntelligence.test.js services/autonomousJobs/ services/layeredIntelligenceBarrel.test.js services/layeredIntelligenceOutcomes.test.js→ 6 files, 559 tests passed.cd client && npm test→ 647 files, 7879 tests passed (covers the reworded reason gloss).… require PostgreSQL) — the identical set fails before the change, and none touch Layered Intelligence.Closes #4166