feat: hold an agent run that shipped clean code for the wrong task - #6119
Merged
Conversation
…5994) PortOS's reviewers answer "is this good code?". None of them could answer "is this the code that was asked for?", because none of them ever see the request — the code-review prompt is handed a unified diff and nothing else. So a CoS agent could ship a PR that was clean, well-reviewed and green in CI, and quietly do something other than the task the user wrote. The run-level evidence gate proved changes existed and shipped; nothing proved they were the requested changes. A finished run's accumulated diff is now re-read against the task's own stated objective, in a fresh context, and answers three questions the quality chain cannot: is anything that was asked for missing, is anything that was not asked for smuggled in, and is the verification evidence real. The verdict is ship, fix-first, or rethink; a rethink records the run as needing attention rather than complete, and raises a Review Hub alert naming what is missing or unrequested. Settings > Code Reviewers picks the local model it runs on (independently of the quality chain, or inheriting it) and can switch it off. Fresh context is the mechanism: the objective comes from the task record, never the agent's transcript, which would hand the reviewer the assumptions that produced the drift in the first place. The gate fails OPEN throughout — a missing local backend, an unreadable diff, or a model that answers with prose all leave the run's outcome exactly as they found it, so a local-model outage can never convert a queue of good runs into held ones. Claude-Session: https://claude.ai/code/session_01ChZ1DzJoaTsPoouRcrf8rJ
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PortOS's reviewers answer "is this good code?". None of them could answer "is this the code that was asked for?" —
CODE_REVIEW_SYSTEM_PROMPThands the model a unified diff and nothing else, and the whole configurable reviewer chain is diff-scoped the same way. The user-visible consequence: a CoS agent could ship a PR that was clean, well-reviewed, green in CI, and quietly did something other than the task that was written.evaluateSuccessCriteriaproved changes existed and shipped; nothing proved they were the requested changes.This adds a second, objective-aware review at agent completion.
server/lib/goalFidelity.js(new, pure) — the verdict vocabulary (ship/fix-first/rethink),taskObjective(task)(the trusted operator-authored objective: the TASK's description + prompt block, never the agent's transcript), the settings resolver, and the parse/validate of the model's structured answer.normalizeGoalFidelityVerdictreturnsnullfor an unusable answer, which is never collapsed into ashippass or arethinkhold.server/lib/gitCommitProbe.js— gainsrunWindowDiff, the diff half of the run-window probes. Base is resolved on committer date exactly ascommitsSincefilters on it, so the two can't disagree about which commits belong to the run. Every failure is areasonwith anulldiff, never''— "git could not answer" must not read as "the run changed nothing".server/services/codeReview.js—GOAL_FIDELITY_SYSTEM_PROMPT+runLocalGoalFidelityReview. Objective and diff ride one message in separate adaptive fences, each labelled with its trust level: the objective is the requirement, the diff is untrusted evidence that can never become an instruction. Returns a validated verdict or an error — never model prose.server/services/agentFinalization.js— the completion gate. Runs only on a run that would otherwise be recorded a success (same posture as the branch-jack downgrade). Arethinkdowngrades it to needs-attention with a namedsuggestedFix;fix-firstis recorded as advisory and still ships.server/services/review.js— bridges a hold into a Review Hub alert, so "the agent built the wrong thing" reaches someone who was not watching/cos/agents./cos/agents— pick the local model (or inherit the chain's), switch the gate off, and read the verdict with its named missing/unrequested items.Decisions worth flagging
finalizeAgent, in the server process, as one synchronous request; the CLI reviewers are invoked by the follow-up agent from a prompt and have no server-side entry point. A configured backend outside that set declines rather than being silently swapped for one the user did not pick.missing/unrequestedare model-authored strings derived from an untrusted diff. They are trimmed, capped, and have the](markdown link/image separator split apart, because the Review Hub renders an alert description through PortOS's markdown renderer.Test plan
server/lib/goalFidelity.test.js(new) — objective composition and its cap, config resolution and its three decline paths, verdict validation, thenull-means-nothing-judged sentinel, and link neutralization.server/services/agentFinalization.goalFidelity.test.js(new) — the gate end to end: it judges the task objective and not the transcript, recordsshipwithout disturbing the run, treatsfix-firstas advisory, holdsrethinkwith the right category/error/event, fails open across five distinct failure shapes, and never re-judges an already-failed run. Verified to fail before the fix.server/lib/gitCommitProbe.test.js—runWindowDiff: window resolution, empty-vs-unreadable, four decline paths, and cap-inclusive truncation.server/lib/validation.test.js,server/services/codeReview.test.js,client/.../CodeReviewersTab.test.jsx,client/.../AgentCard.test.jsx— schema accept/reject, prompt shape + fence escaping, settings round-trip, rendered verdict.ollamareviewer (the configured chain) returnedNo findings.Closes #5994
https://claude.ai/code/session_01ChZ1DzJoaTsPoouRcrf8rJ