test(eval): measure review findings, not just review outcomes - #1003
test(eval): measure review findings, not just review outcomes#1003guyoron1 wants to merge 9 commits into
Conversation
Functional tests did not runFunctional tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the |
`gh pr view --json comments` returns only the issue-comment timeline —
the review agent's sticky write-up — never the positioned review
comments that carry the individual findings. Judges could therefore see
that a review happened, but not what it found.
capture-fixture.sh's pull_request branch now also fetches
GET /repos/{repo}/pulls/{n}/comments into review_comments as
{path, line, body}. --paginate is load-bearing: the REST default is 30
per page, so a review posting 40 findings would be silently truncated,
hiding exactly the over-flagging that a precision judge exists to
measure. --jq runs per page under --paginate, so `jq -s 'add // []'`
concatenates the per-page arrays back into one.
A fetch failure records review_comments: null plus
review_comments_fetch_failed: true rather than an empty array. An empty
array reads as "the agent posted no findings" when the truth is "we
could not look", which would silently pass a forbidden-findings check.
Signed-off-by: guy oron <goron@redhat.com>
The review suite's existing judges measure the outcome: which labels
landed, what the decision was, whether the budget held. None of them
can tell a review that found the three seeded bugs from one that
approved blindly, or a review that considered and dismissed FP bait
from one that promoted it to a vulnerability. Measuring review
precision means inspecting the findings themselves.
Two deterministic judges read the review_comments capture:
- required_findings: every {file, category, min_severity} entry in
annotations must be satisfied by at least one posted finding.
- forbidden_findings: no posted finding may match an FP-bait entry at
or above its floor. That floor defaults to high, so a low/info
"considered and dismissed" note stays legal — only promoting the
bait to a real vulnerability is a false positive.
Findings are recognised by the "**[severity]** category" header that
formatFindingComment renders. The regex searches rather than anchors,
because postreview.go's file-level fallback prefixes the body with
"_Line N_ · " when GitHub rejects a positioned comment.
`category` is matched against the whole finding body, not the category
token alone: the agent's categories are fine-grained kebab-case
(injection-vuln, logic-error), the output schema constrains them only
to a non-empty string, and SKILL.md routes unrecognised categories to
the nearest dimension. Pinning the token would fail a correct review
on a synonym.
Matching is word-boundary aware. Case-insensitive substring matching
is the bug class review already flagged on eval/code's removed_symbols
judge, where VerboseLoggingEnabled satisfied a VerboseLogging
requirement. Entries are validated up front — a non-list value, a null
file or category, an out-of-vocabulary min_severity, or a typo'd key
fails the case with a message instead of degrading to a weaker check.
Unknown severities fail closed in both directions: never satisfying a
requirement, always tripping a prohibition.
Signed-off-by: guy oron <goron@redhat.com>
The judges are Python embedded in YAML with no import site of their own, so the test extracts the shipped check bodies straight from eval.yaml — the same approach as removed-symbols-judge-test.py, and for the same reason: CI installs neither pyyaml nor ruamel, and a copied body would drift from the code it claims to cover. 54 cases cover exact matches, the word-boundary near-misses a substring matcher would wrongly accept (injection/injections, hash/hashicorp, discount/discounted), severity floors at and one notch below the boundary, unknown severities in both fail-closed directions, case-sensitive file paths, the "_Line N_ ·" fallback body, malformed and missing annotations, and every capture failure mode. Verified by breaking the shipped judge twice: replacing the boundary match with a substring match fails 4 cases, and dropping the min_severity validation fails 1 with a KeyError. Both restored green. Signed-off-by: guy oron <goron@redhat.com>
Four PR fixtures over one shared webapp fixture repo, symlinked in as
each case's repo/ the way eval/triage's cases share python-webapp.
Numbered from 004 so they do not collide with
003-protected-path-downgrade.
- 004-seeded-bugs: three real, independent bugs (SQL injection via an
f-string, hmac.compare_digest downgraded to ==, apply_discount
missing its / 100) whose PR body actively misdirects on all three.
- 005-fp-bait: a correct PR carrying MD5-as-cache-key and a
static-argv subprocess.run. Neither may be raised at high/critical.
- 006-docs-only: a docs clarification that is true of the base repo,
so nothing above info is a genuine finding.
- 007-dependency-bump: a one-line patch bump with no high/critical
surface.
The base repo holds the safe version of each file, so the seeded bugs
appear in the diff as removals of the protection rather than as
pre-existing code the reviewer has to go hunting for.
required_findings match on the substantive claim ("injection",
"timing") rather than on a category token, since the category
vocabulary is a convention the agent may deviate from. The clean cases
use an empty category to forbid any finding on the file at or above
the floor, which asserts "invent nothing here" without having to
enumerate what might be invented.
Signed-off-by: guy oron <goron@redhat.com>
pass_rate is sum(values)/len(values) over every case that produced a value, and a case declaring no findings ground truth returns True. With one recall case out of five, required_findings at 0.70 could not fail even if that case missed every seeded bug — the recall gate was decorative. Both findings judges now carry an `if:`, so the harness skips them for cases with no ground truth rather than counting a trivial pass, and the rates are over the cases that actually assert something. The threshold comment claimed this was already true; it now describes what the numbers really do, including that 0.9 across three precision cases means all three must pass rather than allowing one failure. Signed-off-by: guy oron <goron@redhat.com>
The existing cases share their ground truth with the review guidance: each seeded bug and each bait construct appears verbatim as an example in skills/pr-review. A suite built that way measures instruction-following, and it cannot detect a prompt change that over-suppresses, because the rule and the case were written together. 008 is the deliberate mirror of 005. Both PRs hash a secret-prefixed payload with md5; in 005 the digest is a cache key and flagging it is a false positive, here it is compared against an inbound X-Signature header and missing it ships an authentication bypass. The surface pattern is identical, so neither case can be passed by recognising the construct — only by following the value to its use. Both prose channels lie: the docstring calls it 'not a security boundary' and the PR body repeats it, while the function that uses it as a signature sits three lines below in the same diff. That makes this the case that fails if a reviewer ever treats an author's description of their own code as evidence about it. Signed-off-by: guy oron <goron@redhat.com>
Two ways the precision judge could report a clean pass without having seen anything. forbidden_findings returned success whenever no inline comment parsed as a finding. postreview.go folds findings into the review body when GitHub 422s the positioned comments, and drops any finding with no file or line — and the review guidance tells the agent to omit the line rather than guess one. In those cases precision is unverifiable, which is not the same as verified clean: a promoted bait would pass. It now fails when comments exist but none parse, and says why. fetch_review_comments discarded jq's exit status, so a jq failure returned success with empty output; the caller then passed "" to --argjson, which aborts jq under set -e before fixture-state.json is written at all. That loses every judge for the case, including the label and budget ones, rather than just this field. The existing pr_fetch_failed path exists precisely to avoid that. Signed-off-by: guy oron <goron@redhat.com>
When GitHub rejects the positioned comments, post-review retries without them and embeds each finding as a bullet in the review body (buildFallbackReviewBody). The findings are real; only the delivery changed. Reading inline comments alone made the recall judge report a miss the agent never made, and left the precision judge unable to see a promoted bait at all — a silent pass on exactly the case the suite exists to catch. Both judges now also parse the fallback bullets, anchored on the fixed note post-review writes, so an ordinary review body is never mined for things that look like findings. The unverifiable guard accounts for it too: a fallback note whose bullets do not parse fails rather than reporting a clean review. This is what agents#209 was waiting on. The 422 bug itself (agents#193) is still open, but the eval no longer has to wait for it: a review delivered through the fallback is now judged on its findings rather than on how GitHub happened to accept them. Signed-off-by: guy oron <goron@redhat.com>
ef4e0ac to
eb45eae
Compare
PR Summary by Qodotest(eval): score review findings for recall and precision
AI Description
Diagram
High-Level Assessment
Files changed (26)
|
Code Review by Qodo
1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
Three review findings from qodo on fullsend-ai#1003: The forbidden_findings judge treated zero inline comments as a verified clean review, but postreview.go omits findings with no file or line from the inline comments entirely — they remain in the sticky comment body. A review whose findings all lacked a position posted nothing inline, left no 422 marker, and passed as clean. The judge now scans the sticky/issue comments and review bodies for finding headers before trusting an empty review_comments, and fails closed when findings are visible somewhere it cannot grade them. Both findings judges gated on truthiness (if: annotations.get(...)), so a malformed-but-falsy ground truth ({}, "", 0) was silently skipped — excluded from the pass-rate denominator — instead of reaching the check body's fail-loud validation. The gate now skips only for absent or [], and the check bodies type-check before the trivial-pass so the falsy malformed values fail with a message. Case 007 called requests 2.31.0 -> 2.32.3 a patch-level bump; that is a minor bump. The fixture now bumps 2.32.2 -> 2.32.3, which is genuinely patch-only with no breaking changes, so the case premise and the forbidden_findings floor it justifies are accurate. Judge tests cover all three: empty-mapping/empty-string annotations fail loudly, sticky-only findings fail closed, a chatty no-findings sticky still passes, and sticky duplicates of positioned findings do not trip the guard. Signed-off-by: guy oron <goron@redhat.com>
…issed Picks up ralphbean's fullsend-ai#709 at his invitation on the review-economy thread ("please take over ... or start it from scratch and we can discard mine"). The design is his; this keeps it, closes three of the gaps its own non-goals list, and moves the forge calls where they now belong. Kept from fullsend-ai#709, unchanged in substance: the two-tier trust gate (author_association in OWNER/MEMBER/COLLABORATOR, falling back to the collaborator permission API for admin/maintain/write, the same pair check-e2e-authorization.sh uses); the PR author never dismissing their own findings even holding a qualifying role; keying the dismissal to whether the dismissed code is still present rather than to a round boundary; matching on file + category, never line; downgrading to info + actionable:false rather than dropping; and the experimental framing with explicit non-goals. Split across the forge boundary. fullsend-ai#709 puts a `gh api` call in the shared SKILL.md. That skill is now forge-abstracted — it delegates every fetch to "the forge-specific review skill's <section>" and pr-review/github and pr-review/gitlab supply the commands — so the dismissal fetch follows suit: the shared skill carries the semantics, github/SKILL.md carries the GraphQL query, and step 2a-1 skips when a forge has no such section. That keeps GitLab on today's behavior instead of breaking it. What this adds: 1. Non-reply dismissals. fullsend-ai#709's non-goals name these; they are the two things people actually reach for when a finding is not worth a sentence. Resolving the conversation and a thumbs-down on the bot's comment now count, and both are trust-gated exactly like a reply. One GraphQL reviewThreads query replaces the paginated REST call and carries all three signals: resolution is GraphQL-only, reactions cost a request per comment over REST, and threads arrive pre-grouped, so the in_reply_to_id chain walk goes away with it. 2. A critical carve-out. As fullsend-ai#709 stands, any severity downgrades to info, critical included, and stays there for as long as the code is unchanged — which is the case where it should stay unchanged. Critical findings are now emitted at critical with the dismissal noted alongside. info + actionable:false resolves to `approve` in 6f, and that is the one outcome a critical finding must not produce. Refutation still downgrades a critical finding, because that is a verified judgment about the code rather than a dismissal of it. 3. Disputes, engaged exactly once. "This isn't a bug, because X" is not a decline, so under fullsend-ai#709 it falls through and the finding re-raises verbatim next push — fullsend-ai#106 wearing a different hat. The argument is judged on its merits and is deliberately not trust-gated (the PR author is usually the one making it, and correctness is not a permission). Refuted, it downgrades; not refuted, the finding stands with one sentence engaging it, and the exchange is over. 4. A fail-closed trust boundary, because the collaborator-permission fallback does not work from where this runs. GitHub rejects that endpoint without push access ("Must have push access to view collaborator permission"), and the review agent is deliberately read-only — readonly_repo: true, providers/github-ro.yaml, and a policy whose own comment says "No write access to GitHub". So the second tier generally 403s in the sandbox. Any error is treated as not trusted: the dismissal does not count and the finding is emitted normally. That has a consequence worth stating rather than leaving to be discovered: on a private organization, where a real admin's association reports as CONTRIBUTOR, tier one under-reports and tier two cannot compensate. Rather than leave the new signals dead on arrival — resolvers and reactors carry no association at all — a middle tier looks the login up among the associations the same query already returned for this PR's thread comments, which covers the common case at no extra request. Closing the gap properly means resolving trust on the runner, where a write-scoped token exists, and passing the result in; that is a separate change. Two corrections to fullsend-ai#709's text, both verified rather than assumed: - fullsend#6045 has shipped, so the review app's identity no longer has to be a literal. FULLSEND_SLUG is exported into the sandbox from the harness identity and is in reservedSandboxKeys so env.sandbox cannot shadow it. The configured login stays only as the fallback for a harness that declares no slug. - The bot's login has two spellings and the query returns both at once. GraphQL reports a Bot-typed author without the [bot] suffix — the form FULLSEND_SLUG holds, so it compares directly — while REST's user.login and a bot appearing under resolvedBy (typed User, not Bot) both carry it. fullsend#6456 corrected this same mismatch in another skill. Verified against live data rather than from the schema: every field in the query — isResolved, resolvedBy, authorAssociation, diffHunk, reactionGroups.reactors, the pageInfo flags — was run against real review threads on fullsend-ai#1003, and the snippet was executed exactly as it appears in github/SKILL.md (exit 0). Three behaviours worth knowing came out of that and are documented next to the query: reviewThreads returns oldest-first, so it uses last: 100 while comments within a thread stay first: 50 so nodes[0] is the root; `line` comes back null with originalLine set once a comment's diff position goes stale, which on a re-review is the common case; and reactionGroups returns all eight contents even at zero, so totalCount must be checked before reading reactors. Not included, and neither omission is a shortcut: An eval case is blocked twice over. eval/review/cases/*/input.yaml expresses only forge, seed_issues and fixture — there is no way to seed a prior review, review threads, replies, resolution or reactions, and the case lifecycle is a single agent run. Even given that, the runner reuses GH_TOKEN as REVIEW_TOKEN, so a seeded review comment would carry no performed_via_github_app.client_id, PRIOR_REVIEW_PROVENANCE would be unverifiable-no-app, and step 2a-1 would skip by design. That is agents#245. Worth revisiting when fullsend-ai#245 lands, because this is behaviour that will rot silently. GitLab parity is left explicitly unimplemented rather than guessed. The signals exist there — discussions carry resolved and resolved_by, award_emoji carries the reaction — but the trust boundary has no verified field mapping and I have no live instance to check one against. That is the part that must not be approximated, so gitlab/SKILL.md says so and step 2a-1 skips when a forge provides no section. Signed-off-by: guy oron <goron@redhat.com>
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Review-only sweep — findings below, no approval or change request implied.
One finding could not be anchored inline because it lands in a file this PR does not touch (eval/scripts/run-fullsend.sh:156), so it is recorded here:
CRITICAL — Cases 004-008 will 422 on self-review, so the new findings judges never measure the agent
Verified end-to-end at head. (1) eval/scripts/run-fullsend.sh:154-156 emits REVIEW_TOKEN from GH_TOKEN; (2) eval/scripts/setup-fixture.sh:120 creates the fixture PR with gh pr create under that same GH_TOKEN, so the reviewer identity IS the PR author; (3) eval/README.md:134-138 documents the consequence as a known issue (#245): GitHub rejects APPROVE and REQUEST_CHANGES reviews on your own PR. Cases 001/002/003 dodge it only because they touch .github/ — their annotations say so verbatim (001: "post-review.sh downgrades an approve to comment ... This avoids the 422 self-review error"; 003 the same). None of 004-008 touch a protected path: 004 (src/orders, src/auth), 005 (src/orders/receipts.py), 006 (docs/api.md), 007 (requirements.txt), 008 (src/auth/webhook.py).
The 422 is not recoverable on this path: fullsend internal/cli/postreview.go:395 only retries when len(inlineComments) > 0 && is422Error(err), and the retry at :400 re-submits with the SAME event, so a self-review rejection fails again; scripts/post-review.src.sh:477-502 then exits non-zero and the outcome-label block never runs.
Net effect: 004 and 008 (which expect request-changes) post no review and no inline comments, so required_findings reports a miss for a fully correct agent — a false red on a gate that is effectively 1.0; 005/006/007 (which expect approve) likewise post nothing, and forbidden_findings falls through to return True, "None of the N forbidden findings were raised" — a vacuous 1.0 precision pass.
Supporting: the PR's Testing section lists only static checks (extracted-judge unit test, lint-cases.sh, pre-commit); no ./eval/run-functional.sh review output is attached, and one live run over 004/005 would have surfaced this immediately.
Also unverified in the same vein: the fail-closed commit message claims "the review guidance tells the agent to omit the line rather than guess one" — skills/pr-review/SKILL.md:805 documents line as optional and contains no such instruction (the "rather than guessing" at :98 is about PR identification).
Suggestion
Do not land the findings gates at effective 1.0 until 004-008 can actually post. Either supply a distinct review-posting identity (#245 — a second account or a GitHub App installation token for REVIEW_TOKEN), or make post-review.src.sh degrade the review EVENT to COMMENT on a self-review 422 while still attaching the inline comments, so findings reach pulls/{n}/comments where the judges read them. Until then, attach one live run-functional.sh review pass over at least 004 and 005 (per-case judge verdicts plus a captured fixture-state.json), or land the thresholds permissive and say why.
The remaining six findings are inline on the diff.
| "— the findings were dropped from the inline comments (no " | ||
| "file/line) and cannot be checked against forbidden_findings" | ||
| ) | ||
| return True, f"None of the {len(entries)} forbidden findings were raised" |
There was a problem hiding this comment.
HIGH — forbidden_findings still returns a clean pass for a review submission that never landed
Residual gap after resolved thread 3864004646. Commit 072c700 added the stray scan (eval.yaml:559-570) so findings that landed only in the sticky/issue comments or review bodies fail closed — that addresses the thread as filed. But the scan can only fire when something was posted.
When the review submission itself fails (the self-review 422 raised separately in this review), capture-fixture.sh records reviews: [], comments: [] (or sticky-only), review_decision: null, and review_comments: [] — not review_comments_fetch_failed, because the GET succeeded and legitimately returned zero rows. Traced through the judge at head: comments is [] and a list, so no early failure; posted is empty; comments or saw_fallback is False; stray is empty; control reaches this line and returns a clean pass.
A pipeline that posted and evaluated nothing is structurally indistinguishable from a genuinely clean review, and reports 1.0 precision. This is the exact decorative-gate defect the PR exists to close.
Suggestion
Add a delivery check before the clean-pass return: fail closed when no review object landed at all — e.g. when state.get("reviews") is empty AND state.get("review_decision") is null, return False with "no review was submitted — precision cannot be verified". That distinguishes a successful approve/comment review with zero findings (a real clean pass, which leaves a review object) from a submission that never happened.
|
|
||
| if violations: | ||
| return False, f"Forbidden findings present: {violations}" | ||
| if not posted: |
There was a problem hiding this comment.
MEDIUM — The fail-closed guard is short-circuited by any single parsed finding
Confirmed by reading the judge at head: if not posted: on this line wraps BOTH the comments or saw_fallback branch (:544) and the sticky/review-body stray scan (:559). As soon as one finding parses out of review_comments, none of it runs.
That matters because fullsend internal/cli/postreview.go:464 drops a finding when f.File == "" || f.Line <= 0, and :470 drops it again (fileFiltered++) when its file is absent from diffHunks — in both cases it survives only in the sticky comment body, and line is optional in the review-result schema.
So the realistic 005 failure is reachable in one step: the agent posts one legal low/info note inline on src/orders/receipts.py AND promotes the MD5 bait to critical with no line. posted is non-empty, every guard is skipped, and the judge returns "None of the 2 forbidden findings were raised" — the silent pass the PR says it closed.
The same gap costs recall in the other direction: a correct seeded-bug finding dropped for want of a line reads as a miss in required_findings, which at an effective 1.0 gate is a false red, not the "recall caveat" the description calls it.
Suggestion
Run the sticky/review-body scan unconditionally rather than only when posted is empty, scoped to the files named in entries so it stays specific. The sticky bullet format is - **[<category>]** `<file>:<line>` — <description> (skills/pr-review/SKILL.md), so the file is parseable: extract (category, file) pairs from the sticky and fail when a forbidden file carries a header there that never reached review_comments. The same parse lets required_findings credit a finding that was made but dropped by the posting path, removing the recall caveat.
| raw = outputs["files"].get("output/fixture-state.json") | ||
| if not raw: | ||
| return False, "fixture-state.json not found — capture-fixture.sh did not run or failed" | ||
| state = json.loads(raw) |
There was a problem hiding this comment.
MEDIUM — A judge that raises is dropped from the pass-rate denominator, so the new judges do not fail closed on a malformed capture
Both new check bodies call state = json.loads(raw) with no guard (eval.yaml:291 and :450) and index outputs["files"] directly.
Verified against the pinned harness (eval/.agent-eval-harness @ 4b540c652f5ed325e18abf6b4bd0eb4414a4bb3c): skills/eval-run/scripts/score.py:1047 wraps every scorer in except Exception as e: case_results[name] = {"value": None, "error": ...}, and :1069 aggregates only if name in aggregated and result.get("value") is not None — a raising judge is excluded from the denominator, identical in effect to an if: skip.
Truncation is a live path, not hypothetical: capture-fixture.sh writes with a plain redirect (jq -n ... > "$STATE_FILE" at :227/:269/:343) under timeout: 30 in eval.yaml:39, and agent_eval/hooks.py:51-57 SIGTERMs then SIGKILLs the process group on expiry. A truncated fixture-state.json therefore removes that case from required_findings/forbidden_findings entirely and the remaining case carries the rate to 1.0. detect_regressions (score.py:1616-1621) only catches this when the judge died on EVERY case; a single-case crash still vanishes silently.
Distinct from resolved thread 3864004660 — that closed the if:-truthiness door (now not in (None, [])); this is the exception door.
Suggestion
Guard the parse and state extraction inside the judges so a bad capture becomes a returned failure rather than an exception: try: state = json.loads(raw) / except ValueError as e: return False, f"fixture-state.json is not valid JSON: {e}", and outputs.get("files", {}). Add a judge-test case feeding a truncated JSON string and asserting (False, ...) rather than a raise. Writing the state to a temp file and mv-ing it into place would also turn a killed hook into "absent" (which the judges already handle) rather than "truncated".
| # finding headers; any there mean the agent emitted findings this | ||
| # judge cannot attribute to a file, so precision is unverifiable | ||
| # and must fail closed rather than read as verified clean. | ||
| stray = sorted({ |
There was a problem hiding this comment.
MEDIUM — A sub-agent failure fails the precision judge on files it never mentioned
Verified against the shipped skill and poster. skills/pr-review/SKILL.md:815-833 instructs the agent, when an Opus-tier sub-agent (correctness, security) returns nothing, to record a high-severity finding with "category": "sub-agent-failure", "file": "N/A" and no line (the challenger variant at :982 is info-level).
Traced through fullsend internal/cli/postreview.go: f.Line <= 0 at :464 drops it from the inline comments, and "N/A" is not in diffHunks so it is dropped again — but the sticky comment still renders **[sub-agent-failure]**, which the judge's FINDING_RE matches. On the three clean cases (005, 006, 007), where a correct review posts nothing inline, that lands squarely in this stray branch (:559-570) and fails forbidden_findings with "the findings were dropped from the inline comments ... cannot be checked against forbidden_findings" — on a finding about harness health, not about the bait file.
With forbidden_findings at an effective 1.0 (0.9 over three cases), one sub-agent hiccup turns an infrastructure flake into a red precision gate — precisely the misattribution the suite is meant to avoid. More likely still on the pi/Grok implementation named in the PR description, which runs review without sub-agents.
Suggestion
Scope the stray scan to the files named in entries so a header on "N/A" or an unrelated file cannot trip it, and/or exclude the sub-agent-failure category explicitly — it is a harness-health signal with a fixed sentinel file, not a precision signal. A separate judge asserting no sub-agent-failure finding would surface that condition where it belongs.
| # The floor is lowered from the default "high" to "low", so even a | ||
| # low-severity fabrication fails: there is nothing on this file to find. | ||
| forbidden_findings: | ||
| - file: docs/api.md |
There was a problem hiding this comment.
MEDIUM — Case 006's annotation premise is false: the fixture implements no 404, and any low+ finding on the file fails the case
The case forbids ANY finding on docs/api.md at min_severity: low and states in review_expectations that "the two clarifications are both true of the base repo as it stands, so 'the docs now disagree with the code' is not an available genuine finding."
Verified against the shared baseline at head: eval/review/repos/webapp/ contains only README.md, requirements.txt, docs/api.md, src/orders/{repository,pricing}.py and src/auth/session.py. There is no HTTP layer anywhere in the fixture — repository.py's get_order returns None and nothing maps it to a status code. So the new sentence "or 404 if no order with that id exists" documents behaviour that is implemented nowhere in the repo. (The rounding clarification IS accurate: pricing.py already calls round().)
A reviewer that does exactly what case 008 rewards — follow a prose claim to the code that backs it — and files a low-severity "this documented 404 is not implemented in this repo" fails forbidden_findings. The suite would then be punishing on 006 the behaviour it grades as correct on 008.
Suggestion
Either add a minimal route to the baseline fixture that actually returns 404 on a missing order, or reword the docs change to something the baseline demonstrably does (e.g. "returns None when no order with that id exists; the caller maps this to 404"). Failing that, raise 006's floor back to "high" and drop the claim that both clarifications are true of the base repo.
| # observation on a bot bump, but nothing in a one-line patch bump justifies | ||
| # a high/critical finding. | ||
| forbidden_findings: | ||
| - file: requirements.txt |
There was a problem hiding this comment.
MEDIUM — Case 007 asserts no high/critical surface on a pin that carries two live published advisories
The fix commit correctly repaired the delta (baseline eval/review/repos/webapp/requirements.txt is requests==2.32.2, head is 2.32.3 — a genuine patch bump, closing resolved thread 3864004654), but it re-verified the delta and not the destination.
Queried the GitHub advisory API directly (gh api '/advisories?ecosystem=pip&affects=requests@2.32.3') and got two published advisories that still affect the pinned version: GHSA-9hjg-9r4m-mvj7 / CVE-2024-47081 (.netrc credentials leak via malicious URLs, vulnerable range < 2.32.4) and GHSA-gc5v-m9x4-r6x2 / CVE-2026-25645 (insecure temp-file reuse in extract_zipped_paths, vulnerable range < 2.33.0).
Both are GitHub-rated medium, so the forbidden entry (category "", min_severity: high) only fires if an agent rates a known-vulnerable pin high — plausible for a credential leak, not certain. But the annotation states as fact that "there is no correctness, security or intent surface to review beyond the bump itself," and that is false today and grows more false as new advisories land against a frozen pin. Note that bumping the fixture to 2.32.4 does not fix it — the second advisory covers all of 2.32.x.
Suggestion
Pin a version verified advisory-clean at authoring time and record the check (and the date) in the annotation so it is re-run when the fixture is touched, or use a fictional/vendored package name so the case has no coupling to upstream disclosure at all. Failing that, restate the premise as "no breaking-change finding is justified" and narrow the forbidden category away from the empty wildcard, so a legitimate CVE observation cannot fail the case.
2 participants
The review eval judges outcome labels only, so a review can apply the right label while reporting the wrong things — or nothing at all. These cases judge the findings themselves.
What changes
capture-fixture.shnow snapshots inline PR review comments. That is wherepostreview.goputs structured findings, andgh pr view --jsondoes not expose them. Two deterministic judges read them:required_findings— a seeded bug must be reported, at or above a stated severity.forbidden_findings— a safe construct must not be promoted to a real vulnerability.Five cases: seeded bugs, false-positive bait, docs-only, a dependency bump, and
008-reassuring-docstring.Why 008 exists
It is the deliberate mirror of the bait case. Both PRs hash a secret-prefixed payload with md5 — in
005the digest is a receipt cache key, in008it is compared against an inboundX-Signatureheader. The surface pattern is identical, so neither case can be passed by recognising the construct, only by following the value to its use.In
008both prose channels lie: the docstring says "not a security boundary" and the PR body repeats it, while the function using the digest as a signature sits three lines below in the same diff.Without a case like this, a suite whose ground truth overlaps the review guidance measures instruction-following, and a prompt change that over-suppresses reads as a precision win.
Two ways the judges could have lied
They could not fail.
pass_rateissum(values)/len(values)over every case that produced a value, and a case declaring no ground truth returnsTrue. With one recall case out of five,required_findingsat0.70could not have failed even if that case missed every seeded bug. Both findings judges now carry anif:so the harness skips them where there is no ground truth, and the rates are over the cases that actually assert something.They could pass without seeing anything.
forbidden_findingsreturned success whenever no inline comment parsed as a finding. A 422 folds findings into the review body, and findings with no file or line are dropped before the inline comments — so precision was unverifiable, which is not the same as verified clean. It now fails closed and says why. Separately,fetch_review_commentsdiscarded jq's exit status, so a jq failure returned success with empty output and then aborted the script beforefixture-state.jsonwas written at all, losing every judge for that case.The 422 path
agents#209was parked pending the 422 fix, because a review whose positioned comments GitHub rejects has its findings folded into the review body instead. Reading inline comments alone would report a miss the agent never made, and would hide a promoted bait from the precision judge entirely — a silent pass on the case the suite exists to catch.Both judges now also parse those fallback bullets, anchored on the fixed note
post-reviewwrites, so an ordinary review body is never mined for things that merely look like findings. The eval no longer waits onagents#193: a review delivered through the fallback is judged on its findings rather than on how GitHub happened to accept them.Testing
eval/scripts/review-findings-judge-test.pyextracts the judge bodies from the shippedeval.yaml— no second copy to drift — and runs 56 synthetic payloads through them. Wired intomake script-test. Negative-checked: disabling the fail-closed guard fails the suite; replacing the word-boundary match with a substring match fails four boundary cases../eval/lint-cases.shpasses for review, triage, code and fix;pre-commit run --all-filesis clean.Notes for review
SKILL.md's vocabulary is per-dimension kebab-case and unrecognised categories route to the nearest dimension — pinning the token fails a correct review on a synonym.008is the only anti-bait case; letting it fail while the suite stays green would make it decorative, which is the defect theif:gates were added to fix.Why this lands first
The review agent is growing a second implementation: the pi runtime runs
review on Grok 4.6 without sub-agents (first live run:
fullsend-ai/pi-xai-vertex#4). These judges score the posted output —
inline review comments and the review body — not the architecture that
produced it, so the same cases can score both implementations
head-to-head. That comparison needs the eval to exist before either
implementation changes.