feat: add generic critique agent - #87
Conversation
PR Summary by QodoAdd generic critique agent with verdict routing and child-issue creation
AI Description
Diagram
High-Level Assessment
Files changed (12)
|
|
🤖 Finished Review · ✅ Success · Started 5:47 PM UTC · Completed 6:03 PM UTC |
Code Review by Qodo
1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
ReviewFindingsHigh
Medium
Low
Next steps:
Previous runReviewFindingsHigh
Medium
Low
Next steps:
Previous run (2)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (3)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (4)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (5)ReviewFindingsHigh
Medium
Low
Labels: PR adds a new agent with 16 files under protected paths and has security findings (GHA workflow command injection) Next steps:
Previous run (6)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (7)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (8)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (9)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (10)ReviewFindingsHigh
Medium
Low
Labels: PR adds a new critique agent with supporting infrastructure — enhancement label fits the content classification convention. Next steps:
Previous run (11)ReviewFindingsHigh
Medium
Low
Previous run (12)Review — #87
SummaryThe latest commit ( The three medium-severity findings from the prior review remain unresolved and are carried forward unchanged. No new issues were introduced by the latest commit. The PR is functional for the primary GitHub flow; the remaining findings affect Jira integration completeness, display accuracy for edge-case verdicts, and defense-in-depth security hardening. Medium1. GHA workflow command injection via unsanitized interpolated values (unresolved from prior review)
An attacker controlling the refine agent's output JSON (e.g., via prompt injection through an issue description) can craft child titles containing Remediation: Import or duplicate 2. Schema does not require
|
| Verdict | Request Changes |
| Reviewed | 6d0c112 |
| Findings | 1 high, 3 medium |
| Re-review | 5 of 9 prior findings addressed |
Summary
The fix commit addresses the most critical findings from the prior review: the cross-platform parent key mismatch (via resolve_github_parent_number()), the unguarded source, the disallowedTools bypass, the FULLSEND_OUTPUT_FILE mismatch, sandbox GH_TOKEN leakage, and the refine-approved/refine-escalated label disambiguation. The escalation history mutation is also corrected — verdict is now preserved as revise with an escalated flag rather than being rewritten to approved.
However, three prior findings remain unresolved (Makefile test registration, GHA workflow command injection, schema/post-script assessment contract mismatch), and one new logic gap was found (Jira label handling is only implemented for one of four verdict paths).
High
1. Test suite not registered in Makefile (unresolved from prior review)
scripts/post-critique-test.sh (35 tests) is not added to the script-test target in the Makefile. CI runs make test via the script-test target. Every other agent's test suite is registered (triage, prioritize, code, review, fix, retro, scribe, validate-output-schema). Critique tests are never executed in CI.
Remediation: Add $(call run-timed,bash scripts/post-critique-test.sh) to the script-test target in the Makefile.
Medium
2. GHA workflow command injection via unsanitized interpolated values (unresolved from prior review)
create-children.sh, post-critique.sh, and pre-critique.sh interpolate unsanitized values into ::notice:: and ::warning:: GHA workflow commands. The repo has an established sanitize_gha() pattern in post-scribe.sh that strips ::, %0A, %0D, and newline characters.
3. Schema does not require assessment for non-approved verdicts (unresolved from prior review)
The JSON schema's allOf conditional validation only requires assessment when verdict is approved.
4. Jira label handling missing for 3 of 4 verdict paths
The revise under limit verdict path adds labels to Jira issues. The other three verdict paths only add labels on GitHub.
Low
- Tests re-implement logic
- Missing token masking
- Implicit contract in
create-children.sh
Previous run
Review — #87
| Verdict | Request Changes |
| Reviewed | 23d7556 |
| Findings | 3 high, 6 medium |
Summary
This PR adds a well-structured critique agent following the repo's established pre-script → sandbox → post-script pipeline. The agent prompt is thorough, the schema uses conditional validation correctly, and the test suite covers core verdict routing. However, there are several code-level issues.
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
🤖 Finished Review · ✅ Success · Started 6:12 PM UTC · Completed 6:42 PM UTC |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
🤖 Finished Review · ✅ Success · Started 7:57 PM UTC · Completed 8:07 PM UTC |
Superseded by updated review
fullsend-ai-review
Bot
added
the
requires-manual-review
There was a problem hiding this comment.
Move this to the harness, then env.sandbox.
rh-hemartin
left a comment
rh-hemartin
left a comment
There was a problem hiding this comment.
I want to move some vars to the harness file.
Update for konflux-ci/refinement
|
|
🤖 Review · |
020dd01 to
94e934c
Compare
CI fix pass (
|
|
🤖 Review · |
Introduce critique harness, agent prompt, scripts, and schema. Consume optional ORG_KNOWLEDGE for install-injected org context and score program_grounding when a real pack is present. Resolve companion script helpers for base-composition installs. Signed-off-by: Adam Scerra <ascerra@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
94e934c to
71d0509
Compare
|
Rebased onto latest |
|
🤖 Finished Review · ✅ Success · Started 8:13 PM UTC · Completed 8:32 PM UTC |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
fullsend-ai-review
Bot
removed
the
requires-manual-review
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Remove /tmp/workspace host_files from the URL base (fullsend 0.34.0 rejects absolute paths there — install overlays mount them). Score 0.0–5.0 with one decimal. Rename key→issue_id for redaction safety. Assisted-by: Cursor Signed-off-by: Adam Scerra <ascerra@redhat.com>
|
🤖 Finished Review · ✅ Success · Started 5:04 PM UTC · Completed 5:20 PM UTC Commit: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
format_assessment_table_md hardcoded /100 after scores moved to 0–5. Co-authored-by: Cursor <cursoragent@cursor.com>
|
🤖 Finished Review · ✅ Success · Started 6:21 PM UTC · Completed 6:34 PM UTC Commit: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Pass --wrap-detail to markdown-to-adf so Epic/Feature detail after the first HR collapses like parent issues. Harden jq --argjson inputs and silence createmeta parse noise when the live types API returns empty. Co-authored-by: Cursor <cursoragent@cursor.com>
|
🤖 Finished Review · ✅ Success · Started 7:04 PM UTC · Completed 7:20 PM UTC Commit: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Stop re-appending Acceptance Criteria when the child description already has them, promote Requirements behind an explicit Detailed Specification heading, and peel the generator footer out of the expand so Jira children are scannable once Description is shown. Co-authored-by: Cursor <cursoragent@cursor.com>
|
🤖 Finished Review · ✅ Success · Started 7:38 PM UTC · Completed 7:56 PM UTC Commit: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Walk children[].dependencies after create and open directed Jira Blocks edges so triage can see sequencing. Cache link-type resolution per preferred name so Relates never shadows Blocks, and teach critique not to praise all-KONFLUX when routing lists team projects. Signed-off-by: Adam Scerra <ascerra@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
🤖 Review · Commit: |
Stage Jira create API needs inwardIssue=blocker and outwardIssue=blocked to read back as blocker blocks blocked. The prior mapping inverted every dependency edge. Signed-off-by: Adam Scerra <ascerra@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
🤖 Finished Review · ✅ Success · Started 10:10 PM UTC · Completed 10:32 PM UTC Commit: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Companion fetch must match explore tip so critique/create companions stay aligned on public-only authenticated clones. Signed-off-by: Adam Scerra <ascerra@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
🤖 Review · Commit: |
Reject explicit target_project values outside parent ∪ project-field-config ∪ issue-context routable_projects so LLM-invented keys cannot create issues in arbitrary Jira projects. Signed-off-by: Adam Scerra <ascerra@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
🤖 Finished Review · ✅ Success · Started 11:14 PM UTC · Completed 11:33 PM UTC Commit: |
| local response http_code body | ||
| response=$(GIT_TERMINAL_PROMPT=0 curl "${curl_args[@]}" \ | ||
| "https://api.github.com/repos/${ref}" 2>/dev/null || printf '\n000') | ||
| http_code=$(printf '%s' "$response" | tail -1) |
There was a problem hiding this comment.
[high] logic-error
The validate_repo function uses jq -r '.private // true' to check whether a repo is public. However, jq's // (alternative) operator treats both null AND false as falsy. When a public GitHub repo returns {"private": false}, jq evaluates .private // true as true (the fallback), not false. The subsequent == "false" test therefore never succeeds, causing validate_repo to reject ALL repositories. Every referenced repo will be skipped.
Suggested fix: Replace .private // true with an explicit conditional, e.g. jq -r 'if .private == null then true else .private end' or use jq's exit code: jq -e '.private == false' >/dev/null 2>&1.
| CHILD_SCOPE=$(jq -r ".children[${i}].estimated_scope // \"M\"" "${RESULT_FILE}") | ||
|
|
||
| FULL_BODY="$(compose_child_markdown "$CHILD_DESC" "$CHILD_AC" "$CHILD_PRIORITY_TEXT" "$CHILD_SCOPE")" | ||
|
|
There was a problem hiding this comment.
[medium] logic-error
In the orphan fallback loop, the target_platform == 'gitlab' case is not handled. The main topological loop (lines 709-718) correctly skips GitLab children with a 'not yet supported' message and continue, but the orphan loop (lines 800-808) only checks for github and jira platforms. A child with target_platform: gitlab falls through to the default creation path.
Suggested fix: Add elif [[ "$CHILD_TARGET_PLATFORM" == "gitlab" ]] with a continue and skip message, matching the main loop's behavior.
| fi | ||
|
|
||
| if [[ -z "$resolved" ]]; then | ||
| echo "::warning::Could not discover Jira issue link type for '${preferred}'; using '${preferred}'" >&2 |
There was a problem hiding this comment.
[low] api-contract
The jira_link_blocks function's inwardIssue/outwardIssue mapping relies on a specific Jira Cloud instance's Blocks link type orientation. The code comment documents verification (2026-08-08). If deployed against an instance with a non-standard Blocks definition, the direction could be inverted.
| # triage cannot see blocked/blocking edges. | ||
| DEP_LINKS_OK=0 | ||
| DEP_LINKS_SKIP=0 | ||
| if [[ "${ISSUE_SOURCE:-}" == "jira" && -n "${JIRA_HOST:-}" && -n "${JIRA_EMAIL:-}" ]]; then |
There was a problem hiding this comment.
[low] edge-case
The dependency materialization guard checks for JIRA_HOST and JIRA_EMAIL but not JIRA_API_TOKEN. If the token is empty, every API call fails with HTTP 401.
| } | ||
| } | ||
| }, | ||
| "program_grounding": { |
There was a problem hiding this comment.
[low] api-contract
The program_grounding dimension object is missing additionalProperties: false, unlike all other dimension objects in the assessment schema.
| fi | ||
|
|
||
| export RESULT_FILE="$REFINE_RESULT_FILE" | ||
| source "$(_resolve_companion create-children.sh)" |
There was a problem hiding this comment.
[low] logic-error
When AUTO_CREATE=true, source create-children.sh runs in the same shell with set -euo pipefail. If create-children.sh fails, post-critique.sh aborts after posting the approval comment but before posting the child creation summary, leaving the issue in an inconsistent state.
| body=$(echo "$response" | sed '$d') | ||
|
|
||
| if [[ "$http_code" -ge 400 ]]; then | ||
| echo "::warning::Failed Blocks link ${blocker_key} blocks ${blocked_key} (HTTP ${http_code}): ${body}" >&2 |
There was a problem hiding this comment.
[low] injection-vuln
GHA workflow commands (::warning::) interpolate unsanitized values from Jira API responses ($body). While the most dangerous GHA command injections (set-env, add-path) are disabled by default on modern runners, cosmetic injection remains possible.
Closing without merge — keep the branchClosing #87 (critique). Critique / create-children is not cancelled. Same situation as #11 / #86: open draft PR + pin-by-SHA iteration kept re-triggering fullsend review. We considered extending Team decision 2026-08-10 (notes). Upstream: explore #11 ( |
|
🤖 Finished Retro · ✅ Success · Started 12:41 AM UTC · Completed 12:55 AM UTC Commit: |
Retro: PR #87 — feat: add generic critique agentTimelinePR #87 added a generic critique agent to Findings1. Protected-path finding was a permanent, unresolvable blocker. All 24 files fell under protected paths ( 2. Identical findings repeated verbatim 9–14 times. Core findings (GitLab orphan loop, GHA injection vulnerability, missing 3. No manual mechanism to pause review on a PR. The team's decision to close 3 PRs simultaneously to escape review noise strongly supports fullsend#5650 ( 4. Review quality was high where it ran. The review agent correctly identified real bugs (jq 5. Significant token waste from redundant runs. 14 full review runs on a 6,787-line PR, each dispatching sub-agents across 6 dimensions, with diminishing returns after the first 2–3 runs. This supports agents#343 (scope re-review to finding verification when push only addresses prior findings). Proposals filed
|
2 participants
Summary
create-children.shfor child issue creation (GitHub sub-issues API + Jira hierarchy with fallbacks)comment-helpers.sh,pre-explore.sh, andmarkdown-to-adf.pyRelated Issue
Continuation of agent migration from
konflux-ci/refinementto generic agents repo.Changes
agents/critique.mdharness/critique.yamlenv.runner/env.sandboxformatpolicies/critique.yamlschemas/critique-result.schema.jsondescription_clarityandproject_routingdimensionsscripts/pre-critique.shscripts/post-critique.shscripts/post-critique-test.shscripts/create-children.shenv/critique.envdocs/critique.mdconfig.yamlharness/critique.yamlREADME.mdTesting
bash scripts/post-critique-test.sh— 35/35 tests passingChecklist
PUTverb blocked indisallowedToolsadd_labelhelper consistently (not rawgh api)customized/overlay referencesenv.runner/env.sandboxformatdescription_clarityandproject_routingin critique schemaMade with Cursor