Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
71d0509
feat: add generic critique agent with ORG_KNOWLEDGE program_grounding
ascerra Jul 24, 2026
cedbaaa
fix: human-scannable critique comments and companion resolution
ascerra Jul 28, 2026
5f5f196
refactor: thin platform-*.md shared with refine (docs + contract)
ascerra Jul 28, 2026
e745dd4
docs: fix broken forge links in platform-*.md
ascerra Jul 28, 2026
1ff4310
fix: fetch companions from origin URL; install schema/pack hooks
ascerra Jul 28, 2026
3208800
fix: resolve platform context via companion fetch in pre-critique
ascerra Jul 28, 2026
40f16bf
fix: ship pre-explore.sh on branch for same-commit companion fetch
ascerra Jul 28, 2026
c431e7f
fix: prefetch sibling companions into shared fetch dir
ascerra Jul 28, 2026
2031f4f
feat(critique): honor open_question resolution buckets
ascerra Jul 28, 2026
bb848fd
chore: sync pre-explore knowledge-seeded clone fix for companion fetch
ascerra Jul 28, 2026
f3b79c2
fix: sync data-sources linkify + jira search/jql hints
ascerra Jul 29, 2026
b572301
chore: sync duplicate-block helpers from explore
ascerra Jul 29, 2026
96af908
fix(critique): drop abs base host_files; 0–5 scores; issue_id
ascerra Aug 8, 2026
361fca5
fix(critique): show assessment overall on 0–5 scale in Jira sticky
ascerra Aug 8, 2026
ca99854
fix(create): wrap child descriptions in Detailed Specification expand
ascerra Aug 8, 2026
94c3fef
fix(create): keep Scope/AC visible; collapse only Detailed Specification
ascerra Aug 8, 2026
bb708d5
fix(create): materialize Blocks links from refine dependencies
ascerra Aug 8, 2026
4947f44
fix(create): correct Blocks link create orientation
ascerra Aug 8, 2026
db6e0b8
chore: sync pre-explore GitHub auth + prefetch-manifest fix
ascerra Aug 8, 2026
861c7d5
fix(create): hard-allowlist target_project against field-config
ascerra Aug 8, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ script-test:
$(call run-timed,bash scripts/post-fix-test.sh)
$(call run-timed,bash scripts/post-retro-test.sh)
$(call run-timed,bash scripts/post-scribe-test.sh)
$(call run-timed,bash scripts/post-critique-test.sh)
$(call run-timed,bash scripts/validate-output-schema-test.sh)
$(call run-timed,bash scripts/gitlint-forbidden-type-scope-test.sh)
$(call run-timed,bash .github/scripts/check-e2e-authorization-test.sh)
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ First-class agents for the [fullsend](https://github.com/fullsend-ai/fullsend) p
| **Review** | Dispatches parallel sub-agents across six review dimensions | PR events, `/fs-review` |
| **Fix** | Implements targeted fixes from review feedback | Review comments, `/fs-fix` |
| **Prioritize** | Scores issues using the RICE framework | Schedule, `/fs-prioritize` |
| **Critique** | Adversarial reviewer and quality gate for refinement plans | `ready-to-critique` label |
| **Retro** | Analyzes completed workflows and proposes improvements | PR close, `/fs-retro` |

See [`docs/`](docs/) for detailed documentation on each agent.
Expand Down
460 changes: 460 additions & 0 deletions agents/critique.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@ agents:
- source: harness/review.yaml
- source: harness/retro.yaml
- source: harness/prioritize.yaml
- source: harness/critique.yaml
71 changes: 71 additions & 0 deletions docs/critique.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Critique Agent

Adversarial reviewer and quality gate for refinement plans. Evaluates a proposed decomposition from the refine agent and decides: approve (ready for issue creation), revise (send back to refine), or needs_input (escalate to a human for clarification).

> **Note**: The explore and refine agents are added in separate PRs. Cross-references to their docs pages will resolve once all three PRs merge.

## How the agent works

The critique agent runs after the refine agent has produced a decomposition plan. It reads the original work item, exploration context, the proposed plan, and any prior critique history (for revision rounds). It evaluates the plan across seven dimensions — coverage, granularity, dependency coherence, implementability, scope accuracy, assumption grounding, and description clarity — and produces a structured verdict.

The agent runs in a read-only sandbox. It cannot modify issues or create children directly — those mutations are performed by the post-script on the runner. The sandbox policy permits read-only egress to GitHub and Jira APIs plus inference endpoints, but the agent's only output is a structured JSON critique result consumed by the post-script, which posts a summary comment, attaches feedback, and applies labels to signal the next pipeline step.

## How it helps

- Prevents over-decomposition (15 issues when 6 would suffice) and under-decomposition (one giant epic).
- Catches scope creep — children that exceed what the parent asked for.
- Validates that acceptance criteria are testable, not vague ("should be fast").
- Detects assumption laundering — plans that look confident but are built on unverified guesses.
- Creates child issues automatically on approval when `AUTO_CREATE=true`.

## Commands

| Command | Where | Effect |
|---------|-------|--------|
| `/fs-create` | Issue comment | Creates child issues from an approved plan |
| `/fs-refine` | Issue comment | Restarts the refinement pipeline |

## Pipeline integration

The critique agent is the third stage of a three-stage refinement pipeline:

1. **Explore** — gathers technical context
2. **Refine** — decomposes the work item
3. **Critique** — reviews and approves/revises the decomposition (this agent)

### Verdict outcomes

| Verdict | Post-script action |
|---------|-------------------|
| `approved` + `AUTO_CREATE=true` | Creates child issues immediately via `create-children.sh` |
| `approved` + `AUTO_CREATE=false` | Posts approval, adds `refine-approved` label for human gate |
| `revise` + under round limit | Posts feedback, adds `ready-to-refine` label for revision |
| `revise` + at round limit | Escalates to human, adds `refine-needs-human` and `refine-escalated` labels |
| `needs_input` | Posts question, adds `refine-needs-input` label |

## Control labels

These labels are managed by the critique pipeline:

| Label | Meaning |
|-------|---------|
| `refine-approved` | Plan approved; ready for human review or auto-creation |
| `ready-to-refine` | Revisions requested; refine agent should re-run |
| `refine-revision-round-N` | Tracks which revision round the pipeline is on |
| `refine-needs-input` | Human must answer a question before proceeding |
| `refine-needs-human` | Max review rounds reached; human decision needed |
| `refine-escalated` | Distinguishes max-rounds escalation from genuine approval |

## Child issue creation

When the critique agent approves a plan and `AUTO_CREATE=true`, the post-script sources `create-children.sh` to create child issues. This script:

- Creates issues in topological order (parents before children)
- Supports both GitHub (sub-issues API) and Jira (parent-child hierarchy with fallback to "Relates" links)
- Deduplicates against existing children to prevent double-creation on re-runs
- Resolves Jira issue types against the project's available types
- Handles cross-project creation when a routing skill provides `target_project`

## Source

[`harness/critique.yaml`](../harness/critique.yaml)
7 changes: 7 additions & 0 deletions env/critique.env

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this to the harness, then env.sandbox.

Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
export ISSUE_CONTEXT=/sandbox/workspace/issue-context.json
export EXPLORE_CONTEXT=/sandbox/workspace/exploration_context.json
export REFINE_RESULT=/sandbox/workspace/refine-result.json
export CRITIQUE_HISTORY=/sandbox/workspace/critique-history.json
export PROJECT_ROUTING=/sandbox/workspace/routing-skill.md
export PLATFORM_CONTEXT=/sandbox/workspace/platform-context.md
export ORG_KNOWLEDGE=/sandbox/workspace/org-knowledge.md
61 changes: 61 additions & 0 deletions harness/critique.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
agent: agents/critique.md
doc: docs/critique.md
model: opus
image: ghcr.io/fullsend-ai/fullsend-sandbox:latest
policy: policies/critique.yaml

Comment thread
qodo-code-review[bot] marked this conversation as resolved.
role: critique
slug: fullsend-ai-critique

host_files:
- src: common/env/gcp-vertex.env
dest: /sandbox/workspace/.env.d/gcp-vertex.env

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] missing-host-file

env/critique.env sets ORG_KNOWLEDGE=/sandbox/workspace/org-knowledge.md but harness/critique.yaml has no host_files entry to inject this file into the sandbox. The pre-critique.sh generates this file via pack_org_knowledge(), but without the host_files mapping it will never exist in the sandbox, making program_grounding permanently un-scorable.

expand: true
- src: env/critique.env
dest: /sandbox/workspace/.env.d/critique.env
expand: true
- src: ${GOOGLE_APPLICATION_CREDENTIALS}
dest: /tmp/.gcp-credentials.json
- src: ${GCP_OIDC_TOKEN_FILE}
dest: /sandbox/workspace/.gcp-oidc-token
optional: true
# Absolute /tmp/workspace/* host_files are intentionally omitted here.
# fullsend 0.34.0 resolveBaseHostFiles rejects absolute paths on URL-fetched
# bases (security). Runtime files (issue-context, refine-result, etc.) are
# staged into /tmp/workspace by pre_script; absolute host mounts belong in
# the install overlay (e.g. konflux-ci/refinement .fullsend/harness/critique.yaml),
# not this generic agents base — same pattern as explore/refine bases.

pre_script: scripts/pre-critique.sh
post_script: scripts/post-critique.sh

validation_loop:
script: scripts/validate-output-schema.sh
schema: schemas/critique-result.schema.json
max_iterations: 2

timeout_minutes: 15

env:
runner:
ISSUE_KEY: "${ISSUE_KEY}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] contract-violation

FULLSEND_OUTPUT_FILE set to critique-result.json in env.runner but agent prompt writes agent-result.json and post-script searches for agent-result.json. Currently benign but creates a latent inconsistency.

Suggested fix: Remove the FULLSEND_OUTPUT_FILE override or align the agent prompt to write critique-result.json.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed earlier (6d0c112): removed FULLSEND_OUTPUT_FILE override — all paths use agent-result.json.

ISSUE_SOURCE: "${ISSUE_SOURCE}"
REPO_FULL_NAME: "${REPO_FULL_NAME}"
REFINE_RUN_ID: "${REFINE_RUN_ID}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] excessive-privilege

GH_TOKEN passed to sandbox via env.sandbox but agent reads pre-staged context files and does not need live GitHub API access. Violates least privilege.

Suggested fix: Remove GH_TOKEN from env.sandbox.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6d0c112: removed GH_TOKEN from env.sandbox.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed earlier (6d0c112): removed GH_TOKEN from env.sandbox.

GITHUB_ISSUE_NUMBER: "${GITHUB_ISSUE_NUMBER}"
REVIEW_ROUND: "${REVIEW_ROUND}"
MAX_REVIEW_ROUNDS: "${MAX_REVIEW_ROUNDS}"
AUTO_CREATE: "${AUTO_CREATE}"
sandbox:
REVIEW_ROUND: "${REVIEW_ROUND}"
MAX_REVIEW_ROUNDS: "${MAX_REVIEW_ROUNDS}"
Comment thread
qodo-code-review[bot] marked this conversation as resolved.

forge:
github:
pre_script: scripts/pre-critique.sh
post_script: scripts/post-critique.sh
env:
runner:
GH_TOKEN: "${GH_TOKEN}"
PUSH_TOKEN: "${PUSH_TOKEN}"
71 changes: 71 additions & 0 deletions policies/critique.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
version: 1

# Sandbox policy for the critique agent.
#
# Read-only agent: needs GitHub API for issue context and Jira for
# attachment retrieval. No write access — the post-script handles
# mutations on the runner.

filesystem_policy:
include_workdir: true
read_only: [/usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log]
read_write: [/sandbox, /tmp, /dev/null]
landlock:
compatibility: best_effort
process:
run_as_user: sandbox
run_as_group: sandbox

network_policies:
vertex_ai:
name: vertex-ai
endpoints:
- host: "api.anthropic.com"
port: 443
protocol: rest
enforcement: enforce
access: read-write
- host: "*.googleapis.com"
port: 443
protocol: rest
enforcement: enforce
access: read-write
binaries:
- path: "**/claude"
- path: "**/node"

github_api:
name: github-api
endpoints:
- host: "api.github.com"
port: 443
protocol: rest
enforcement: enforce
access: read-only
- host: "github.com"
port: 443
protocol: rest
enforcement: enforce
access: read-only
- host: "raw.githubusercontent.com"
port: 443
protocol: rest
enforcement: enforce
access: read-only
binaries:
- path: "**/gh"
- path: "**/git"
- path: "**/node"

jira_api:
name: jira-api
endpoints:
- host: "*.atlassian.net"
port: 443
protocol: rest
enforcement: enforce
access: read-only
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
binaries:
- path: "**/curl"
- path: "**/node"
Loading
Loading