feat: Add factory-approve action for auto-approving trivial PRs

factory-approve/README.md

@@ -0,0 +1,88 @@
+# Factory approve — auto-approval for trivial PRs
+
+Auto-approves very simple PRs (copy changes, styling, small self-contained tweaks) so they don't
+need another engineer's review. Deterministic safety gates run first; only if they all pass do two
+Claude reviewers judge the diff, and only their unanimous approve makes the factory bot account
+post an approving review. Everything fails closed — it never requests changes and never merges.
+
+## How to use
+
+Add the `factory-approve` label to a PR (drafts wait until ready). The label is a human opt-in
+flag the bot never touches: while it's on, every push is re-reviewed; remove it to opt out — the
+pipeline stands down and any active factory approval is dismissed with it, so an approval can
+never outlive the label that authorized it.
+
+- Approve → the factory account posts an approving review locked to the reviewed commit.
+- Reject / error → the report lands as a new PR comment with a collapsed details section, any
+  stale factory approval is dismissed, and older report comments are folded as outdated.
+- Label removed → nothing is posted, but any active factory approval is dismissed.
+
+A run stands down silently — no review, no comment, no cost — when a human review is already
+active, or when the content is unchanged since the last factory verdict (each verdict embeds a
+fingerprint of the title + diff, so develop-syncs, rebases, and empty pushes skip the paid review).
+
+## How it works
+
+1. **Static gates** (`scripts/prepare_review.mts`) — all must pass or the PR is rejected without
+   calling Claude: trusted author and actor, open and mergeable, targets the base branch, within
+   size limits, JS/TS only, no denied paths, no risky added lines, Conventional Commit title.
+2. **Two Claude reviewers** (via `anthropics/claude-code-action`) — the cheaper model first, the
+   second adversarial, and a rejection short-circuits. Each judges whether the change needs a
+   human, is correct, and follows the conventions of the surrounding code. The model can only
+   read code and write its verdict file — it cannot touch the PR, and all PR content enters the
+   prompt fenced as untrusted data.
+3. **Posting** (`scripts/post_verdict.mts`) — the only place GitHub is written to, as the factory
+   account. Any crash or invalid verdict means no approval.
+
+## Usage
+
+```yaml
+on:
+  pull_request_target: # runs the pipeline from the default branch, out of the PR's reach
+    types: [labeled, unlabeled, synchronize, opened, reopened, ready_for_review]
+# ... label guard (must let `unlabeled` runs through, even on drafts, so the approval is
+# dismissed when the label is removed), base-branch checkout, Node setup ...
+- uses: apify/actions/factory-approve@main
+  with:
+    pr-number: ${{ github.event.pull_request.number }}
+    actor: ${{ github.actor }}
+    github-token: ${{ secrets.GITHUB_TOKEN }}
+    factory-github-token: ${{ secrets.APIFY_FACTORY_GITHUB_TOKEN }}
+    anthropic-api-key: ${{ secrets.FACTORY_APPROVE_ANTHROPIC_API_KEY }}
+    policy: |
+      { "denyGlobs": ["infra/**"] }
+```
+
+See apify-core's `.github/workflows/factory_approve.yaml` for a complete workflow. The action
+exposes a `verdict` output (`approve` / `reject` / `error`).
+
+## Configure
+
+Defaults in `scripts/policy.mts` are the generic org-wide baseline: base `develop`, ≤5 files /
+≤150 lines, JS/TS modifications plus added test files, Conventional Commit titles, authors from
+`apify/product-engineering`, reviewers `claude-sonnet-5` + `claude-opus-4-8`.
+The optional `policy` input overrides them per repo: label, base branch, size and LLM limits,
+allowed extensions, title regex, author gate (org, teams, extra users), reviewer models (1–2),
+and the repo tier of `denyGlobs`; `denyGlobsAdd`, `riskyContentPatternsAdd`, and
+`authorGate.deniedUsersAdd` append. A core tier of supply-chain deny globs (workflows, manifests,
+lockfiles, env files, Dockerfiles, migrations, secrets) and the built-in risky-content patterns
+can never be removed. Invalid overrides fail closed as an error verdict at zero LLM cost, and any
+policy change invalidates memoized verdicts. The reviewer prompt (`scripts/prompt.mts`) is
+deliberately not overridable.
+
+## Setup
+
+1. Secrets: `APIFY_FACTORY_GITHUB_TOKEN` (the factory bot account, `repo` + `read:org`) and
+   `FACTORY_APPROVE_ANTHROPIC_API_KEY`.
+2. Create the `factory-approve` label.
+3. Branch protection: confirm one factory approval makes these PRs mergeable, and enable
+   "dismiss stale approvals when new commits are pushed".
+
+## Backtest
+
+Replay the whole pipeline over recent human PRs (requires the `claude` CLI,
+authenticated, run from a base-branch checkout):
+
+```bash
+GITHUB_TOKEN="$(gh auth token)" \
+node backtest/backtest.mts --repo apify/apify-core --last 200 [--policy overrides.json] [--output results.jsonl]

factory-approve/action.yaml

@@ -0,0 +1,126 @@
+name: Factory approve
+description: >-
+  Label-gated auto-approval for trivial PRs. Deterministic safety gates run first; only when every
+  gate passes does Claude judge the diff, and only a valid approve verdict makes the factory bot
+  account post an approving review. Fails closed everywhere, never requests changes, never merges.
+
+# Dependency-free Node scripts — nothing is installed at runtime. Generic defaults live in
+# scripts/policy.mts; per-repo tuning goes through the `policy` input.
+
+inputs:
+  pr-number:
+    description: Number of the pull request under review.
+    required: true
+  actor:
+    description: User whose action triggered the run (labeler or pusher); pass github.actor.
+    required: true
+  github-token:
+    description: Token used for GitHub API reads (secrets.GITHUB_TOKEN).
+    required: true
+  factory-github-token:
+    description: Token of the bot account that posts approvals (needs repo + read:org).
+    required: true
+  anthropic-api-key:
+    description: Anthropic API key for the claude-code-action verdict step.
+    required: true
+  policy:
+    description: >-
+      Optional JSON document with per-repo policy overrides (see the README for the allowed keys).
+      Empty means the built-in defaults. Invalid or out-of-range values fail closed: the run
+      reports an error verdict and approves nothing.
+    required: false
+    default: ''
+
+outputs:
+  verdict:
+    description: Final verdict — approve, reject, or error.
+    value: ${{ steps.post.outputs.verdict }}
+
+runs:
+  using: composite
+  steps:
+    # Never fails the step: crashes are captured into gates.json and surface as an `error` verdict.
+    - name: Run static safety gates
+      id: prepare
+      shell: bash
+      env:
+        GITHUB_TOKEN: ${{ inputs.github-token }}
+        FACTORY_GITHUB_TOKEN: ${{ inputs.factory-github-token }}
+        POLICY_OVERRIDES: ${{ inputs.policy }}
+      run: |
+        node "${{ github.action_path }}/scripts/prepare_review.mts" \
+          --pr "${{ inputs.pr-number }}" \
+          --repo "${{ github.repository }}" \
+          --actor "${{ inputs.actor }}" \
+          --out-dir "${{ runner.temp }}/factory-approve"
+
+    # The model only reads the checkout and writes its own verdict file; --disallowedTools blocks the
+    # GitHub tools so it cannot touch the PR, and posting happens only in post_verdict.mts. The Edit()
+    # rule (not Write()) governs the Write tool; the doubled slash marks an absolute path.
+    # continue-on-error so an LLM outage still reaches the post step (a missing verdict = error).
+    - name: Judge PR with Claude
+      if: steps.prepare.outputs.gates_passed == 'true'
+      continue-on-error: true
+      uses: anthropics/claude-code-action@v1
+      with:
+        anthropic_api_key: ${{ inputs.anthropic-api-key }}
+        github_token: ${{ inputs.github-token }}
+        show_full_output: true
+        prompt: ${{ steps.prepare.outputs.prompt }}
+        claude_args: |
+          --model ${{ steps.prepare.outputs.model }}
+          --max-turns ${{ steps.prepare.outputs.max_turns }}
+          --add-dir ${{ runner.temp }}/factory-approve
+          --allowedTools "Read,Glob,Grep,Edit(/${{ runner.temp }}/factory-approve/verdict.json)"
+          --disallowedTools "mcp__github,mcp__github_comment,mcp__github_inline_comment"
+
+    # Both reviewers must approve, so skip the second (more expensive) reviewer when the first did not
+    # approve. A missing or invalid first verdict counts as not-approved.
+    - name: Check first reviewer's verdict
+      id: first
+      if: steps.prepare.outputs.gates_passed == 'true' && steps.prepare.outputs.prompt2 != ''
+      shell: bash
+      env:
+        VERDICT_FILE: ${{ runner.temp }}/factory-approve/verdict.json
+      run: |
+        node -e '
+          const { readFileSync, appendFileSync } = require("node:fs");
+          let approved = false;
+          try { approved = JSON.parse(readFileSync(process.env.VERDICT_FILE, "utf-8")).verdict === "approve"; } catch {}
+          appendFileSync(process.env.GITHUB_OUTPUT, `approved=${approved}\n`);
+          console.log(`First reviewer approved: ${approved}`);
+        '
+
+    # Second independent reviewer with an adversarial stance; runs only if the first approved.
+    - name: Judge PR with Claude (second independent reviewer)
+      if: steps.prepare.outputs.gates_passed == 'true' && steps.prepare.outputs.prompt2 != '' && steps.first.outputs.approved == 'true'
+      continue-on-error: true
+      uses: anthropics/claude-code-action@v1
+      with:
+        anthropic_api_key: ${{ inputs.anthropic-api-key }}
+        github_token: ${{ inputs.github-token }}
+        show_full_output: true
+        prompt: ${{ steps.prepare.outputs.prompt2 }}
+        claude_args: |
+          --model ${{ steps.prepare.outputs.model2 }}
+          --max-turns ${{ steps.prepare.outputs.max_turns }}
+          --add-dir ${{ runner.temp }}/factory-approve
+          --allowedTools "Read,Glob,Grep,Edit(/${{ runner.temp }}/factory-approve/verdict2.json)"
+          --disallowedTools "mcp__github,mcp__github_comment,mcp__github_inline_comment"
+
+    # `!cancelled()` so the post step still runs when a reviewer step hard-fails (a missing verdict
+    # aggregates to `error`, which never approves), but is skipped when a newer commit supersedes this
+    # run (concurrency cancel) to avoid churning the PR-body comment.
+    - name: Post verdict
+      id: post
+      if: ${{ !cancelled() }}
+      shell: bash
+      env:
+        FACTORY_GITHUB_TOKEN: ${{ inputs.factory-github-token }}
+        POLICY_OVERRIDES: ${{ inputs.policy }}
+        WORKFLOW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+      run: |
+        node "${{ github.action_path }}/scripts/post_verdict.mts" \
+          --pr "${{ inputs.pr-number }}" \
+          --repo "${{ github.repository }}" \
+          --out-dir "${{ runner.temp }}/factory-approve"

factory-approve/backtest/backtest.mts

@@ -0,0 +1,173 @@
+// Backtests the factory-approve pipeline against recent closed PRs without posting anything to
+// GitHub. It replays the exact CI pipeline — static gates, then for gate-passing PRs the same
+// dual-reviewer LLM step via the local `claude` CLI — over the last N human-authored PRs, and
+// prints how many would have been auto-approved.
+//
+// Usage: node backtest.mts [--repo owner/repo] [--last 200] [--policy overrides.json] [--output results.jsonl]
+// Env: GITHUB_TOKEN (required);
+// Needs the `claude` CLI installed and authenticated; run from a checkout of the
+// base branch so Read/Grep context matches CI. `--policy` takes the same JSON document a repo would
+// pass to the action's `policy` input, so overrides can be replayed before enabling them.
+
+import { appendFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { parseArgs } from 'node:util';
+
+import { errorMessage, listRecentPullRequests } from '../scripts/github_api.mts';
+import { resolvePolicy } from '../scripts/policy.mts';
+import { buildReviewerContext, runGates } from '../scripts/prepare_review.mts';
+import { aggregateVerdicts, type ReviewerVerdict } from '../scripts/verdict.mts';
+import { runClaudeCliVerdict } from './claude_cli.mts';
+
+const CONCURRENCY = 4;
+const REVIEW_TIMEOUT_MS = 600_000;
+
+// Runs `worker` over every item with at most `poolSize` in flight. JS is single-threaded, so the
+// shared counters need no locking.
+async function forEachWithConcurrency<T>(items: T[], poolSize: number, worker: (item: T) => Promise<void>): Promise<void> {
+    let nextIndex = 0;
+    const runners = Array.from({ length: Math.min(poolSize, items.length) }, async () => {
+        while (nextIndex < items.length) {
+            await worker(items[nextIndex++]);
+        }
+    });
+    await Promise.all(runners);
+}
+
+const { values } = parseArgs({
+    options: {
+        repo: { type: 'string', default: process.env.GITHUB_REPOSITORY ?? '' },
+        last: { type: 'string', default: '200' },
+        policy: { type: 'string' },
+        output: { type: 'string' },
+    },
+});
+
+const githubToken = process.env.GITHUB_TOKEN;
+if (!githubToken) {
+    console.error('GITHUB_TOKEN is required');
+    process.exit(2);
+}
+if (!values.repo) {
+    console.error('--repo (or the GITHUB_REPOSITORY env var) is required');
+    process.exit(2);
+}
+const limit = Number(values.last);
+if (!Number.isInteger(limit) || limit <= 0) {
+    console.error('--last must be a positive integer');
+    process.exit(2);
+}
+let policy;
+try {
+    policy = resolvePolicy(values.policy ? readFileSync(values.policy, 'utf-8') : '');
+} catch (error) {
+    console.error(errorMessage(error));
+    process.exit(2);
+}
+
+// Only PRs a human engineer could label: skip bots, the denied service accounts, and release PRs.
+const isHumanPr = (pull: any) =>
+    pull.user?.type === 'User' &&
+    !pull.user?.login?.includes('[bot]') &&
+    !policy.authorGate.deniedUsers.includes(pull.user?.login) &&
+    !pull.head?.ref?.startsWith('release/');
+
+const { pulls, scanned } = await listRecentPullRequests(values.repo, {
+    token: githubToken,
+    limit,
+    state: 'closed',
+    filter: isHumanPr,
+});
+console.log(`Backtesting ${pulls.length} closed human PRs from ${values.repo} (scanned ${scanned}).`);
+
+const outDir = join(tmpdir(), 'factory-approve-backtest');
+mkdirSync(outDir, { recursive: true });
+if (values.output) writeFileSync(values.output, '');
+
+const failureCounts = new Map<string, number>();
+const llmCounts = new Map<string, number>();
+let staticPassCount = 0;
+let completed = 0;
+
+await forEachWithConcurrency(pulls, CONCURRENCY, async (pull) => {
+    try {
+        const { pr, files, gates } = await runGates({
+            repo: values.repo,
+            prNumber: pull.number,
+            actor: null,
+            backtest: true,
+            policy,
+            tokens: { github: githubToken, factory: githubToken },
+        });
+        if (gates.staticPassed) staticPassCount += 1;
+        for (const check of gates.staticChecks) {
+            if (!check.pass) failureCounts.set(check.id, (failureCounts.get(check.id) ?? 0) + 1);
+        }
+
+        let llm: ReviewerVerdict | null = null;
+        if (gates.staticPassed) {
+            const prDir = join(outDir, `pr-${pull.number}`);
+            mkdirSync(prDir, { recursive: true });
+            try {
+                // Same fetch-then-build as CI, so the backtest runs byte-identical prompts.
+                const { reviewerPrompts } = await buildReviewerContext({
+                    repo: values.repo,
+                    pr,
+                    files,
+                    headSha: gates.headSha,
+                    outDir: prDir,
+                    token: githubToken,
+                    policy,
+                });
+                const runs = await Promise.all(
+                    reviewerPrompts.map(async ({ verdictPath, prompt, model }) =>
+                        runClaudeCliVerdict({
+                            prompt,
+                            verdictPath,
+                            verdictDir: prDir,
+                            policy,
+                            model,
+                            timeoutMs: REVIEW_TIMEOUT_MS,
+                        }),
+                    ),
+                );
+                llm = aggregateVerdicts(runs);
+            } catch (error) {
+                llm = { verdict: 'error', reason: errorMessage(error) };
+            }
+            llmCounts.set(llm.verdict, (llmCounts.get(llm.verdict) ?? 0) + 1);
+        }
+
+        const line = {
+            prNumber: pull.number,
+            title: pull.title,
+            author: pull.user?.login,
+            merged: Boolean(pull.merged_at),
+            staticPassed: gates.staticPassed,
+            failedChecks: gates.staticChecks.filter((check) => !check.pass).map((check) => check.id),
+            ...(llm ? { llmVerdict: llm.verdict, llmReason: llm.reason } : {}),
+        };
+        if (values.output) appendFileSync(values.output, `${JSON.stringify(line)}\n`);
+        completed += 1;
+        console.log(
+            `[${completed}/${pulls.length}] #${pull.number} ${gates.staticPassed ? 'gates-pass' : 'gates-fail'}` +
+                `${llm ? ` → llm:${llm.verdict}` : ''} — ${pull.title}`,
+        );
+    } catch (error) {
+        completed += 1;
+        console.error(`[${completed}/${pulls.length}] #${pull.number} crashed: ${errorMessage(error)}`);
+    }
+});
+
+console.log('\n=== Summary ===');
+console.log(`PRs analyzed: ${completed}`);
+console.log(
+    `Passed static gates: ${staticPassCount} (${((staticPassCount / Math.max(completed, 1)) * 100).toFixed(1)}%)`,
+);
+const counts = ['approve', 'reject', 'error'].map((verdict) => `${verdict} ${llmCounts.get(verdict) ?? 0}`);
+console.log(`LLM verdicts on gate-passing PRs: ${counts.join(', ')}`);
+console.log('Gate failures by check:');
+for (const [id, count] of [...failureCounts.entries()].sort((a, b) => b[1] - a[1])) {
+    console.log(`  ${id}: ${count}`);
+}